Published on 25.05.15 in Vol 3, No 2 (2015): April-June
Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/4106, first published Dec 04, 2014.
Comprehensive Evaluation of Electronic Medical Record System Use and User Satisfaction at Five Low-Resource Setting Hospitals in Ethiopia
Background: Electronic medical record (EMR) systems are increasingly being implemented in hospitals of developing countries to improve patient care and clinical service. However, only limited evaluation studies are available concerning the level of adoption and determinant factors of success in those settings.
Objective: The objective of this study was to assess the usage pattern, user satisfaction level, and determinants of health professional’s satisfaction towards a comprehensive EMR system implemented in Ethiopia where parallel documentation using the EMR and the paper-based medical records is in practice.
Methods: A quantitative, cross-sectional study design was used to assess the usage pattern, user satisfaction level, and determinant factors of an EMR system implemented in Ethiopia based on the DeLone and McLean model of information system success. Descriptive statistical methods were applied to analyze the data and a binary logistic regression model was used to identify determinant factors.
Results: Health professionals (N=422) from five hospitals were approached and 406 responded to the survey (96.2% response rate). Out of the respondents, 76.1% (309/406) started to use the system immediately after implementation and user training, but only 31.7% (98/309) of the professionals reported using the EMR during the study (after 3 years of implementation). Of the 12 core EMR functions, 3 were never used by most respondents, and they were also unaware of 4 of the core EMR functions. It was found that 61.4% (190/309) of the health professionals reported over all dissatisfaction with the EMR (median=4, interquartile range (IQR)=1) on a 5-level Likert scale. Physicians were more dissatisfied (median=5, IQR=1) when compared to nurses (median=4, IQR=1) and the health management information system (HMIS) staff (median=2, IQR=1). Of all the participants, 64.4% (199/309) believed that the EMR had no positive impact on the quality of care. The participants indicated an agreement with the system and information quality (median=2, IQR=0.5) but strongly disagreed with the service quality (median=5, IQR=1). The logistic regression showed a strong correlation between system use and dissatisfaction (OR 7.99, 95% CI 5.62-9.10) and service quality and satisfaction (OR 8.23, 95% CI 3.23-17.01).
Conclusions: Health professionals’ use of the EMR is low and they are generally dissatisfied with the service of the implemented system. The results of this study show that this dissatisfaction is caused mainly and strongly by the poor service quality, the current practice of double documentation (EMR and paper-based), and partial departmental use of the system in the hospitals. Thus, future interventions to improve the current use or future deployment projects should focus on improving the service quality such as power infrastructure, user support, trainings, and more computers in the wards. After service quality improvement, other departments (especially inter-dependent departments) should be motivated and supported to use the EMR to avoid the dependency deadlock.
JMIR Med Inform 2015;3(2):e22
Electronic medical record (EMR) systems are increasingly being implemented in hospitals to achieve the following six aims of improved care: (1) safety, (2) effectiveness, (3) patient centeredness, (4) timeliness, (5) efficiency, and (6) quality . However, many EMR systems which are technically sound for developers and healthcare managers, face resistance from users and may end up in failure [ ].
Measuring success of an information system is difficult because success does not have a common explicit definition , and is dependent on expectations. The agreed hypothesis to say an information system is successful is when the implemented system is accepted to be used by the end user and the users are satisfied with the system. As a result, a number of researchers suggested that user satisfaction and system use are the primary determinants of user adoption, and therefore are suitable to measure information system success [ - ]. Mazzoleni et al describe health professionals' satisfaction towards EMR system as “essential to the survival” of the system [ ]. The different implementation projects which were reported as failed have often been those in which the end users were dissatisfied or the core system functions were not properly used [ ].
Even though most health professionals generally perceive that technology can help eliminate the burden of paper-based documentation and the unavailability of patient data in critical situations, they also get easily dissatisfied when an introduced system or support does not meet their expectations . Pare et al [ ] assessed the factors in clinical information system implementation success and reported on the necessity of identifying risk factors and involving health professionals starting from the development and pilot phase to avoid failure. Many factors affect the adoption of EMR systems and they vary within the system users, hospital setting, and the type of system in use [ , ].
EMR systems are also increasingly being incorporated into the healthcare organizations of developing countries that do not have well-developed infrastructure and well-trained technical personnel to use and manage the systems. As outlined by Sood , the determinant factors that affected the information system success in those settings might be different from factors in developed countries. Hence, rigorous evaluation studies on different health information system implementation projects in those settings are necessary to understand the critical success and failure factors. To date, since only a few reports are available [ ], this study was conducted to fill this gap by evaluating the use and user satisfaction of an EMR system implemented in Ethiopia.
In the current health sector development plan of Ethiopia, strengthening the health management information system and incorporating a computerized health management information system (HMIS) are priority policy plans to ensure health service quality and equity . As a result, an HMIS documentation package was implemented to standardize the patient documentation and reporting systems in all health facilities of the country [ ]. The implementation of this standardized documentation system was mandatory and a prerequisite to implement the EMR system to make sure that the paper workflow is in line with the EMR.
In 2009, the Ministry of Health, with support of the Tulane University Technical Assistance Project in Ethiopia (TUTAPE), started the development and implementation of a comprehensive EMR system for hospitals called SmartCare. The system was deployed in 5 hospitals in Addis Ababa  and other hospitals in regional cities. In 2013, the Ministry of Health adapted the system as a national EMR for all hospitals, and planned to scale it up to further hospitals and regions [ ].
Even though the system developers claimed it the best EMR, it had not been thoroughly evaluated by a neutral investigator. As explained by Joaquin  and Fraser [ ], information system projects, especially those which were developed by NGO’s, need thorough independent evaluation prior to scale-up to determine if system expansion is both worthwhile and feasible. Hence, this study conducted by an independent and neutral investigator, filled this gap by identifying the main factors which needed to be addressed before a costly expansion.
The main objectives of this study are (1) to assess the current EMR use rate among health professionals, (2) to assess the use level of core EMR functions, (3) to determine the user satisfaction level of health professionals, and (4) to identify determinants factors of user satisfaction towards the EMR system in the study hospitals. The study was conducted in accordance with the Guidelines for Good Evaluation Practices in Health Informatics (GEP-HI)  and reported based on the Statement on Reporting of Evaluation Studies in Health Informatics (STARE-HI) [ ].
This evaluation study was conducted in 5 hospitals in Ethiopia. All are government hospitals located within a 15 km radius in Addis Ababa, the capital city of Ethiopia. Of the 5 hospitals, one is an 80-bed children and mothers care specialized hospital with inpatient and outpatient clinics, 2 are 300-bed teaching referral hospitals with different specialized clinics, and the remaining 2 are 200-bed general hospitals with both inpatient and outpatient services. All of the hospitals implemented the HMIS in 2009, and started EMR implementation in 2011 .
System Detail and System in Use
SmartCare is a portable, integrated EMR system that is currently used by three African countries (Zambia, Ethiopia, and South Africa), and presumably is the largest EMR system in use in Africa . The system was designed in Africa to be robust in environments with limited infrastructure. The system also offers a touch screen interface to minimize the learning curve.
This comprehensive EMR system has different components (modules) that can be used in the various units of healthcare facilities (). The main modules of SmartCare include registration, outpatient department, inpatient (to admit, follow, and discharge patients in wards), tuberculosis, pediatrics, HIV/AIDS (to manage patients in antiretroviral therapy clinics), antenatal care, postpartum, pharmacy, drug stock control, laboratory (to store and send laboratory results to the requesting clinic), eHMIS (to generate monthly, quarterly, and annual reports), and finance. Currently all but the financial module are implemented and used in the hospitals of this study.
Installation of the network, server infrastructure, and the EMR system at all hospital sites was conducted by TUTAPE. After implementation, 5 day-long onsite user training sessions were provided to all health professionals of each hospital. Additionally, TUTAPE computer and network experts are responsible to provide continuous on-call service for technical assistance during system failure.
On average, the SmartCare system has been in use in the 5 hospitals of this study since 2011. In parallel, the paper-based medical record system is also still in use which means that the health professionals are expected to document both on paper and within the EMR system. The plan of the government is to expand the system to the other 127 existing hospitals in the country after the pilot testing. Additionally, the government is training health informatics professionals to support the health management information system and implementation of EMR in the country .
A quantitative, cross-sectional study design based on a validated questionnaire was used to assess the use pattern, user satisfaction level, and determinant factors of SmartCare in 5 government hospitals located in Addis Ababa. To better understand the use and challenges of the system, we also assessed the current method of documentation on the EMR server and the fluctuation levels of power access in the study hospitals. The selected hospitals were chosen because the EMR system had been implemented for 3 years. Additionally, as 2 are teaching hospitals and 3 are general hospitals, representative hospital types were included.
This study was conducted based on the DeLone and MacLean (D&M) information system success evaluation model , a validated and the most commonly used information system success evaluation method among the informatics community [ ]. The basic dimensions in this model are system quality, information quality, service quality, system use, user satisfaction, and net benefit.
For this evaluation, we chose 5 factors from the D&M model that are relevant for user satisfaction and use rate evaluation, by excluding net benefit. Instead of net benefit, user background was included as a determinant factor to be tested because many researchers reported it as a determinant factor especially in low literacy working environments . Additionally, we assessed the level of use of core EMR functions since they have been found to be a main factor of user satisfaction [ , ].
Participants and Sample Size
The participants of this study, health professionals across the 5 study hospitals, were categorized into the following four groups (1) physicians (doctors and health officers), (2) nurses (clinical and midwifery), (3) lab and pharmacists (laboratory and pharmacy professionals), and (4) HMIS (health data entry and management secretaries, information system officers, and data clerks). The sample size of 422 participants was calculated assuming a 95% Cl and 10% non-response rate. All health professionals, who were selected by a simple random sampling technique among their professional category and who also served for >6 months in the hospitals, were approached to complete the questionnaire.
This study began in January, 2014 after obtaining ethical clearance. The first step was to choose data collectors from each hospital and familiarize them with the objective and methodology of the research. Seven data collectors were chosen and trained on how to collect the questionnaire and the level of support they should give to avoid bias. The questionnaires were distributed to the participants by visiting them in their offices, mostly during the afternoon. To motivate participants, we provided one Samsung Galaxy III phone as a reward, by a lottery method, to all of the participants who fully completed the questionnaire. Data collection took place over a one-month period.
Outcome Measure and Evaluation Criteria
The outcome measures and corresponding evaluation criteria are shown in.
Data Acquisition and Measurement
A questionnaire was developed based on standardized and previously validated instruments. The questions were divided into three categories. The first category, on the user background, had 15 questions about general socio-demographic data, computer training, and current use of the EMR system. Some of them were adapted from Mahmood et al , Lawrence and Low [ ], and Igbara and Nachman [ ]. The second category was designed to measure the perceived system quality, information quality, service quality, satisfaction, and expectation towards future benefits. To assess system quality, 7 items were used, whereas 10 were used to assess information quality, 9 to assess service quality, 5 to assess user satisfaction questions, and 3 to assess expectations towards future benefit. The items were adapted from Seddon et al [ ] and Doll et al [ ]. For the service quality, we added additional setting-specific questions to reflect the power interruptions and the computer access challenges faced in the study hospitals. The third category contained 12 questions on core EMR functions which were adapted from Moustafa et al [ ] with amendments from EMR officers on the main core functionalities of the system.
A pretest of the questionnaire was conducted in a hospital that was not part of the study in which 5 physicians, 8 nurses, 3 lab/pharmacists, and 5 HMIS staffs participated. Based on the pretest results, 2 questions were amended for wording as they were reported to be unclear from a health professional’s perspective. The reliability of the items was evaluated with Cronbach’s alpha, and the values were all above .84, indicating satisfactory reliability of the questionnaire.
Descriptive statistics were performed to describe the characteristics of the participants, EMR use rate, and user satisfaction. Binary logistic regression analysis was used to identify determinant factors of user satisfaction among the study participants.
The selected dependent variable for this study was “user satisfaction”. In the questionnaire, participants were asked to rate their satisfactions on a 5-point Likert scale. Median and interquartile ranges (IQRs) with percentages were used. In the cross tabulation of our data, we found that responses of “very satisfied” and “very dissatisfied” were very low. Consequently, for the logistic regression, the 5-item scales were merged into two groups from “very satisfied and satisfied” and "very dissatisfied and dissatisfied" to “satisfied” “dissatisfied”, respectively. After this dichotomization, the determinant factors were analyzed using binary logistic regression. All analyses were performed using SPSS Software version 22.
Ethical approval was granted by the Institutional Review Board (IRB) of the University of Gondar and the Addis Ababa City Administration Health Bureau. Permission for data collection was also obtained from each of the hospitals. The participants were informed about the study, its importance, and confidentiality of the information collected, as well their right to leave the study at any time. Written consent was obtained from participants in a form provided with the questionnaire and the procedure was approved by the IRB.
Out of the 422 participants of this study, 96.2% (406/422) completed the questionnaire. Of all the questionnaires, 7 were not completed and 9 were not returned. The mean age of the participants was 34 years (SD 8.5). Of all the participants, 53.4% (217/406) were males, and the majority of the participants were nurses (43.3%, 176/406), followed by physicians (20.4%, 83/406), HMIS staff (18.2%, 74/406), and laboratory and pharmacy staff (18%, 73/406). The participants had a mean work experience of 8.6 years (SD 7.2) in the current hospital. The detailed socio-demographic characteristics of the respondents are shown in.
|Characteristics||Frequency||Relative frequency, %|
|Age of respondents, years|
|Work experience in current hospital, years|
|Lab and pharmacists||73||18.0|
EMR Use Pattern and Related Characteristics
Among the respondents, 76.1% (309/406) started using the system immediately after implementation and user training. In this context, 'use' refers to a complete use of the EMR to document patient information, in addition to the patient card. Among them, the major proportion of users were HMIS staff 20.7% (64/309), followed by nurses 44.0% (136/309), laboratory and pharmacy staff, and physicians 18.4% (57/309). However, during the data collection, only 31.7% (98/309) of the professionals reported to use the system with the majority being HMIS staff (54.0%, 53/98), followed by nurses (33.6%, 33/98), lab and pharmacy staff (33.6%, 33/98), and physicians (7.1%, 7/98). Those who completely stopped using the EMR reported that they were using the computers for other purposes such as browsing the Internet, word processing, while others returned them to the store.
Training, Information Technology Qualification, and EMR Experience
In terms of training, 64.0% (260/406) participated in the EMR user training and 60.6% (246/406) had been previously trained on the HMIS implementation. Almost half of the staff (47.2%, 192/406) responded to having a “reasonable information technology (IT) qualification”. The majority (76.1%, 309/406) did not have previous EMR experience, 20.4% (83/406) reported to having individual computer access in the office, and of those, the majority were HMIS staff (55.4%, 46/83). The others shared the computer access with 2 people (11.5%, 47/406), 3 people (16.7%, 68/406), 4 people (17.7%, 72/406) and the majority shared with more than 5 people (57.8%, 235/406), mainly nurses (94.8%, 223/235) ().
|Characteristics||n (%)NursesLaboratory& PharmacyHMIS|
|Physicians||Nurses||Laboratory & Pharmacy||HMIS|
|Computer access in hospital|
|Individual||11 (20.4)||12 (9.0)||14 (26.9)||46 (71.9%)|
|For 2 practitioners||4 (7.4)||25 (18.7)||11 (21.2)||7 (10.9)|
|For 3 practitioners||13 (24.4)||33 (24.6)||14 (26.9)||8 (12.5)|
|For 4 practitioners||23 (42.6)||42 (31.3)||5 (9.6)||2 (3.1)|
|For >5 practitioners||3 (5.6)||223 (16.4)||8 (15.4)||1 (1.6)|
|Yes||22 (26.5)||131 (74.4)||35 (47.9)||58 (78.4)|
|No||61 (73.5)||45 (25.6)||38 (52.1)||16 (21.6)|
|No qualification||18 (21.7)||43 (24.4)||37 (50.7)||6 (8.1)|
|Reasonable qualification||58 (69.9)||82 (46.6)||25 (34.2)||27 (36.5)|
|Good qualification||7 (8.4)||51 (29.0)||11 (15.1)||41 (55.4)|
|Yes||28 (33.7)||120 (68.2)||45 (61.6)||67 (90.5)|
|No||55 (66.3)||55 (31.3)||28 (38.4)||7 (9.5)|
|Another EMR experience|
|Yes||35 (42.7)||16 (9.1)||25 (34.2)||20 (27.0)|
|No||47 (57.3)||160 (90.9)||48 (65.8)||54 (73.0)|
|SmartCare use since implementation|
|Yes||57 (68.7)||136 (77.3)||52 (71.2)||64 (86.5)|
|No||26 (31.3)||40 (22.7)||21 (28.8)||10 (13.5)|
|Current SmartCare use|
|Yes||7 (12.1)||33 (24.3)||5 (9.4)||53 (81.5)|
|No||51 (87.9)||103 (75.7)||48 (90.6)||12 (18.5)|
Usage Pattern From Server Log Analysis
The observations of the patient chart in the registration department of the hospitals showed that on average 184,594 patients per hospital have paper-based records. Out of those, 58.7% (108,450/184,594) were also available in the EMR system database. However, only 4.8% (5244/108,450) of those patients had a documented main diagnosis patient history in the EMR server.
In terms of infrastructure, hospitals on average had 61 computers and one server for the EMR system. Four of the hospitals had one or more IT staff members, however, they were not specifically hired for the EMR system. Rather, they primarily worked for the statistics office and they took the EMR system work as their secondary task. The number of IT staff, computers, and the number of medical records in the hospital paper and server databases is shown in.
|Characteristics||Hospital 1||Hospital 2||Hospital 3||Hospital 4||Hospital 5|
|Number of IT staffs||1||2||0||1||1|
|Number of computers for the EMR system||73||61||55||66||51|
|Number of patients with paper-based records||222,937||179,327||71,985||171,292||277,421|
|Number of patients registered in the EMR system||199,866||155,967||55,644||95377||35,398|
|Number of patients who have a main diagnosis in the EMR||7841||4848||4500||5323||3721|
Main Reported Reasons of Not Using the System
An open-ended item question was provided to assess the main reasons for not using the system (). Among the respondents who reported that they did not have the time to use it, the majority were physicians (75.7%, 25/33). Among the respondents who reported that the main reason for not using the system was that the computers were not working were nurses (60.6%, 37/61), while 69.7% (30/43) reported that power fluctuations in their department were the main obstacle. Of the respondents who reported that “the other departments were not using the system” was the main reason for not using the EMR, 48% (12/25) were laboratory and pharmacy staff.
Use of the Main EMR Components in the Study Hospitals
Participants were also assessed on the use of the main components of the EMR as shown in. The most frequently used functionalities of the respondents were “find patients with certain characteristics” (45.6%, 141/309), and “produce patient summary” (45.9%, 142/309). The detailed use of the EMR components by the various categories of health professionals is shown in .
|Component||Physicians, n (%)|
|Nurses, n (%)|
|Lab and pharmacists, n (%)|
|HMIS staff, n (%)|
|Find patient with certain characteristics||5 |
|Create notes (history and physical exam)||45 |
|Enter order (lab, radiology)||42 |
|Review/obtain lab and radiology results||4 |
|Update diagnosis||42 |
|Review currently received medications||6 |
|Write prescriptions||43 |
|Admit a patient||6 |
|Refer a patient||9 |
|View/schedule appointment for a patient||5 |
|Communication using SmartCare\'s communication||10 |
|Produce patient summary reports||10 |
aUnaware of the function (U)
bRarely used the function (R)
cFrequently used the function (F)
EMR Satisfaction and Expectation for Future Benefit
Among the participants, 64.4% (199/309) responded to be dissatisfied with the use of the implemented EMR system. Of those dissatisfied, 24.6% (49/199) were physicians and 52,7% (105/199) were nurses. The participants responded with a strong disagreement towards the statements “The system helps me finish my task faster” (median=5, IQR=1) and “The system has a positive effect on quality of care” (median=5, IQR=1). Of all the professionals, 67.9% (210/309) preferred the paper-based record to the EMR system. Overall, the median satisfaction level was at the range of “Disagree” (median =4, IQR=1). The overall median responses with IQRs and percentages are shown in.
|Characteristics of EMR Satisfaction||Physicians||Nurses||Laboratory and pharmacy||HMIS|
|n (% DAa)|
|mean (IQRb)||n (% DA)|
|mean (IQR)||n (% DA)|
|mean (IQR)||n (% DA)|
|SmartCare help me to finish my work faster||49 (85.9)||5 (1)||89 (65.4)||4 (2)||46 (88.4)||4 (0)||2 (3.1)||2 (1)|
|EMR Improves my productivity||47 (82.4)||4 (0)||52 (38.2)||3 (1)||41 (78.8)||4 (0)||1 (1.5)||3 (1)|
|I prefer the EMR than the paper record||13 (22.8)||4 (1)||50 (36.7)||3 (2)||7 (13.4)||2 (1)||0 (0.0)||2 (2)|
|System has positive impact on quality of care||48 (84.2)||5 (1)||107 (78.6)||4 (0)||42 (80.7)||5 (1)||2(3.1)||2 (1)|
|Overall, I am satisfied with the EMR system||49 (85.9)||5 (1)||105 (77.2)||4 (0)||44 (84.6)||5 (0)||2 (3.1)||2 (1)|
|Category median score (95% CI), median (IQR)||5 (1)||4 (1)||4 (2)||2 (1)|
|Over all median score (95% CI), median (IQR)||4.0 (1.0)|
bInterquartile range (IQR)
Perceived System, Information, and Service Quality
The respondents indicated an agreement with the statement that the implemented EMR system had an acceptable quality with an overall median score in the range of “agree” (median=2, IQR=0.5). Of the health professionals, 77.6% (240/309) found the system easy to learn, 61.1% (189/309) user friendly, 58.2% (180/309) stable, and 55.9% (173/309) found the response time acceptable. Overall, HMIS staff perceived the system to have more quality when compared to the other professional categories (median=2, IQR=1). All the criteria to measure system quality (1-5), information quality (6-12), and service quality (13-21) with percentages are shown in.
The participants of this study also agreed with the statement that the information quality of the implemented system was acceptable with an overall median range of “agree” (median=2, IQR=1.0) with more acceptance within HMIS staff (median=1, IQR=1.0) and less agreement by physicians (median=4, IQR=2.0). Of all the participants, 93.5% (289/309) found the output of the system useful, and 76.6% (237/309) also found the information on the modules sufficient for their clinical practice. Only 42.3% (131/309) reported that they felt secure when using the system.
More dissatisfaction was reported with the service quality with an overall median score in the range of “disagree” (median=4.5, IQR=1.5). Of the respondents, 56.6% (175/309) believed their immediate supervisors were not helpful in using the EMR system, 71.8% (222/309) thought the IT support staff did not understand their needs, and 61.8% (191/309) believed the training given was not adequate. Additionally, 66.9% (207/309) responded that they could not get a computer in the ward during patient treatment, 66.0% (204/309) were unhappy with the computer technicians support, 73.4% (227/309) were also unhappy with the frequent power interruptions, and among them, 58.2% (180/309) responded that their department was not backed up by the standby generator.
Expectation Towards Future Benefit
Expectations of the respondents about the benefit of the EMR system for the patient, professionals, and the hospital were also assessed (). The majority of the respondents who never use the EMR system (91.7%, 89/97) and the current users (53.6%, 52/97) expect that the EMR system will be beneficial. Of the respondents who used to use the system, 45.3% (140/309) reported that the EMR system will be beneficial to the hospital. An independent sample t test revealed a statistically significant difference between “those who never use the system” and “those who used to use the system” (P<.001), but not significant between “those who used to use the system” and “current users”.
|Characteristic||Those who never use EMR, n (%Aa), n=97||Those who used to use, n (%A), n=309||Current users, n (%A), n=98|
|I expect EMR to benefit patients in the future||89 (91.7)||140 (45.3)||52 (53.0)|
|I expect EMR to benefit staff in the future||80 (82.4)||130 (42.0)||46 (46.9)|
|I expect EMR to benefit the hospital in the future||94 (96.9)||120 (38.9)||50 (51.0)|
Power Interruption Rate in the Study Hospitals
During the study period, the power fluctuation frequency in the study hospitals was monitored for one month. In one of the hospitals, the power supply was too weak to run the computers and the EMR was not functional during the study period. In the other hospitals, the daily hours of the power interruption were recorded, and the average of the 4 hospitals is shown in. Accordingly, the mean time the power was interrupted for was 1.7 hours per day (SD 0.3). Of all the hospitals, 3 had a standby generator, but the generators could only reach the emergency and surgical departments and hence could not support the full running of the EMR in all of the departments.
Determinants of EMR User Satisfaction
In the binary logistic regression analysis, the following were found to be significantly associated with EMR satisfaction (): computer access method in the hospital, IT qualification, EMR use, training, perceived system quality, perceived information quality, and perceived service quality. Respondents who reported to have good IT qualification were 3 times (adjusted OR 3.21, 95% CI 3.05-8.12) more likely to be satisfied with EMR systems when compared with those reported not having IT qualification, and those who had individual computer access were 4 times (adjusted OR 4.10, 95% CI 2.85-21.95) more likely to use the EMR than those who shared the computer with more than 5 people. Respondents who were currently using the system were 8 times more likely to be dissatisfied (adjusted OR 7.99, 95% CI 5.62-9.10), and those who received initial EMR training were 3 times (adjusted OR 3.04, 95% CI 2.05-8.12) more likely to be satisfied with EMR systems. The respondents who perceived the system to be of good quality were 2 times (adjusted OR 2.2 95% CI 1.34-3.09) more likely to be satisfied with the EMR, while those who perceived the information to be of good quality (adjusted OR 1.94, 95% CI 1.12-3.23) and those who perceived service to be of good quality (adjusted OR 8.23, 95% CI 3.23-17.01) were 2 and 8 times more likely to be satisfied, respectively. The result of respondent characteristics and its associated factors are shown in .
|Characteristics||EMR satisfaction, n (%)||OR (95% CI)||AORa (95% CI)|
|Computer access in hospital|
|Individual||18 (21.6)||65 (78.3)||17.77 (12.62-26.42)||4.10 (2.85-21.95)|
|2 practitioners||27 (57.4)||20 (42.5)||11.85 (2.53-55.34)||2.91 (1.94-6.13)|
|3 practitioners||51 (75.0)||17 (25.0)||5.33 (1.15-24.64)||1.5 (0.10-2.25)|
|4 practitioners||66 (91.6)||6 (8.4)||1.45 (0.27-7.61)||1.11 (0.18-2.37)|
|>5 practitioners||32 (94.1)||2 (5.9)||1.0||1.0|
|None||70 (86.4)||11 (13.6)||1.0||1.0|
|Reasonable||77 (60.1)||51 (39.9)||4.21 (2.03-8.72)||2.11 (1.58-8.77)|
|Good||52 (52.0)||48 (48.0)||5.87 (2.78-12.39)||3.21 (3.05-8.12)|
|Yes||109 (53.1)||96 (46.9)||8.01(18.56-27.78)||3.04(2.31-7.34)|
|No||89 (86.4)||14 (13.6)||1.0||1.0|
|Current SmartCare use|
|Yes||12 (12.2)||86 (87.8)||11.8 (6.68-26.86)||7.89 (3.62-9.10)|
|No||187 (88.6)||24 (11.4)||1.0||1.0|
|Perceived system quality|
|Good||160 (68.9)||72 (31.1)||3.21 (1.34-4.23)||2.2 (1.34-3.09)|
|Not good||23 (29.8)||54 (69.2)||1.0||1.0|
|Perceived information quality|
|Good||140 (65.1)||75 (34.9)||2.8 (1.23-3.78)||1.94 (1.12-3.23)|
|Not good||32 (34.0)||62 (66.0)||1.0||1.0|
|Perceived system quality|
|Good||73 (90.1)||8 (9.8)||9.34 (4.23-18.34)||8.23 (3.23-17.01)|
There was one unexpected observation we want to point out. There was no dedicated information communication technology (ICT) support center in all of the hospitals despite the implementation of such an expensive server and network infrastructure. Of all 5 hospitals, 3 did not even have any professional IT technical support. The other 2 hospitals did have professional IT support; however, they were not primarily responsible for the EMR. The country is indeed training health information technicians at both the diploma and masters level to manage such systems but there were no health information technicians hired in all of the hospitals. Additionally, we observed that the technical support from TUTAPE was not sufficient during the study duration. The technical support team usually took 2-3 days to visit the hospital and solve the problem.
The purpose of the study was to assess the use and the determinant factors of user satisfaction in an implemented EMR system through a comprehensive assessment of usage patterns, user satisfaction, and determinant factors which affect the EMR system. This study had four main results.
First, the usage of the system was found to be low. To increase the use of the system, most of the physicians expressed the need to hire secretaries as the nurses expressed a lack of time to input information and the proper maintenance of computers, and the laboratory and pharmacy staff complained about the lack of use of the system by other departments. This result is not actually surprising given that health professionals are expected to do dual documentation both on the computer and on paper which makes them feel that transferring data to the EMR is not their duty. Hence, most of the doctors and the nurses were complaining about the lack of time and most of them demanded secretaries to be hired so that the secretary can transfer the paper documentation to the EMR. The other aspect of the challenge is the partial use of the system in the hospital departments. Hospital work flows are interconnected, in which the activity of one department affects the other. Therefore, there is a need to implement the system to interdependent departments especially to those that are pillars of the hospital system (eg, laboratory, pharmacy, and radiology).
Second, the result of this study shows that only 2 of the core EMR functions were frequently used, 3 of the functionalities were never used, and participants were also unaware of 4 out of 12 core EMR functions. The participants enrolled in this study were all employed for >6 months. Thus, they were likely to be familiar with the main system functionalities. The low rate of use and awareness might be attributed to the general low adaptation rate of the EMR in the hospital and the training quality.
Third, the user satisfaction of the respondents was also found to be low. The majority of them reported to be dissatisfied with the use of the system. The main reported reason of the dissatisfaction was the service quality in the hospital. This was mainly due to lack of IT support, the shared computer access, and frequent power interruption.
Fourth, the user satisfaction was strongly correlated with service quality and system use. It was also moderately correlated with IT qualification, computer access method, perceived system, and information quality. Given the infrastructural and organizational challenges, such a strong correlation between service quality and use and user satisfaction was expected. However, the level of strength of the relationship was high, which shows that there was a need to improve the service quality and the current way of using the EMR in the hospitals.
Study Strengths and Weaknesses
Totally, 406 professionals (96.2% response rate) from 5 hospitals participated in this study. The response rate was very high when compared to other evaluation studies, which can be attributed to the use of data collectors from each hospital and our encouragement for participation by providing rewards. We addressed different potential system users by including physicians, nurses, laboratory, pharmacy, and HMIS staff.
There are some limitations to this study. Firstly, our data collection period was short (1 month). As pointed out by Meijden , system implementation is dynamic and the level of use rate fluctuates over time. Hence, this result may not exactly reflect the current status of the EMR implementation in those hospitals. The second limitation is with respect to the hypothesized determinant factors. This study is only based on the six constructs of the D&M model but there are many other organizational and human factors which affect acceptance and thus implementation success of an EMR system. Future studies can include those additional variables to have a complete picture of EMR success in those settings.
Results in Relation to Other Studies
There are different evaluation studies regarding the use and user satisfaction of health professionals with respect to EMR systems and different adoption rates were reported. For example, Laerum et al  reviewed EMR system use in 19 hospitals and reported that system use was low and only 2 out of 7 implemented functions were frequently used, which is in line with our results. Another similar study in Saudi Arabia [ ] also reported that the use of different core EMR components were minimal. In that study, 54.9% of the physicians never used ≥1 of the 10 investigated core EMR functions. Mikkelsen et al also assessed the challenges of parallel documentation (EMR and paper-based records) and found that it is a source of dissatisfaction and inconsistency, which is similar to our study [ ].
Alharthi et al  assessed physicians’ satisfaction of an EMR system and reported only 40% of them were satisfied. A similar low satisfaction rate was reported in Malaysia [ ], Oman [ ], and Kenya [ ] which all is similar to our result. However, a recent study by Jia-lin [ ] in two big hospitals in China reported a satisfaction rate of 70.7%. This difference might be because of the infrastructural differences in the study setting hospitals. Another study by Palm et al [ ] reported that medical secretaries were more satisfied than nurses and physicians and Moody et al [ ] and others [ , , ] reported that nurses were more satisfied than physicians with the use of EMR, which is similar to our result.
Common in most studies and in our study are the factors that affect the success of implementation of an EMR system. Palm et al , in his assessment of determinants of user satisfaction of EMR systems, reported that female gender, perceived system quality, usefulness, and service quality are strongly correlated with satisfaction. Similarly, another study by Chatzoglou et al [ ] reported that user background, information quality, and service quality directly and positively affect user satisfaction. Consistent with many evaluation studies and models [ , , , , - ], system quality, information quality, and service quality are determinant factors for user satisfaction. However, in our study we found out that there was a strong correlation between user satisfaction and system quality. This difference might be due to the infrastructural challenge in our study hospitals in which there were frequent power interruptions and no dedicated ICT support centers.
We agree with Meijden  that the D&M’s conceptual framework does not address different contingent factors for the success of an EMR system. In our study, user IT qualification and computer access methods were found to be significant determining factors but we were not able to accommodate them in the framework. These are also reported as determinant factors in other studies [ , ], but computer access method was a significant determinant factor in our study. We believe that this factor is significant for low-resource settings, given that most clinicians (43% of the physicians and 31% of nurses) shared one computer for ≥4 people. .
Meaning and Generalizability
Even though many hospitals implemented an EMR system in developing countries, very few evaluation studies exist on use, user satisfaction, and factors affecting it. In this study we attempted to close this gap by assessing use and user satisfaction in low-resource setting hospitals, and we believe that the result will be helpful to health care managers and decision makers as an input for future EMR implementation or expansion projects. The ministry of health of Ethiopia plans to expand the EMR to all other hospitals, and we are hopeful that this study will help them as an input. As outlined above, our result shows that more emphasis must be given to service quality in implementing the EMR system to the other hospitals. Since our study includes both teaching and general hospitals as well different professionals in the hospitals, we believe that our findings are generalizable to other similar setting hospitals in developing countries.
Unanswered and New Questions
The informatics community perceives the D&M model as the best and most validated model to measure the success of an implemented information system [, , - ]. However, as also stated by Meijden [ ], the D&M’s conceptual framework does not address all information system success factors. In our study results, we were unable to categorize the computer skill and experiences into the model. Hence, a more comprehensive model, which takes into account the different factors in low-resources setting hospitals, is necessary.
The other necessary but unaddressed aspect of the evaluation is the financial feasibility of the system. In the study hospitals, the system was implemented by external NGO-funding that provided a higher budget for EMR implementation than governmental funding. Therefore, we recommend the Ministry of Health to conduct more comprehensive, cost-benefit analyses which include a qualitative evaluation on the system before implementing the system to the other hospitals.
Health professionals’ use of the EMR is low and they are generally dissatisfied with the service of the implemented system. The result of this study showed that the dissatisfaction was caused mainly and strongly by the poor service quality, the current practice of double documentation (EMR and paper-based), and partial departmental use of the system in the hospitals. Thus, future interventions to improve the current use or future deployment projects should focus on improving the service quality, such as power infrastructure, user support, trainings, and more computers in the wards. After service quality improvement, other departments (especially inter-dependent departments) may be motivated and supported to use the EMR to avoid the dependency deadlock. Further evaluation studies that include a cost-benefit analysis are recommended.
We would like to acknowledge Professor Martin Dugas for his valuable comments on the study design and final manuscript. We acknowledge support by Deutsche Forschungsgemeinschaft and Open Access Publication Fund of University of Muenster for supporting the final publication cost of this paper. We also would like to acknowledge anonymous reviewers, the data collectors, and all participants of this study.
Conflicts of Interest
Binyam Tilahun initiated the study, developed the study design, collected and analysed the data and wrote the manuscript. Fleur Fritz contributed to the study design and critically revising all versions of the manuscript. All authors read and approved the final manuscript.
- Otieno GO, Hinako T, Motohiro A, Daisuke K, Keiko N. Measuring effectiveness of electronic medical records systems: towards building a composite index for benchmarking hospitals. Int J Med Inform 2008 Oct;77(10):657-669. [CrossRef] [Medline]
- Garcia-Smith D, Effken JA. Development and initial evaluation of the Clinical Information Systems Success Model (CISSM). Int J Med Inform 2013 Jun;82(6):539-552. [CrossRef] [Medline]
- Van Der Meijden MJ, Tange HJ, Troost J, Hasman A. Determinants of success of inpatient clinical information systems: a literature review. J Am Med Inform Assoc 2003;10(3):235-243 [FREE Full text] [CrossRef] [Medline]
- DeLone W, MacLean E. The DeLone and McLean Model of Information Systems Success: A Ten-Year Update. J Manag Inf Syst 2003;19(4):9-30.
- Palm JM, Colombet I, Sicotte C, Degoulet P. Determinants of user satisfaction with a Clinical Information System. AMIA Annu Symp Proc 2006:614-618 [FREE Full text] [Medline]
- Likourezos A, Chalfin DB, Murphy DG, Sommer B, Darcy K, Davidson SJ. Physician and nurse satisfaction with an Electronic Medical Record system. J Emerg Med 2004 Nov;27(4):419-424. [CrossRef] [Medline]
- Ives B, Olson M, Baroudi J. Commun ACM. The measurement of user information satisfaction URL: https://archive.nyu.edu/bitstream/2451/14594/1/IS-82-27.pdf [accessed 2015-04-30] [WebCite Cache]
- Mazzoleni MC, Baiardi P, Giorgi I, Franchi G, Marconi R, Cortesi M. Assessing users' satisfaction through perception of usefulness and ease of use in the daily interaction with a hospital information system. Proc AMIA Annu Fall Symp 1996:752-756 [FREE Full text] [Medline]
- Nour El Din MM. Physicians' use of and attitudes toward electronic medical record system implemented at a teaching hospital in saudi arabia. J Egypt Public Health Assoc 2007;82(5-6):347-364. [Medline]
- Chisolm DJ, Purnell TS, Cohen DM, McAlearney AS. Clinician perceptions of an electronic medical record during the first year of implementaton in emergency services. Pediatr Emerg Care 2010 Feb;26(2):107-110 [FREE Full text] [CrossRef] [Medline]
- Paré G, Sicotte C, Jaana M, Girouard D. Prioritizing the risk factors influencing the success of clinical information system projects. A Delphi study in Canada. Methods Inf Med 2008;47(3):251-259. [Medline]
- Adam Mahmood M, Burn JM, Gemoets LA, Jacquez C. Variables affecting information technology end-user satisfaction: a meta-analysis of the empirical literature. International Journal of Human-Computer Studies 2000 Apr;52(4):751-771. [CrossRef]
- Msukwa MKB. User Perceptions on Electronic Medical Record System (EMR) in Malawi. 2011. URL: http://www.medcol.mw/commhealth/mph/dissertations/martin%20msukwa_Approved.pdf [accessed 2015-05-02] [WebCite Cache]
- Sood S, Nwabueze S, QMbarika V, Prakash N, Chatterjee S, Ray P, et al. Electronic Medical Records: A Review Comparing the Challenges in Developed and Developing Countries. 2008 Presented at: Proceedings of the 41st Hawaii International Conference on System Sciences -; 2008; Hawaii Int Conf p. 1-10 URL: http://www.computer.org/csdl/proceedings/hicss/2008/3075/00/30750248.pdf
- Verbeke F, Ndabaniwe E, Van Bastelaere S, Ly O, Nyssen M. Evaluating the Impact of Hospital Information Systems on the Technical Efficiency of 8 Central African Hospitals Using Data Envelopment Analysis. Journal of Health Informatics in Africa 2013;1(1).
- Federal Ministry of Health Health Management Information System (HMIS) / MonitoringEvaluation (M & E) Strategic Plan for Ethiopian Health Sector. Addis ababa; 2008 URL: http://phe-ethiopia.org/resadmin/uploads/attachment-58-Health_Managment_Information_System_%28HMIS%29.pdf [accessed 2015-05-02] [WebCite Cache]
- Hirpa W, Tesfaye H, Nigussie F, Argaw H. Q Heal Bull. Implementation Of An Integrated Health Management Information System And Monitoring And Evaluation System In Ethiopia: Progress And Lessons From Pioneering Regions URL: http://www.who.int/healthmetrics/library/countries/ETH_HIS_LessonsLearned.pdf [accessed 2015-05-02] [WebCite Cache]
- Mengesha T. Electronic solutions for Ethiopian health sector. 2011. URL: https://www.theseus.fi/bitstream/handle/10024/36264/Mengesha_Tewodros.pdf?sequence=1 [accessed 2015-05-02] [WebCite Cache]
- Biruk S, Yilma T, Andualem M, Tilahun B. Health Professionals' readiness to implement electronic medical record system at three hospitals in Ethiopia: a cross sectional study. BMC Med Inform Decis Mak 2014;14:115 [FREE Full text] [CrossRef] [Medline]
- Nguyen L, Bellucci E, Nguyen LT. Electronic health records implementation: an evaluation of information system impact and contingency factors. Int J Med Inform 2014 Nov;83(11):779-796. [CrossRef] [Medline]
- Fraser HSF, Blaya J. Implementing medical information systems in developing countries, what works and what doesn't. AMIA Annu Symp Proc 2010;2010:232-236 [FREE Full text] [Medline]
- Nykänen P, Brender J, Talmon J, de Keizer N, Rigby M, Beuscart-Zephir MC, et al. Guideline for good evaluation practice in health informatics (GEP-HI). Int J Med Inform 2011 Dec;80(12):815-827. [CrossRef] [Medline]
- Talmon J, Ammenwerth E, Brender J, de Keizer N, Nykänen P, Rigby M. STARE-HI--Statement on reporting of evaluation studies in Health Informatics. Int J Med Inform 2009 Jan;78(1):1-9. [CrossRef] [Medline]
- Vital Wave Consulting. Health Information Systems in Developing Countries URL: http://www.minsa.gob.pe/ogei/conferenciaops/Recursos/43.pdf [accessed 2015-05-02] [WebCite Cache]
- Mengesha T. Electronic Solutions for Ethiopian Health Sector. 2011. URL: https://www.theseus.fi/bitstream/handle/10024/36264/Mengesha_Tewodros.pdf?sequence=1 [accessed 2015-05-18] [WebCite Cache]
- Tilahun B, Zeleke A, Fritz F, Zegeye D. New bachelors degree program in health informatics in Ethiopia: curriculum content and development approaches. Stud Health Technol Inform 2014;205:798-802. [Medline]
- Chatzoglou PD, Fragidis LL, Doumpa T, Aggelidis P. Hospital Information System Evaluation. In: Hospital Information System Evaluation.: 10th International Conference on ICT in Health; 2012 Presented at: 10th International conference on Information and Communication technologies for health; Jully 12-14, 2012; Island, Grek p. 12-14.
- Chen RF, Hsiao JL. An investigation on physicians' acceptance of hospital information systems: a case study. Int J Med Inform 2012 Dec;81(12):810-820. [CrossRef] [Medline]
- Bokhari RH. The relationship between system usage and user satisfaction: a meta‐analysis. Journal of Ent Info Management 2005 Apr;18(2):211-234. [CrossRef]
- Lawrence M, Low G. Exploring Individual User Satisfaction within User-Led Development. MIS Quarterly 1993 Jun;17(2):195-208. [CrossRef]
- Igbaria M, Nachman SA. Correlates of user satisfaction with end user computing. Information & Management 1990 Sep;19(2):73-82. [CrossRef]
- Seddon PB, Kiew M, Agency R. A partial test and development of delone and mclean’s model of is success. Australia: Australian Journal of Information systems; 1995:90-109.
- Doll BWJ, Torkzadeh G. The Measurement of End-User Computing Satisfaction. MIS Quarterly 1988 Jun;12(2):259-274. [CrossRef]
- Laerum H, Ellingsen G, Faxvaag A. Doctors' use of electronic medical records systems in hospitals: cross sectional survey. BMJ 2001 Dec 8;323(7325):1344-1348 [FREE Full text] [Medline]
- Mikkelsen G, Aasly J. Concordance of information in parallel electronic and paper based patient records. Int J Med Inform 2001 Oct;63(3):123-131. [Medline]
- Alharthi H, Youssef A, Radwan S, Al-Muallim S, Zainab A. Physician satisfaction with electronic medical records in a major Saudi Government hospital. Journal of Taibah University Medical Sciences 2014 Sep;9(3):213-218. [CrossRef]
- Mohd Amin I, Sumarni Hussein S, Isa WARWM. Assessing User Satisfaction of using Hospital Information System ( HIS ) in Malaysia. 2011 Presented at: International Conference on Social Science and Humanity; 2011; Singapore p. 210-213 URL: http://www.ipedr.com/vol5/no1/45-H00097.pdf
- Al Farsi M, West DJ. Use of electronic medical records in Oman and physician satisfaction. J Med Syst 2006 Feb;30(1):17-22. [Medline]
- Kipturgo MK, Kivuti-Bitok LW, Karani AK, Muiva MM. Attitudes of nursing staff towards computerisation: a case of two hospitals in Nairobi, Kenya. BMC Med Inform Decis Mak 2014;14:35 [FREE Full text] [CrossRef] [Medline]
- Jia-lin L, Siru L, Fei L. Physician satisfaction with electronic medical record in a huge hospital (China). Stud Health Technol Inform 2013;192:920. [Medline]
- Moody LE, Slocumb E, Berg B, Jackson D. Electronic health records documentation in nursing: nurses' perceptions, attitudes, and preferences. Comput Inform Nurs 2004 Dec;22(6):337-344. [Medline]
- Bossen C, Jensen LG, Udsen FW. Evaluation of a comprehensive EHR based on the DeLone and McLean model for IS success: approach, results, and success factors. Int J Med Inform 2013 Oct;82(10):940-953. [CrossRef] [Medline]
- Zaied ANH. An Integrated Success Model for Evaluating Information System in Public Sectors. Journal of Emerging Trends in Computing and Information Sciences 2012;3(6):814-825 [FREE Full text]
- Cheung CS, Tong EL, Cheung NT, Chan WM, Wang HH, Kwan MW, et al. Factors associated with adoption of the electronic health record system among primary care physicians. JMIR Med Inform 2013;1(1):e1 [FREE Full text] [CrossRef] [Medline]
- Alquraini H, Alhashem AM, Shah MA, Chowdhury RI. Factors influencing nurses' attitudes towards the use of computerized health information systems in Kuwaiti hospitals. J Adv Nurs 2007 Feb;57(4):375-381. [CrossRef] [Medline]
- Chin WW, Lee MKO. A proposed model and measurement instrument for the formation of IS satisfaction: the case of end-user computing satisfaction. : Twenty-First International Conference on Information Systems; 2000 Presented at: Proceedings of the twenty first international conference on Information systems; 2000; Atlanta,USA p. 553-563 URL: https://www.researchgate.net/profile/Wynne_Chin/publication/220268832_A_proposed_model_and_measurement_instrument_for_the_formation_of_IS_satisfaction_the_case_of_end-user_computing_satisfaction/links/09e4151492510e5cdb000000.pdf
- Xiao L, Dasgupta S. Measurement of user satisfaction with web-based information systems: an empirical study. : Eighth Americas Conference on Information Systems; 2002 Presented at: Eighth Americas Conference on Information Systems; 2002; USA p. 1149-1155 URL: http://www.sighci.org/amcis02/CR/Xiao.pdf
- Petter S, DeLone W, McLean E. Measuring information systems success: models, dimensions, measures, and interrelationships. Eur J Inf Syst 2008 Jun;17(3):236-263. [CrossRef]
- Perezmira B. PhD Dissertation.: Louisiana State University and Agricultural and Mechanical College; 2010. Validity of Delone and McClean's Model of Information Systems Success at the Web Site Level of Analysis URL: http://etd.lsu.edu/docs/available/etd-04162010-001906/unrestricted/Perez-Mira_diss.pdf [accessed 2015-05-02] [WebCite Cache]
|D&M: DeLone and MacLean|
|EMR: Electronic medical record|
|HMIS: Health management information system|
|ICT: Information communication technology|
|IQR: Interquartile range|
|IRB: Institute Review Board|
|IT: Information technology|
|NGO: Non governmental organization|
Edited by G Eysenbach; submitted 04.12.14; peer-reviewed by M Wong, F Verbeke; comments to author 05.02.15; revised version received 12.02.15; accepted 17.02.15; published 25.05.15
©Binyam Tilahun, Fleur Fritz. Originally published in JMIR Medical Informatics (http://medinform.jmir.org), 25.05.2015.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on http://medinform.jmir.org/, as well as this copyright and license information must be included.