User Perceptions and Use of an Enhanced Electronic Health Record in Rwanda With and Without Clinical Alerts: Cross-sectional Survey

Introduction

Background

Effective and high-quality health care requires high-quality, timely health information—“Information is care” [1]. Scaling up effective care for millions of patients with HIV in resource-limited settings such as sub-Saharan Africa required the development of new paradigms for the collection, storage, viewing, and analysis of clinical data and health information [2]. Most health centers treating HIV started with only structured paper records. As the volume of patient data grew and in-country digital capacity improved, electronic tools were introduced. Many early electronic health records (EHRs) in resource-limited settings have been developed for HIV care, including those in Malawi [3], Kenya [4], and Haiti [5]. These projects demonstrated the feasibility of deploying health information systems, improvements in reporting to ministries of health (MoHs) and donors, and the ability to monitor the continuum of care. Furthermore, this initial evidence also suggested that the use of EHR systems for HIV, tuberculosis (TB) and multidrug-resistant TB treatment could improve the quality of care [2]. A critical challenge to improving the quality of care in low-income settings is the ability to achieve long-term, consistent EHR use at a large scale. To better understand the perceptions and clinical uses of EHR systems that support improved use and care in Rwanda, we conducted a quantitative user survey supplemented by free-text questions. For the purposes of this study, the terms electronic health record and electronic medical record are used interchangeably.

HIV Care in Rwanda

Rwanda is an East Central African country bordering Tanzania, Uganda, Burundi, and the Democratic Republic of the Congo. Rwanda had a per capita income of US $773 in 2018, up from US $241 in 2004 [6], and has made great progress in rebuilding its health care systems after the genocide against Tutsi in 1994. A major health challenge Rwanda has faced, along with neighboring countries in Africa, is the HIV epidemic. A 2018 to 2019 survey indicated that HIV prevalence among adults aged 15 to 49 years was 2.6% [7]. Great strides have been made in the treatment of patients who are HIV positive, including improvements in the prevention of mother-to-child transmission uptake, and reduction in the rate of loss to follow-up for patients receiving antiretroviral therapy (ART) in Rwanda. This is demonstrated by the near achievement in 2019 of the 2020 Joint United Nations Programme on HIV/AIDS 90-90-90 goal, with 84% of adults who were HIV positive knowing their status, 98% of those knowing their status on ART, and 90% of those on ART having a suppressed viral load [7]. From the beginning of the HIV treatment scale-up, the Government of Rwanda has emphasized care and prevention in rural areas as well as in urban settings; recruitment, training, and supervision of community health workers; and the use of health information systems. These information systems included national-level surveillance systems for HIV care [8,9], mobile health systems to support antenatal and primary care, and patient information or EHR systems mainly for supporting HIV care in health centers and hospitals. The 3 main EHR systems used have been OpenMRS (OpenMRS Inc) in health centers and 36 district hospitals offering HIV services, IQcare (International Quality Care, Palladium Inc) [10] in some health centers (now replaced by OpenMRS), and OpenClinic (OpenClinic GA) [11] in some hospitals. Since 2009, the MoH has moved to using OpenMRS for all HIV health centers and most hospitals in the country.

OpenMRS

OpenMRS is an open-source software platform for building EHRs, with a focus on health care needs in low- and middle-income countries (LMICs). Founded in 2004, the OpenMRS community set goals to create a public software platform to assist health care organizations worldwide in developing EHR systems that were adaptable to local needs, owned by local organizations, and programmed by local developers as much as possible [12] (Textbox 1).

Initially, OpenMRS was used for HIV and TB treatment in outpatient settings, supporting projects funded by the US President’s Emergency Plan for AIDS Relief and the Global Fund for AIDS, Tuberculosis, and Malaria. Currently, it covers a wide range of clinical areas. Partners In Health implemented and currently supports OpenMRS in 46 health centers and 3 hospitals in Rwanda covering HIV care, pediatrics, primary care, cardiology, and oncology.

Between 2009 and 2013, the Rwanda MoH deployed OpenMRS to >300 health centers providing HIV care throughout the country [13]. Before and during deployment, OpenMRS had to be customized to support the Rwanda MoH requirements. A dedicated 9-month course led by Partners In Health/Inshuti Mu Buzima trained programmers in enterprise Java and health information system design [14]. Several graduates were hired by the MoH and created custom OpenMRS modules for HIV and primary care using OpenMRS version 1.6 core code. This is the version of OpenMRS used in the control sites for this study. Unstable internet connectivity in rural Rwanda (similar to many low-income countries) required each site to run its own instance on a local server, requiring stable power and local technical support.

Impact of EHR Systems in Resource-Limited Countries

Over the last two decades, EHR systems have been implemented in a wide range of countries, including those with the lowest income levels. The scale-up of HIV care and transition from an emergency outbreak response to a lifelong chronic care model was a major driver for the expansion of EHR system use and the development of common, shared information system tools. Countries, including Rwanda, Kenya, Uganda, Mozambique, and Nigeria, have scaled up the use of OpenMRS EHR systems for HIV care to hundreds of their clinical sites. Other EHRs, including IQcare, have been widely used in countries such as Kenya [15].

The OpenMRS community has prioritized support for effective and safe clinical care as well as reporting and research. Smaller-scale studies have evaluated the impact of EHR system improvements on the aspects of clinical care, the systems efficacy. Were et al [16] studied the addition of alerts to printed patient summaries generated by OpenMRS on a range of clinical actions for the care of children who were HIV positive in Eldoret, Kenya. In a randomized controlled trial (RCT), they showed that health care workers receiving the summaries with alerts were 4 times more likely to carry out actions such as ordering CD4 counts (a T lymphocytes test) and polymerase chain reaction tests for HIV antigen. In a larger study, Oluoch et al [17] studied the impact of improved decision support tools implemented in an EHR in Kenya on the quality of HIV care. In a cluster RCT of 13 health centers and 41,062 patients, they showed that sites with the decision support tools were quicker and more effective in responding to HIV treatment failure [17]. Critical questions remain regarding the key factors that determine individual EHR use, facilitate scaling up to tens or hundreds of smaller health facilities, support long-term use, and influence the clinical impact of these systems in routine care—the effectiveness of EHRs in LMICs [18].

Methods

Overview

The aims of this study are to evaluate the following questions in a large number of health centers in Rwanda: (1) staff and stakeholder expectations and perceptions of health information system performance; (2) staff and stakeholder expectations and perceptions around effort expended to use health information systems; (3) infrastructural, organizational, and individual conditions that are barriers and facilitators to using such tools (including training and technical support); (4) staff perceptions of technology fatigue; and (5) any differences in the experiences of staff in intervention and control sites and between clinical and technical users.

The EHR Implementation Science Study

The focus of this manuscript is the electronic medical record (EMR) user survey component of a process evaluation, which is part of a larger, 3-part implementation science study on the use of an enhanced EHR to support HIV care in 56 randomly allocated health centers that commenced in July 2018. It included the evaluation of (1) EHR use, performance, and data quality; (2) the clinical impact in an RCT; and (3) the cost of development and implementation of the enhanced EHR functionality.

For enrollment in the overall study, first, the enhanced EHR package (Textbox 2) was piloted in 2 health centers in Kigali (Kicukiro Health Centre and Kagugu Health Centre), and improvements were made in response to the user experience and comments. Next, the following selection criteria were applied: (1) the presence of ≥3 computers, 1 printer, and a local area network; (2) active HIV case numbers between 50 and 700; and (3) successful installation of the Server Monitoring Tool (Textbox 1) and evidence of regular data entry by staff. Using these criteria, a total of 112 sites were selected to participate in the clustered RCT. These sites were a mix of urban and rural health centers and some district hospitals. Of the 112 sites, 56 (50%) were randomized into the intervention sites, which had the enhanced EHR installed on the servers between June 25 and July 5, 2018. All 56 sites had the alerts for delayed patient enrollment, 28 sites also had alerts for delayed viral load testing, and 14 had the alerts for evidence of treatment failure. For the analysis of the survey, sites with at least the top-level alerts (delayed HIV care registration) were classed as intervention. Health facility staff, including clinicians, data managers, local information technology (IT) staff, local clinic managers, and district IT specialists, in all 112 study sites were trained on general EHR use and data management. Additional training was provided for staff in the intervention sites on the enhanced EHR and equivalent training on the control EHR.

Study Environment

The user survey was conducted at primary health care facilities, referred to here as health centers, offering HIV treatment services, located throughout Rwanda, approximately 5 months after the installation of the enhanced EHR.

Study Design

This study used a cross-sectional, key informant structured interview design within control and intervention sites. The data were collected through structured interviews to ensure high response rates and avoid technical limitations that may have impacted a web-based survey and biased results toward better-supported sites and users. The goal is to gain insights into the adoption, functionality, use, and perceptions of EHRs by clinical staff (nurses, physicians, and social workers) and technical staff (IT staff, data entry staff, and data managers) in health centers. Care of patients in smaller health facilities in East Africa, including those with HIV, is mostly carried out by nurses or junior clinician grades and rarely by physicians. The study questions were as follows: (1) whether the actions and perceptions of staff using the enhanced EHR intervention would be different from those using the control EHR and (2) whether clinicians have different experiences with the EHR than technical staff.

Sampling and Sample Size

This study drew from the sample frame of the clustered RCT implementation study. The RCT enrolled 112 health centers from >300 that use the OpenMRS EHR for HIV care. Of the 112 sites, 54 (48.2%) were randomly selected, including 27 (50%) from the enhanced EHR sites (intervention) and 27 (50%) control sites (Figure 1). Randomization was performed with R.

A total of 100 participants were approached for the structured interview, with the goal of 1 clinician (nurse or physician) and 1 data manager at each health center. If not available, other EHR users were recruited if possible.

‎View this figure

Data Collection Tool

The structured interview and observation (survey) tool included sections on demographics, experience with IT, EHR training received, frequency of EHR use for different tasks, overall ease of use, usefulness for specific tasks, technical and user support, and system stability and infrastructure issues. The survey tool included 5-point Likert scale quantitative close-ended questions and qualitative open-ended questions. It was adapted from a form originally used by Médecins Sans Frontières, piloted in 20 clinics in Rwanda in 2012 [13], and translated from English into Kinyarwanda.

Data Collection

The survey was conducted by 10 trained data collectors from the Rwanda School of Public Health. Survey responses were documented and recorded in a preprogrammed Android tablet using ODK [20]. Free-text comments were documented in Kinyarwanda, translated into English, and reviewed by bilingual research team members before analysis. Written informed consent was obtained, and participants’ confidentiality was assured using a private interview room at each health center surveyed.

Data Analysis

Descriptive statistics were carried out using Excel (Microsoft Corp). JMP Statistical Software (SAS Institute) and Excel were used for the chi-square tests for the 5-point Likert scale responses. For the comparison of clinicians and technical users, all Likert scale questions were tested for significance. For the comparison of the intervention and control sites, the 2 groups of questions (12 and 18) most directly related to the technical improvements in the enhanced EHR were tested. P values were adjusted for multiple comparisons using the Benjamini and Hochberg method and R p.adjust [21]. Analyses were designed and carried out with assistance from a statistician and a data scientist at Brown University (see Acknowledgments).

Free-text comments, which were all short statements, were analyzed thematically using inductive coding by one author (HSF) and recoding by a second author (MM), with discrepancies resolved by discussion. Common concepts were described and rated based on the number of user responses matching each code.

Ethical Considerations

This study was approved by the following investigational review boards: Rwanda National Ethics Committee, Kigali (#913/RNAC/2016) and the University of Leeds School of Medicine Research Ethics Committee, Leeds, United Kingdom (MREC16-176). This study was reviewed in accordance with the US Centers for Disease Control and Prevention human research protection procedures (approval #CGH HSR 2014-270a) and determined to be research. However, investigators of the Centers for Disease Control and Prevention did not interact with human participants or have access to identifiable data during this research.

Results

Participant Characteristics

A total of 100 participants were approached for interview and consented to participate. Of these 100 participants, 90 (90% response rate) were available at the time of the visit by study staff, with 44 (49%) from the intervention sites. The participants had a mean age of 35.9 (SD 6.2; range 23-58) years, a mean of 7.5 (SD 2.0; range 0-38) years working at the health centers, and 47% (42/90) were female. Their educational attainment was reported as some secondary schooling (8/90, 9%), completed secondary schooling (18/90 20%), and postsecondary schooling (64/90, 71%). Their occupations were nurse (41/90, 46%), physician (1/90, 1%), social worker (2/90, 2%), data manager (42/90, 47%), IT officer (1/90, 1%), and data entry staff (3/90, 3%). Respondents had a mean of 3.3 (SD 2.0) years of experience with the EHR and a mean of 1.9 (SD 1.4) trainings and 9.7 (SD 7.9) training days.

General Use of Technology

A large majority of respondents used mobile phones, with 82% (74/90) using them “about half the time,” “most of the time,” or “all of the time” for texting, and 82% (74/90) similarly for mobile data. Computer use outside of work was reported by 31% (28/90) and internet use by 49% (44/90). Clinicians (physicians, nurses, and social workers) reported significantly less use than technical staff (IT officers, data managers, and data entry staff; P=.009 and P=.04, respectively).

Training on EHR

Respondents agreed or strongly agreed that their training on the EHR was effective (82/84, 98%), and they were confident in using the EHR (81/87, 93%). However, 77% (66/86) of respondents disagreed or strongly disagreed with the statement “I am generally not concerned making errors in EHR.” There were no statistically significant differences in responses on training between clinicians and technical staff. However, in free-text comments, 81% (73/90) of respondents requested more training. These requests included refreshers, training on new modules or updates, and more practical hands-on training. There were also requests for training in reports and data analysis. Mentorship, supportive supervision, or more technical backup were requested by many respondents.

Use of EHR Functions

Tables 1-7 show and summarize the results for the following question: “Please indicate how often you use the EMR to assist you with the following tasks.” Combining the categories most of the occasions and always/almost always, the percentages for common tasks were 56% (49/88) for creating new patient records, 63% (56/89) for updating existing patient records, 40% (36/89) for generating patient summaries before visits, 48% (43/89) for reviewing previous patient encounters, 30% (21/69) for ordering laboratory analyses, 43% (36/83) for viewing laboratory results, 33% (25/75) for following test results over time, 45% (38/85) for generating automatic reports, 45% (38/85) for generating ad hoc reports (eg, quarterly or TracNET reports), and 49% (41/84) for referring patients to another health facility. The results were 22% (18/82) for generating consult sheets and 16% (14/85) for generating clinician summaries.

Staff in intervention sites were significantly more likely to use the EHR for “Updating existing patient records” (P=.04), “Generating patient summaries before visits” (P<.001), “Viewing laboratory results” (P=.04), and “Generating clinician summaries” (ie, on-screen summaries; P=.01). Clinician responses indicated that they carried out the following tasks significantly less frequently than technical staff: “Creating new patient records” (P=.02) and “Updating existing patient records” (P=.04).

A total of 42% (37/89) of respondents stated that they used the EHR always/almost always or most of the time, as opposed to 76% (68/89) for the paper records. They agreed or strongly agreed >85% (71/84) of the time (Tables 1-7) with the following statements about the EMR: “I can find where to document care,” “The screen displays are easy to read,” “Content lay out is understandable,” and “It is easy to retrieve records in EHR.” For the statement “It is easy to correct errors in EHR,” agreement was 72% (64/89).

Respondents agreed or strongly agreed >90% (81/90) of the time that “the EHR makes it easier to manage patients’ medical file and patient’s medical follow up,” “the EHR makes it easier to make informed decisions,” “the EHR is worth the time and energy to use,” and “quality of information has improved due to the EHR.” For the statement “the EHR makes it easier to exchange patient information with other health care providers,” agreement was 71% (64/90). For the statement “the EHR provides useful alerts and reminders,” agreement was 78% (64/82) with significantly stronger agreement in the intervention sites (P=.01).

Answers to questions on technical and user support received mixed responses. Respondents agreed or strongly agreed with these questions with the following scores: “It is easy to report problems with the EHR,” 71% (63/89); “I get feedback when I report errors or problems,” 60% (53/89); “Effective help is available with the EHR,” 55% (49/89); “I use EHR because of the proportion of coworkers who use it,” 41% (35/86); “My supervisor is very supportive of EHR use on the job,” 75% (67/89); and “In general, the MOH supported the use of EHR,” 93% (84/90).

Infrastructure

Infrastructure problems were a significant issue (Tables 1-7). The following were stated to be available always/almost always or most of the occasions: a computer when you need the EHR (77/89, 87%), grid power (74/90, 82%), wired internet connectivity (56/89, 63%), cellular internet (41/85, 48%), and a backup generator (30/88, 34%). For the question “How often can you count on EHR to be up and available?” response was 75% (66/88), with 18% (16/88) saying it was available about half the time and 7% (6/88) almost never.

Table 8 shows the analysis of free-text comments. The most frequent responses to the question “What are three functions you like about the electronic medical record?” were “to get client data easily and/or quickly” (62/90, 69%), “it helps to generate reliable reports in a short time” (39/90, 43%), “it stores client information safely and/or securely” (31/90, 34%), and “it helps to monitor clients on a daily basis” (20/90, 22%). In response to the question “What are three functions you do not like about the electronic medical record?” most frequent comments were “often unstable or blocked” (20/90, 22%), “hard to correct errors or unsubscribe patients” (11/90, 12%), “cannot work with OpenMRS outside the health facility/not online” (9/90, 10%), and “poor internet” (6/90, 7%).

Discussion

Principal Findings

Overall, the results suggest that most users of OpenMRS at Rwanda MoH health centers perceive the EHR as a valuable tool for patient care and reporting activities. The responses showed a high level of EHR use and acceptability across most health centers despite the challenges of implementing EHR systems in these environments. This finding provides foundational evidence to implementers who have an urgent need to understand how well EHRs can be scaled up to hundreds or thousands of health facilities (addressing objectives on performance and scalability). An unusual feature of the Rwanda OpenMRS implementation is the long interval since the original deployment. Some MoH health centers have used the EHR continuously for 8 or 9 years, with no major upgrades in control sites for >5 years. Therefore, this study allows assessment of the long-term performance of the EHR by typical users (objective on sustainability). Such data are not available from any existing studies that we are aware of, which have mostly focused on larger hospitals in more controlled settings with better infrastructure [11,22] or small numbers of test sites.

Responses to the 2 groups of questions most relevant to the features of the enhanced EHR package showed, in the intervention sites, more frequent use of core clinical tools, including updating records, using patient summaries, and viewing laboratory results. Significantly more respondents in the intervention sites agreed that “The electronic medical record provides useful alerts and reminders,” indicating support for more advanced EHR features added in the enhanced EHR.

There were some differences in the level of EHR use between clinicians and technical staff, including core clinical activities such as creating and updating records. Clinicians, as expected, had less technical experience and were significantly less likely to use computers outside work or access the internet for a range of applications. These findings indicate the need for further improvements in usability and workflow and in both IT and EHR training for clinicians.

It is important to note that recent versions of OpenMRS have greatly improved user interfaces and general functionality [23,24] and are expected to have significantly higher scores for usability and overall satisfaction. An up-to-date version of OpenMRS was implemented in Rwandan district hospitals in 2020/2021.

Limitations

This survey was conducted through structured interviews with all participants. The less confidential nature of interviews compared with a web-based survey may have increased desirability bias as staff were aware that the study was endorsed by the MoH. There was a strong positive response on the question of MoH support and on statements that the effort to enter data and use the EHR was worthwhile. However, on other questions such as infrastructure, including power and internet connectivity, and availability of technical support, participants were more mixed in their responses, and for the question “I am generally not concerned making errors in EHR” they were clearly prepared to admit that there were problems. Many made clear that they had challenges with using the EHR, and clinicians would appear to rely on data managers and other technical staff to assist with many activities. Free-text comments provided critical insights into the actual experiences of staff, along with many other issues related to usability, use, and the need for training. The lack of significant differences in the experiences of the clinicians and technical staff regarding many questions may be partly due to the survey not being powered to show small differences between these groups. Another limitation was that the 112 sites selected for the broader study had better hardware and evidence of more consistent data entry than the others; therefore, EMR implementations described here may perform better than the full set of EMR sites in Rwanda.

Comparison With Previous Work

Previous studies of EHR users in LMICs have identified a range of experiences. Ojo [25] used the Delone McLean Information Success Model in a study of EHR users in hospitals in Nigeria and showed that system quality and use were the most important in determining EHR success [25]. Tilahun and Fritz [26] conducted a similar study on the experience of users with an EHR in hospitals in Ethiopia. Compared with the survey in this study, they showed high levels of dissatisfaction with the EHR and low use levels owing to poor service quality (power infrastructure, user support, training, and lack of computers in the wards) and the need for double entry of data into the EHR and paper records (also a problem in Rwanda) [26]. A survey of the OpenClinic EHR users at the Kigali University Teaching Hospital in Rwanda showed strongly positive user comments on satisfaction and perception of data quality and usability compared with paper records [11].

Conclusions

This survey provides evidence that EHR systems have become an accepted component of HIV care delivery in Rwanda. Staff were generally supportive of the system, although most wanted further training, technical support, and better power and network infrastructure. Staff at intervention sites were more likely to use or have positive experiences of key functionality that was improved in the enhanced EHR. As this survey is part of a larger evaluation study, the responses will be compared with results from key informant interviews, the costing and data quality studies, monitoring of server performance and use, and clinical impact in the cluster RCT. Further surveys are planned for other large-scale rollouts of OpenMRS in low-income settings, building on the survey form and findings in this study. The results are likely to be generalized to similar EHR systems in low-income settings if they are well tailored to the clinical needs and workflow. They are also highly relevant to the critical need for systems to support accurate, timely, and analyzable primary care data on patients in remote and very underserved clinics in low-income countries, replacing basic tools such as paper registers. This should improve the clinical documentation, care, reporting, and tracking of disease outbreaks, including COVID-19.

Data Availability

The data underlying this paper cannot be shared publicly because of the need for privacy of the individuals who participated in the study. The data will be shared upon reasonable request with the corresponding author.