This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on https://medinform.jmir.org/, as well as this copyright and license information must be included.
Weak health information systems (HISs) hobble countries’ abilities to effectively manage and distribute their resources to match the burden of disease. The Capacity Building and Mentorship Program (CBMP) was implemented in select districts of the Amhara region of Ethiopia to improve HIS performance; however, evidence about the effectiveness of the intervention was meager.
This study aimed to determine the effectiveness of routine health information use for evidence-based decision-making among health facility and department heads in the Amhara region, Northwest Ethiopia.
The study was conducted in 10 districts of the Amhara region: five were in the intervention group and five were in the comparison group. We employed a quasi-experimental study design in the form of a pretest-posttest comparison group. Data were collected from June to July 2020 from the heads of departments and facilities in 36 intervention and 43 comparison facilities. The sample size was calculated using the double population formula, and we recruited 172 participants from each group. We applied a difference-in-differences analysis approach to determine the effectiveness of the intervention. Heterogeneity of program effect among subgroups was assessed using a triple differences method (ie, difference-in-difference-in-differences [DIDID] method). Thus, the β coefficients, 95% CIs, and
Data were collected using the endpoint survey from 155 out of 172 (90.1%) participants in the intervention group and 166 out of 172 (96.5%) participants in the comparison group. The average level of information use for the comparison group was 37.3% (95% CI 31.1%-43.6%) at baseline and 43.7% (95% CI 37.9%-49.5%) at study endpoint. The average level of information use for the intervention group was 52.2% (95% CI 46.2%-58.3%) at baseline and 75.8% (95% CI 71.6%-80.0%) at study endpoint. The study indicated that the net program change over time was 17% (95% CI 5%-28%;
The CBMP was effective at enhancing the capacity of study participants in using the routine HIS for decision-making. We noted that urban facilities had benefited more than their counterparts. The intervention has been shown to produce positive outcomes and should be scaled up to be used in other districts. Moreover, the mentorship modalities for rural facilities should be redesigned to maximize the benefits.
Pan African Clinical Trials Registry PACTR202001559723931; https://tinyurl.com/3j7e5ka5
A health information system (HIS) is an intersection between health care business processes and information systems to deliver better health care services [
Countries have made tremendous efforts to enhance data use practice for patient care and management. For example, a granular ontology model for maternal and child HISs and a national acute care information platform were implemented and improved data analysis skills and policy making in Pakistan [
In Botswana, a task-shifting initiative (ie, development of a dedicated monitoring and evaluation cadre) was implemented to strengthen monitoring and evaluation and build a sustainable HIS. As a result, the intervention brought increased use of health data for disease surveillance, operational research, and planning purposes [
Previous work has indicated that training, supervision, a good perceived culture of information use, having standard indicators, competence on routine HIS (RHIS) tasks, technology enhancement along with capacity building activities, and feedback systems were positively associated with routine health information use [
The investment in health systems infrastructure or training for clinicians and administrators in low- and middle-income countries (LMIC) has been low [
Baseline data were collected from April to May 2019 in 10 selected districts of the Amhara region: five were in the intervention group and five were in the comparison group. A year later, endpoint data were collected from June to July 2020 in the same districts. Thus, to estimate the effectiveness of the intervention, a quasi-experimental, nonequivalent, control group study design was employed [
The Amhara region is located between 8˚45′ N and 13˚45′ N latitude and 35˚46′ E and 40˚25′ E longitude in Northwest Ethiopia [
Map of the study area in the Amhara Region, Northwest Ethiopia, 2021.
The source population of this study included all health department and health facility heads. Individuals responsible for the health departments or health institutions that were expected to use routine health data and who were at the selected facilities for at least 6 months were included in the study. However, health department heads who were on leave, retired, and not supposed to use routine health information for patient monitoring and follow-up were excluded from the study. Since it was an intervention effectiveness study, newly inaugurated facilities with patient stays of up to 3 months were also excluded from the study.
The CBMP is an innovative approach that has been implemented since 2019 in selected districts of Ethiopia through the joint venture between the Federal Ministry of Health (FMoH) and selected universities; its goal is to strengthen the national HIS through proper data documentation, information use, and digitalization. The ARHB along with the UoG were responsible for implementing the intervention in the Amhara region. The intervention had two components: tailored training and mentorship. It targeted service delivery unit heads, case team leaders, health department heads, health facility heads, program officers, and district office managers [
The intervention was designed primarily to improve the capacity of health workers at different levels. Thus, data quality and information use training manuals were distributed to health facilities, and participants from intervention districts received training on DHIS2 (District Health Information Software 2) data analytics, visualization, presentation, troubleshooting, and data use for decision-making. Furthermore, HIS resources (ie, registers, tally sheets, and computers) were provided to health facilities to enhance the availability and quality of data, and joint review meetings, which served as a platform for discussion, were organized every 6 months [
Mentoring is a strategic development activity that supports health workers to attain the vision and goals of the organization [
The dependent variable of the study was the level of information use. The concept of
The study employed a quasi-experimental design with pre- and postassessment and intervention and comparison groups. Considering the difference in the level of information use among intervention and comparison districts after the intervention, we employed a double population proportion formula to estimate the sample size. Mathematically, the sample size was determined using the following formula [
where N is the sample size; P1 is the anticipated proportion of facilities with the attribute of interest (ie, level of information use after the intervention, assuming a 15% increase in information use and considered as 84%) [
As a quasi-experimental study, five districts were recruited for each arm (ie, intervention and comparison groups). The intervention districts were selected by the FMoH from among the low-performance districts regarding RHIS activities in the region. However, comparison districts were chosen randomly among the 146 districts in the region. We applied a multistage sampling procedure to select the study participants and developed a sampling frame that contained a list of the heads of departments and facilities in the selected districts. Thus, study participants were selected from the study population using a simple random sampling technique.
Data collection tools were developed based on the Performance of Routine Information System Management tools (version 3) and adapted to the local context [
The Information Use Assessment Tool is an interviewer-administered tool. It was used to examine the health facilities’ report production, information display, discussions, and decisions based on the RHIS, planning, supervision, and mentorship. The OBAT is a self-administered tool that is used to identify information about the technical, organizational, and behavioral constraints for routine health data use. Eight data collectors and two supervisors participated in the data collection. The principal investigator (PI) delivered training on data recording, document review, and ethical consideration to data collectors and supervisors for 2 days. A data quality checklist was developed and applied during data collection to maintain the quality of data. Daily feedback was provided to data collectors by supervisors. The PI led and coordinated the overall data collection process.
Data collectors requested permission from the facility heads to access the documents and departments. In addition, they provided information about the purpose of the study and obtained written consent from selected participants before interviewing them. Following that, they prepared participants for the interview. They reviewed source documents and charts posted in the department or unit, observed the discussion points made among members of the management body in the logbook, and collected data using the study tool. Subsequently, respondents were provided with a self-administered tool (ie, the OBAT) and informed to take an ample amount of time to complete the questionnaire.
The FMoH with the ARHB selected five intervention districts based on predefined criteria. All of these districts were low performers regarding the RHIS activities. They had a low level of information use and poor data quality but could potentially improve their performance if given the intervention. As a result, random assignments of districts to comparison and intervention arms were not applicable. However, the research team, in collaboration with the ARHB, selected five comparison districts to help in measuring the effectiveness of the intervention. Therefore, the intervention groups received usual service and the new CBMP intervention (ie, tailored training and mentorship), and the comparison groups received usual service (ie, supervision and review meeting by routine service).
The nature of the intervention was designed to be implemented by mobile mentors. As a result, we were unable to mask health workers and program implementers. However, all study participants and data collectors were blinded to the research question and hypothesis that the team generated during implementation, baseline, and endpoint data collection.
The team scrutinized the data to identify missing values before entering the data into the software. The data entry template was developed using EpiData software (version 3.1; EpiData Association) by applying the commands and skipping patterns that minimized errors during entry. Thus, cleaned data were entered into EpiData and exported to R software (version 4.0.4; The R Foundation) in CSV file format to compute the effect size of the intervention. R software has different built-in statistical packages that enable researchers to run statistical models and test the hypothesis in question. Descriptive statistics, such as mean and percentage, were calculated. Tables and graphs were also used for presenting findings.
The data were collected from the intervention and comparison districts before and after the implementation of the intervention. Thus, it entailed the difference-in-differences (DID) method to determine the effectiveness of the CBMP. As alluded to by different scholars, the DID method is one of the most frequently used methods in outcome and impact evaluation studies. Based on a combination of comparisons before and after the intervention as well as comparisons of the treatment and comparison groups, the method has an intuitive appeal and has been widely used in economics, public policy, health research, management, and other fields [
The DID approach applied a linear regression model and calculated the change over time in intervention and comparison groups. It double-differenced the change over time in the intervention group compared to the comparison group. The method also generated a valid estimate of the causal effect if the implementation of the CBMP was the only factor that might cause a change in the association between the CBMP and average information use before and after the intervention, as shown in the equation below:
where Ygt is the average level of information use, β1 is the average difference in Y between the two groups that is common in both time periods, β2 is the average change in Y from the baseline to the endpoint time period that is common to both groups, β3 is the average change in Y from the baseline to the endpoint time period of the intervention group compared to the comparison group, and β4 is the triple difference adjusted for some covariates.
To estimate the average change in information use over time using the DID model, we created a dummy variable by assigning 1 to the intervention group and 0 to the comparison group. Moreover, the preintervention period and postintervention period were assigned 0 and 1, respectively. Subsequently, we employed a difference-in-difference-in-differences (DIDID) method to assess whether the program effects were heterogeneous across sex (male vs female), age (≤30 years vs >30 years), educational level (diploma vs above diploma), location (rural vs urban), salary (≤5000 ETB [Ethiopian birr] vs >5000 ETB; a currency exchange rate of 44.32 ETB=US $1 is applicable), and experience (≤5 years vs >5 years) [
The model provided information about the β coefficients,
The study protocol was developed considering the ethical principles from the Declaration of Helsinki. The research protocol was registered at the Pan African Clinical Trials Registry (PACTR202001559723931), which is a World Health Organization International Clinical Trials Registry Platform primary register [
Baseline data were collected from 344 study participants (intervention: n=172, 50%; comparison: n=172, 50%) across 83 health facilities. However, a total of 321 study participants (intervention: n=155, 48.3%; comparison: n=166, 51.7%) across 79 health facilities (intervention: n=36, 46%; comparison: n=43, 54%) were surveyed at the endpoint of the study; the response rate was 93.3% (321/344). A total of 4 facilities out of 40 (10%) from the intervention arm were excluded because they became part of the newly established districts (
Flow diagram of study participants in the Amhara region, Northwest Ethiopia, 2021.
Among the 321 total study participants, 155 (48.3%) were from the intervention districts and 166 (51.7%) were from the comparison districts. More than half of the study participants were male in both intervention and comparison districts at baseline and endpoint periods. Almost two-thirds of the study participants were below the age of 28 years in both arms. Similarly, more than half of the study participants were diploma holders and resided in rural locations. Two-thirds of the participants earned equal to or below 5000 ETB at baseline, and nearly half of them earned above 5000 ETB at the study endpoint in both the intervention and comparison groups. More than half of the study participants had 5 years or more of experience (
Sociodemographic characteristics of study participants in the Amhara region, Northwest Ethiopia, 2021.
Variable | Baseline | Endpoint | |||||||||||
|
Intervention | Comparison | Intervention | Comparison | |||||||||
|
n (%) | 95% CI | n (%) | 95% CI | n (%) | 95% CI | n (%) | 95% CI | |||||
|
|||||||||||||
|
Female | 68 (39.5) | 32.3-47.3 | 63 (36.6) | 29.5-44.3 | 68 (43.9) | 35.9-52.1 | 60 (36.4) | 29.1-44.2 | ||||
|
Male | 104 (60.5) | 52.7-67.7 | 109 (63.4) | 55.7-70.5 | 87 (56.1) | 47.9-64.0 | 106 (63.6) | 55.8-70.9 | ||||
|
|||||||||||||
|
≤30 | 133 (77.3) | 70.2-83.3 | 147 (85.5) | 79.1-90.2 | 129 (83.2) | 76.2-88.6 | 130 (78.3) | 71.1-84.2 | ||||
|
>30 | 39 (22.7) | 16.8-29.8 | 25 (14.5) | 9.8-20.9 | 26 (16.8) | 11.4-23.8 | 36 (21.7) | 15.8-28.9 | ||||
|
|||||||||||||
|
Rural | 94 (54.7) | 46.9-62.2 | 122 (70.9) | 63.4-77.5 | 80 (51.6) | 43.5-59.7 | 116 (69.9) | 62.2-76.6 | ||||
|
Urban | 78 (45.3) | 37.8-53.1 | 50 (29.1) | 22.5-36.6 | 75 (48.4) | 40.3-56.5 | 50 (30.1) | 23.4-37.8 | ||||
|
|||||||||||||
|
Diploma or below | 104 (60.5) | 52.7-67.7 | 97 (56.4) | 48.6-63.9 | 87 (56.1) | 47.9-64.0 | 90 (54.5) | 46.6-62.2 | ||||
|
Above diploma | 68 (39.5) | 32.3-47.3 | 75 (43.6) | 36.1-51.4 | 68 (43.9) | 35.9-52.0 | 75 (45.5) | 37.8-53.4 | ||||
|
|||||||||||||
|
≤5000 | 131 (76.2) | 68.9-82.2 | 109 (65.3) | 57.5-72.4 | 77 (49.7) | 41.6-54.6 | 80 (48.5) | 40.7-56.4 | ||||
|
>5000 | 41 (23.8) | 17.8-31.0 | 58 (34.7) | 27.6-42.5 | 78 (50.3) | 42.2-58.4 | 85 (51.5) | 43.6-59.3 | ||||
|
|||||||||||||
|
≤5 | 107 (62.9) | 55.2-70.1 | 128 (65.3) | 58.1-71.9 | 89 (57.4) | 49.2-65.2 | 97 (58.8) | 50.9-66.3 | ||||
|
>5 | 63 (37.1) | 29.9-44.8 | 68 (34.7) | 28.1-41.9 | 66 (42.6) | 34.8-50.8 | 68 (41.2) | 33.7-49.1 |
aETB: Ethiopian birr; a currency exchange rate of 44.32 ETB=US $1 is applicable.
The study indicated that a mean of 30.2% (95% CI 23.3-37.2) and 51.7% (95% CI 44.23-59.3) of the department heads in comparison and intervention districts, respectively, used available evidence while making decisions at baseline, whereas a mean of 33.7% (95% CI 26.4-41.0) and 73.5% (95% CI 66.7-80.3) of the department heads in comparison and intervention districts, respectively, used available evidence while making decisions at study endpoint. At baseline, a mean of 41.9% (95% CI 34.5-49.3) and 43.0% (95% CI 35.5-50.5) of the departments in the comparison group calculated target achievement and program coverage, respectively, whereas a mean of 63.9% (95% CI 56.7-71.2) and 50.0% (95% CI 42.5-57.5) of the departments in the intervention group did so. However, the postperiod data showed that a mean of 56.0% (95% CI 48.4-63.7) and 45.2% (95% CI 37.6-52.8) of the comparison groups calculated target achievement and program coverage, respectively, whereas a mean of 85.5% (95% CI 80.1-90.9) and 76.5% (95% CI 70.0-83.0) of the intervention groups did so. At baseline, less than half of the study participants in both groups provided feedback to health workers at the lower levels; however, at the study endpoint, a mean of 34.9% (95% CI 27.5-42.3) of comparison group participants and 62.1% (95% CI 54.6-69.5) of the intervention group participants did so. At baseline, a mean of 41.9% (95% CI 34.5-49.3) and 58.7% (95% CI 51.3-66.2) of the departments in the comparison and intervention groups, respectively, had identified indicators, whereas at the study endpoint, a mean of 48.8% (95% CI 41.1-56.5) and 81.3% (95% CI 75.3-87.3) of the departments in the comparison and intervention groups, respectively, had done so (
Component indicators of routine information use at baseline and at the study endpoint in the comparison and intervention districts in the Amhara region, Northwest Ethiopia, 2021. The mean values are reported on the bars; the whiskers represent 95% CI values.
The average level of information use for the comparison group was 37.3% (95% CI 31.1%-43.6%) at baseline and 43.7% (95% CI 37.9%-49.5%) at the study endpoint. The average level of information use for the intervention group was 52.2% (95% CI 46.2%-58.3%) at baseline and 75.8% (95% CI 71.6%-80.0%) at the study endpoint. The DID analysis indicated that the net program effect change over time was significant (
DID analysis in control and intervention districts in the Amhara region, Northwest Ethiopia, 2021.
Parameter | Program effect size | 95% CI | |
Intercept | 0.356 | 0.300-0.413 | <.001 |
Group (intervention vs comparison) | 0.162 | 0.080-0.243 | <.001 |
Time (study endpoint vs baseline) | 0.08 | 0.000-0.160 | .047 |
DIDa analysis (group × time) | 0.173 | 0.058-0.288 | .003 |
aDID: difference-in-differences.
The DIDID estimate showed that the CBMP increased information use by 16% among department heads who were working in urban facilities, with a DIDID estimate of 0.16 (95% CI 0.026-0.289;
Subgroup analysis for selected variables in assessing program effect heterogeneity in the Amhara region, Northwest Ethiopia, 2021.
Effect modifier | Heterogeneity in program effect (95% CI) | |
Sex (male vs female ) | 0.046 (–0.089 to 0.182) | .50 |
Age (≤30 years vs >30 years) | –0.002 (–0.015 to 0.009) | .65 |
Educational level (diploma vs above diploma ) | –0.055 (–0.190 to 0.079) | .42 |
Salary (≤5000 ETBa vs >5000 ETB) | –0.00016 (–0.0005 to 0.00019) | .37 |
Residence (urban vs rural) | 0.16 (0.026 to 0.289) | .02 |
Experience (≤5 years vs >5 years) | –0.006 (–0.017 to 0.005) | .29 |
aETB: Ethiopian birr; a currency exchange rate of 44.32 ETB=US $1 is applicable.
All five component indicators showed high improvement at the study endpoint compared to baseline in each group. The intervention resulted in a 17% change in the average level of information use among study participants in the intervention districts compared to the comparison districts. We noted that the effect of the intervention was heterogeneous in urban and rural facilities. However, significant differences were not observed based on the sex, age, educational level, salary, and experience of the study participants.
This research revealed that the CBMP was effective in improving the capacity of the department and facility heads in using routine health information for decision-making and action. This finding was higher than that found in a cluster randomized controlled trial conducted in Sierra Leone on a community health data review meeting, which resulted in a 14% increment in evidence generation [
As indicated with these findings, the intervention resulted in the majority of study participants having identified and used indicators to track performance progress in their catchment area. It was consistent with a single study done in Zanzibar and Tanzania where the data use workshop resulted in improvement in the local use of target indicators [
Though the findings highlighted an improvement in providing feedback to health workers at lower levels, it was still unsatisfactory compared to the desired level [
It was evidenced in a subgroup analysis that urban dwellers benefited more from the intervention than their counterparts. This finding is also in line with the baseline study finding, which indicated that work location was a significant factor associated with the level of information use [
One limitation of this study was that we did not employ randomization and blinding because of the nature of the study. This may have introduced some information leakage between intervention and comparison districts. Some facilities from which we took baseline data were not included in the endpoint survey, which may have biased the results. Moreover, social desirability and recall bias may also have been introduced, since participants were part of the intervention.
The CBMP was found to be effective in improving department and facility heads’ capacity in using routine health information for decision-making. The intervention was more beneficial to study participants who resided in urban facilities than their counterparts. A remarkable change was observed in using the available evidence to inform decisions, identify indicators for tracking performance progress, compare targets versus achievements, and calculate program coverage. However, there is still a gap in providing synthesized feedback to health workers at lower levels. Therefore, we propose the following recommendations. The intervention has been proven to produce positive outcomes and should be scaled up to other districts. Moreover, attention should be given to enhance the capacity of health workers employed in rural facilities and to strengthen feedback mechanisms at all levels, in order to reach the desired outcomes of the information revolution.
Amhara Regional Health Bureau
Capacity Building and Mentorship Program
District Health Information Software 2
difference-in-differences
difference-in-difference-in-differences
Ethiopian birr
Federal Ministry of Health
health information system
low- and middle-income countries
Organizational and Behavioral Assessment Tool
principal investigator
routine health information system
University of Gondar
The research team members are thankful to the University of Gondar for providing an ethical review letter. We offer our appreciation to the Amhara Regional Health Bureau and the Amhara Public Health Institute for providing us with the support letter to conduct the research. The district office, health facilities, and departments were also supportive of our data collection, and we offer our gratitude. In the end, we are also grateful to the study participants and data collectors for their cooperation and involvement in the study.
This work would not be possible without the financial support from the Doris Duke Charitable Foundation (grant 2017187) through the University of Gondar. The funder had no role in data design, data collection, analysis, and interpretation of the study findings. The mission of the Doris Duke Charitable Foundation is to improve the quality of people’s lives through grants supporting the performing arts, environmental conservation, medical research, and child well-being, and through preservation of the cultural and environmental legacy of Doris Duke’s properties. We would also like to acknowledge the Norwegian Partnership Programme for Global Academic Cooperation for its financial contribution.
Data are available upon reasonable request from the corresponding author.
MAC participated in data analysis. All authors conceptualized the design of the study, provided a review of the methodology and analysis results, contributed to the writing of the manuscript, and read and approved the final manuscript.
None declared.