A quality assessment of Health Management Information System (HMIS) data for maternal and child health in Jimma Zone, Ethiopia

Abstract

Health management information system (HMIS) data are important for guiding the attainment of health targets in low- and middle-income countries. However, the quality of HMIS data is often poor. High-quality information is especially important for populations experiencing high burdens of disease and mortality, such as pregnant women, newborns, and children. The purpose of this study was to assess the quality of maternal and child health (MCH) data collected through the Ethiopian Ministry of Health’s HMIS in three districts of Jimma Zone, Oromiya Region, Ethiopia over a 12-month period from July 2014 to June 2015. Considering data quality constructs from the World Health Organization’s data quality report card, we appraised the completeness, timeliness, and internal consistency of eight key MCH indicators collected for all the primary health care units (PHCUs) located within three districts of Jimma Zone (Gomma, Kersa and Seka Chekorsa). We further evaluated the agreement between MCH service coverage estimates from the HMIS and estimates obtained from a population-based cross-sectional survey conducted with 3,784 women who were pregnant in the year preceding the survey, using Pearson correlation coefficients, intraclass correlation coefficients (ICC), and Bland-Altman plots. We found that the completeness and timeliness of facility reporting were highest in Gomma (75% and 70%, respectively) and lowest in Kersa (34% and 32%, respectively), and observed very few zero/missing values and moderate/extreme outliers for each MCH indicator. We found that the reporting of MCH indicators improved over time for all PHCUs, however the internal consistency between MCH indicators was low for several PHCUs. We found poor agreement between MCH estimates obtained from the HMIS and the survey, indicating that the HMIS may over-report the coverage of key MCH services, namely, antenatal care, skilled birth attendance and postnatal care. The quality of MCH data within the HMIS at the zonal level in Jimma, Ethiopia, could be improved to inform MCH research and programmatic efforts.

About the Authors:

Mariame Ouedraogo

Roles Conceptualization, Data curation, Formal analysis, Investigation, Writing – original draft

Affiliation: School of Epidemiology and Public Health, University of Ottawa, Ottawa, Ontario, Canada

Jaameeta Kurji

Roles Conceptualization, Methodology, Supervision, Writing – review & editing

Affiliation: School of Epidemiology and Public Health, University of Ottawa, Ottawa, Ontario, Canada

Lakew Abebe

Roles Funding acquisition, Project administration, Supervision, Writing – review & editing

Affiliation: Department of Health Behavior and Society, Public Health Faculty, Jimma University, Jimma, Oromiya, Ethiopia

Ronald Labonté

Roles Funding acquisition, Project administration, Writing – review & editing

Affiliation: School of Epidemiology and Public Health, University of Ottawa, Ottawa, Ontario, Canada

Sudhakar Morankar

Roles Funding acquisition, Project administration, Writing – review & editing

Affiliation: Department of Health Behavior and Society, Public Health Faculty, Jimma University, Jimma, Oromiya, Ethiopia

Kunuz Haji Bedru

Roles Conceptualization, Funding acquisition, Writing – review & editing

Affiliation: Jimma Zonal Health Office, Jimma, Oromiya, Ethiopia

Gebeyehu Bulcha

Roles Data curation, Investigation, Writing – review & editing

Affiliation: Jimma Zonal Health Office, Jimma, Oromiya, Ethiopia

Muluemebet Abera

Roles Funding acquisition, Project administration, Writing – review & editing

Affiliation: Department of Population and Family Health, Public Health Faculty, Jimma University, Jimma, Oromiya, Ethiopia

Beth K. Potter

Roles Methodology, Supervision, Writing – review & editing

Affiliation: School of Epidemiology and Public Health, University of Ottawa, Ottawa, Ontario, Canada

Marie-Hélène Roy-Gagnon

Roles Methodology, Supervision, Writing – review & editing

Affiliation: School of Epidemiology and Public Health, University of Ottawa, Ottawa, Ontario, Canada

Manisha A. Kulkarni

Roles Conceptualization, Funding acquisition, Methodology, Project administration, Resources, Supervision, Writing – review & editing

* E-mail: [email protected]

Affiliation: School of Epidemiology and Public Health, University of Ottawa, Ottawa, Ontario, Canada

ORCID logo http://orcid.org/0000-0002-5084-4960

Introduction

As a comprehensive, disaggregated and comparable source of information about health services in low and middle-income countries, health management information systems (HMIS) have the strong potential to act as the cornerstone for effective policy decision making[1,2]. Yet, numerous issues exist related to the completeness, timeliness, and accuracy of HMIS data [3–13], resulting in the tendency for countries and global health initiatives to rely on indicators measured through population-based surveys, such as the Demographic Health Survey and Multiple Indicator Cluster Survey. These surveys, however, do not provide continuous estimates of population-level indicators given their cross-sectional nature [2,11].

As part of the Federal Ministry of Health (FMoH) “One Plan, One Budget & One Report” policy, Ethiopia contracted the consulting firm John Snow, Inc in the years 2006–2007 to assess and redesign its HMIS. The aim of this effort was to improve the management and optimum use of resources for making timely decisions and promoting effective health care system delivery [14]. This evaluation found that the burden of HMIS data management was high, different partners were independently gathering the same information, standardized indicator definitions among the different partners and regions were lacking, data were of poor quality, and the use of HMIS information was weak and highly centralized at the highest levels of the reporting system [14,15]. Given the paper-based nature of the HMIS at that time, the project also resulted in the design and deployment of an electronic HMIS in health facilities nationwide [14]. Following the 2006 reform, another assessment in 2009–2010 highlighted that the HMIS remained “cumbersome and fragmented” [14], with persisting poor quality of data and inadequate skills for collecting, analyzing and interpreting the information among the health care staff at the lowest levels of the health system [14].

More recent work has found that Ethiopia’s HMIS remains particularly weak at the primary health care unit (PHCU) and district health office levels [15–20]. A study conducted by Teklegiorgis et al. in Eastern Ethiopia indicated that completeness and timeliness of data reporting were below national targets, with rates of 82% and 77%, respectively [19]. In the same study, 65% and 42% of health facilities used the information gathered for decision-making and for observing trends in service delivery, respectively [19]. In another study conducted in Jimma Zone, timeliness was determined to be poor with fewer than 50% of health posts, health centres and district health offices reporting within their given deadline [17]. Thirty-eight percent of health workers recognized that their reports and registration books may contain inconsistencies [17].

To our knowledge, no study in Ethiopia has focused specifically on the quality of maternal and child health (MCH) data collected within the HMIS. Recognizing the importance of collecting quality health data for populations facing a high burden of morbidity and mortality, such as pregnant women, newborns, and children, we assessed the quality of MCH data collected through Ethiopia’s HMIS over a 12-month time period in three districts of Jimma Zone, Southwest Ethiopia. To this end, we used data quality dimensions proposed by the World Health Organization (WHO) and MCH indicator estimates from a cross-sectional household survey conducted within the same time frame.

Materials and methods

Study setting

Jimma Zone is located in Oromiya region, Southwest Ethiopia, approximately seven hours from Addis Ababa, Ethiopia’s capital. The total population of this Zone is estimated at approximately 3.3 million inhabitants. Jimma Zone is further divided into 21 Woredas or districts. Three of the districts in Jimma Zone were selected for a large cluster-randomized trial designed to address barriers to safe motherhood (ClinicalTrials.gov identifier: NCT03299491): Gomma, Kersa, and Seka Chekorsa. Ten PHCUs are located in Gomma, nine in Seka Chekorsa and seven in Kersa, resulting in a total of 26 PHCUs. A PHCU consists of one health centre associated with five health posts. Each health post is managed by two government-employed health extension workers, trained in the delivery of primary care to communities within their catchment area.

Data sources

Health management information data.

The primary level of health service delivery and data collection in Ethiopia occurs in PHCUs. The compiled data from PHCUs are forwarded to and reviewed by the district health offices before being submitted to the zonal health office, where the data are entered into the electronic HMIS and made accessible to regional and national levels policy-makers [21].

For our study, we consulted with the Jimma Zonal Health Office and extracted electronic monthly service reports for the 2015 fiscal year (i.e. July 2014 to June 2015) for Gomma, Kersa and Seka Chekorsa district health offices and their respective PHCUs. Monthly reports from the two previous fiscal years (i.e. July 2012-June 2013 and July 2013-June 2014) were also retrieved to assess internal consistency.

Cross-sectional survey data.

As part of the baseline evaluation for the above-mentioned cluster-randomized trial designed to address barriers to safe motherhood in Jimma Zone, a community-based population-representative cross-sectional survey was performed with women aged 15–49 years from the three districts, who had had a birth outcome (live birth / stillbirth / miscarriage / abortion) within the same time frame as the HMIS extracted reports. Results from this survey are the subject of a separate publication and are not presented herein. To attain the sample size for the trial, a two-stage sampling strategy was used. Twenty-four PHCUs or clusters were first randomly selected from the 26 available in the three study districts. From each PHCU catchment area, 160 eligible women were then randomly selected from registration lists compiled by volunteer community health workers in each village or kebele. A total of 3,784 women provided information on their past use of maternal and child health services as well as on their experiences when they were pregnant, during childbirth, and after delivery (see survey questions in S1 Appendix). After obtaining informed consent, face-to-face surveys were conducted at the women’s households by trained interviewers. For our purposes, we only consider those participants who reported attending any of the MCH services in one of the PHCUs found in the three districts. This resulted in the exclusion of 2.5% of the total sample of women who attended a health facility located in another district.

World Health Organization data quality report card

Recognizing the need for developing countries to regularly evaluate the quality of their routine health information system, the WHO developed several tools to assist in assessing common data quality dimensions, including the Data Quality Report Card (DQRC), which we used in this study. The DQRC considers several data quality dimensions and represents a relatively easy and quick quantitative method to identify inaccuracies and inconsistencies in HMIS data [22]. Four dimensions of data quality are included in the DQRC: completeness of reporting; the internal consistency of reported data; the external consistency of population data; and, external consistency of coverage rates [23]. These four dimensions are further sub-divided into specific elements or indicators (Table 1). Given the unavailability of appropriate district-level census data, we did not assess the external consistency of population data.

[Figure omitted. See PDF.]

Table 1. Data quality dimensions and indicators defined in the World Health Organization data quality report card.

https://doi.org/10.1371/journal.pone.0213600.t001

Maternal and child health indicators

In this data quality assessment, we examined several key maternal health services and immunization coverage indicators recommended by the WHO [23], including: antenatal care first (ANC1) and fourth (ANC4) visit coverage; deliveries attended by a skilled birth attendant (SBA) in health facilities; access to early postnatal care (PNC); and, infant receipt of Diphtheria, Tetanus, Pertussis vaccine first (DTP1) and third (DTP3) dose. We also included two additional indicators: malaria in pregnancy and stillbirth rate.

Statistical analysis

We generated appropriate descriptive statistics for the first three dimensions of data quality according to the DQRC, using percentages, means, and standard deviation estimates. All analyses were performed using SAS statistical software version 9.4. (SAS Institute Inc., Cary, NC, USA).

To assess the fourth dimension of data quality (i.e. external consistency of coverage rates), we considered rigorous statistical tests that were not part of the DQRC. We generated scatter plots along with Pearson correlation coefficients and intraclass correlation coefficients to determine whether coverage rates from HMIS and survey data were linearly associated and the amount of variability between the two methods, respectively. We calculated ICCs using the Deyo’s method, as described by Szklo and Nieto [24]. Deyo’s method quantifies the variance of the difference between measurements taken by two scorers or readings (i.e. between-scorers/readings) as well as the variability within the repeated measurements recorded by each scorer (i.e. within-scorer). In this study, the ICC calculations took into account the variability between the two methods of MCH coverage rate estimation (i.e. HMIS vs. cross-sectional survey) and the variances reported within district by each method. We used Bland-Altman analysis, a standard method to assess agreement between two quantitative measurements, to estimate the concordance between MCH indicator values from the HMIS and the survey [25]. Given the number of data points in each district (i.e. ten, nine and seven PHCUs in Gomma, Seka Chekorsa and Kersa, respectively), adherence to the normal distribution might have compromised the results of the Bland and Altman analysis. We therefore estimated the limits of agreement using non-parametric methods [26]. Results using limits of agreement defined by the WHO are also available upon request.

Ethics approval

Given low levels of literacy in the survey population, verbal informed consent was obtained all survey participants following verbal explanation of the contents of consent forms in the local language (Amharic or Afaan Oromo). Ethical clearance for this study and the verbal consent procedure was obtained from the Research Ethics and Integrity Board at the University of Ottawa and the Jimma University Institutional Review Board.

Results

Completeness of reporting

Completeness and timeliness of reporting.

Data completeness was the highest in Gomma district with 76% of expected monthly health post and health centre reports received by the district health office, followed by 49% in Seka Chekorsa and 33% in Kersa (Table 2). Similar patterns were observed for timeliness of PHCU reporting. Gomma district health office received 71% of its expected PHCU reports on time, compared to 33% and 50% in Kersa and Seka Chekorsa, respectively.

[Figure omitted. See PDF.]

Table 2. Completeness, timeliness and accuracy of reporting of HMIS data in the three districts in Jimma Zone, Ethiopia (2014–2015) based on an assessment of selected MCH indicators using the data quality report card.

https://doi.org/10.1371/journal.pone.0213600.t002

Content completeness.

Of the submitted reports, Kersa had four (0.8%) missing indicator values, while Gomma only had three (0.4%) missing values all found within one PHCU. No zero or missing MCH indicator values were found in Seka Chekorsa reports (Table 2).

Internal consistency of reported data

Accuracy of event reporting.

Kersa had the largest proportion of moderate outliers, with 4.7% of the MCH indicator values being at least two standard deviations from their average value for the two preceding years, compared to 4.3% Gomma and 3.1% in Seka Chekorsa (Table 2). Very few extreme outliers were detected in the three districts.

Consistency over time.

An increase in the uptake of MCH services over time was observed in all districts and in most PHCUs (Table 3). Postnatal care (PNC) showed the greatest increase in coverage, expressed as a ratio of the number of events in 2014–2015 relative to the average number of events in the two previous years. In contrast, ANC1, DTP1 and DTP3 coverage rates were relatively consistent over time in the three districts.

[Figure omitted. See PDF.]

Table 3. Consistency over time ratios for HMIS data in three districts in Jimma Zone, Ethiopia (2012–2015) based on an assessment of selected MCH indicators using the data quality report card.

https://doi.org/10.1371/journal.pone.0213600.t003

Internal consistency between indicators.

Consistency between the number of ANC1 visits and number of DTP1 doses was low in Gomma, with a ratio of 0.58, suggesting that 42% more women attended an ANC1 visit than children receiving their first dose of DTP1 (Table 4). Similar trends were observed in Kersa and Seka Chekorsa, with 30% and 63% greater ANC1 coverage than DTP1 administration, respectively.

[Figure omitted. See PDF.]

Table 4. Internal consistency of HMIS data in three districts in Jimma Zone, Ethiopia, (2014–2015) based on an assessment of selected MCH indicators using the data quality report card.

https://doi.org/10.1371/journal.pone.0213600.t004

All PHCUs located in Gomma and Kersa had a negative percentage difference between number of DTP1 doses and the number of DTP3 doses administered, reflecting higher administration of DTP1 vaccines compared to DTP3. However, four out of nine PHCUs in Seka Chekorsa showed positive differences, a trend that was also seen for the comparison between ANC4 and ANC1 coverage, suggesting that there might be some issues with the quality of the reported data.

External consistency of coverage rates

Most Pearson correlation coefficients suggested weak or moderate linear associations between HMIS and survey estimates in all three districts (Table 5). While strong correlation coefficients were observed for SBA and early PNC at the newborn level in Gomma (0.81 and 0.72, respectively), and for malaria in pregnancy (0.84), most ICC values were close to zero or negative, which demonstrates that the between-method variability (i.e. between HMIS and cross-sectional survey) and/or method error was significantly more important than the within-district variability. This suggests a large degree of discordance between the HMIS and the survey data.

[Figure omitted. See PDF.]

Table 5. Pearson correlation coefficients and intraclass correlation coefficients for the relationship between HMIS and survey estimates from three districts in Jimma Zone, Ethiopia.

https://doi.org/10.1371/journal.pone.0213600.t005

The Bland-Altman analyses also indicate poor agreement between the HMIS and the cross-sectional survey estimates (Table 6 and S1–S4 Figs). In Gomma, the Bland-Altman analysis showed a negative systematic bias of -25.1% (95% CI: -40.9, -20.7%) for the proportion of women who attended ANC1, suggesting that, on average, ANC1 estimates in the HMIS were lower than estimates obtained from the survey. While the concordance between the two methods was relatively high for ANC4 in Gomma and Kersa, poor agreement was observed in Seka Chekorsa (median difference of 47.5% (95% CI: 32.6–73.5)). In all three districts, the HMIS reported higher SBA and PNC service uptake than the survey, while good agreement was observed in all three districts for stillbirth rates and rates of malaria infection during pregnancy.

[Figure omitted. See PDF.]

Table 6. Bland-Altman summary statistics for the agreement analysis between HMIS and survey data from three districts in Jimma Zone, Ethiopia.

https://doi.org/10.1371/journal.pone.0213600.t006

Discussion

Using an established data quality assessment tool developed by the WHO, we identified several issues related to the quality of MCH data collected within the HMIS in Jimma Zone, Ethiopia. Many PHCUs failed to submit complete and timely reports to their respective district health office that reflected the monthly utilization of their MCH services. The observed rates of completeness and timeliness in all districts were lower than the 90% limit sets by the FMoH [18]. The failure to submit reports, and thus MCH coverage estimates, to the next reporting level may result in a partial and incomplete representation of MCH service provision at the district level. This could have important implications for the health of pregnant women and newborns living in these districts, as information reported by the PHCUs is used by the district health offices to guide future plans. Similar challenges with the completeness and timeliness of HMIS reporting have been observed in other studies [18,27,28].

Our assessment of the consistency between indicators revealed potential gaps that warrant further investigation, as well as potential data quality issues. For example, our observation that ANC1 was higher than DPT1 coverage in some PHCUs suggests that more women may be attending antenatal care compared to infants receiving DPT vaccination, or this may reflect issues in data quality. However, this discrepancy may also reflect a higher number of pregnancies than live births, which was not assessed directly. The finding that DPT3 was higher than DPT1 in several PHCUs suggests that many infants who received their first dose may not have received their third one, an issue that warrants further investigation. Higher DPT3 than DPT1 coverage estimates, and ANC4 than ANC1 coverage estimates, are indicative of data quality issues. The latter may also reflect double counting of women at multiple levels of the health system if these discrepancies are not identified and investigated at the PHCU and district levels, and corrected to ensure data quality [29].

Comparison of HMIS data with cross-sectional population-representative survey data also revealed data quality issues, suggesting that some PHCUs and district health offices may over-report certain MCH indicators, including SBA and PNC. This is consistent with a national evaluation conducted in 2016, which identified that over-reporting was important, with 43% of PHCUs in Oromiya region over-reporting ANC attendance by 10% or more, while both under- and over-reporting of SBA data were observed [15]. Over-representing the proportion of women attending health services may have a significant impact on women’s health outcomes if they are not attending these services, and resources may not be going towards improving their access to these essential services.

Although we conducted this evaluation using an established data quality tool and rigorous statistical methods, there are several limitations that constrained our findings. First, we did not perform an exhaustive assessment of all MCH indicators collected in the HMIS. Due to time constraints, indicators related to the use of family planning methods or to the prevention of mother-to-child transmission of HIV were not considered, despite the importance given to them within PHCUs’ service reports [21]. We also did not assess the consistency between information contained in source documents (i.e. health facility registers and district health office summary reports) and the information reported in the HMIS, thereby precluding our assessment of the levels at which most reporting errors occur. Our study is also limited by the fact that we did not consider private health facilities in our assessments. This was mainly due to time constraints, but also due to private health facilities being less likely to provide the common MCH services that are reported to district health offices [21], and only infrequently reporting data on their health service attendance. Focusing on the quality of MCH services data provided in governmental health facilities thus provides an adequate assessment of the quality of MCH data in Jimma Zone. Despite these limitations, a particular strength of this study was the comparison with data from a large household-based cross-sectional survey that recruited women attending the PHCUs of interest, allowing us to perform an agreement analysis. While self-reporting and interviewer biases may have occurred in administering the survey, comprehensive training of survey interviewers was used to limit such bias.

Our findings support those of previous studies, which suggest that strategies to improve the quality of MCH data collected within the HMIS are needed [17]. Important strategies may focus on ensuring effective communication processes between health workers, HMIS supervisors at the health facility level, and district-level officials [17]. Furthermore, given their situation at the primary level of health data collection, there is an urgent need to ensure that health post and health centre workers are well trained in data reporting and the performance of data quality checks. Additional qualitative research on the perceptions of health workers and HMIS officers roles in the management and sustainability of the HMIS would be beneficial to understand their concerns and their ideas about how they can contribute to enhancing the quality of the data gathered within the HMIS.

It is also important to note that barriers other than those relating to the lack of training and knowledge of health workers and supervisors may explain the poor quality of MCH data in the HMIS. While the introduction of an electronic HMIS has resulted in improvements in the reporting, monitoring and evaluation of health data, the availability of other essential material resources to perform key HMIS tasks may be more limited. In other regions of Ethiopia it has been noted that PHCUs and district health offices sometimes lack access to computers to facilitate the compilation of the data and its utilization [20]. Belay et al. also noted that HEWs in health posts may not have access to calculators to perform basic calculations to assess the use of services in their catchment areas, which can limit their ability to report accurate data to higher levels of the health system, as well as a shortage of registration and summary forms [20]. Overcoming material barriers can serve to strengthen HMIS data for informing MCH and other programmatic health efforts in Ethiopia.

Supporting information

[Figure omitted. See PDF.]

S1 Appendix. Survey questions used in HMIS data quality assessment for comparison of MCH coverage estimates.

https://doi.org/10.1371/journal.pone.0213600.s001

(DOCX)

S1 Fig. Bland-Altman plots for the agreement of malaria in pregnancy rate estimates between the HMIS and the cross-sectional survey in three district of Jimma Zone, Ethiopia.

https://doi.org/10.1371/journal.pone.0213600.s002

(DOCX)

S2 Fig. Bland-Altman plots for the agreement of maternal and child health indicator coverage estimates between the HMIS and the survey in Gomma.

https://doi.org/10.1371/journal.pone.0213600.s003

(DOCX)

S3 Fig. Bland-Altman plots for the agreement of maternal and child health indicator coverage estimates between the HMIS and the survey in Kersa.

https://doi.org/10.1371/journal.pone.0213600.s004

(DOCX)

S4 Fig. Bland-Altman plots for the agreement of maternal and child health indicator coverage estimates between the HMIS and the survey in Seka Chekorsa.

https://doi.org/10.1371/journal.pone.0213600.s005

(DOCX)

Acknowledgments

The study team thanks the Jimma Zonal Health Office for their support and assistance in retrieval of the data from the electronic HMIS, as well as Abebe Mamo, Getachew Kiros and Shifera Asfaw for their contribution to the cross-sectional survey data collection. The study team also acknowledges the women who participated in the survey for their time and participation.

Citation: Ouedraogo M, Kurji J, Abebe L, Labonté R, Morankar S, Bedru KH, et al. (2019) A quality assessment of Health Management Information System (HMIS) data for maternal and child health in Jimma Zone, Ethiopia. PLoS ONE 14(3): e0213600. https://doi.org/10.1371/journal.pone.0213600

References

1. Lopez AD. Can epidemiology inform global health and development targets? Int J Epidemiol. 2015;44(1):9–11. pmid:25713308

2. AbouZahr C, Boerma T. Health Information Systems: The Foundations of Public Health. Bull World Health Organ. 2005;83(8):578–83 pmid:16184276

3. Glélé Ahanhanzo Y, Ouendo E-M, Kpozèhouen A, Levêque A, Makoutodé M, Dramaix-Wilmet M. Data quality assessment in the routine health information system: an application of the Lot Quality Assurance Sampling in Benin. Health Policy Plan. 2015;30(7):837–43. pmid:25063699

4. Nisingizwe MP, Iyer HS, Gashayija M, Hirschhorn LR, Amoroso C, Wilson R, et al. Toward utilization of data for program management and evaluation: quality assessment of five years of health management information system data in Rwanda. Glob Health Action. 2014;7(0):25829.

5. Bosch-Capblanch X, Ronveaux O, Doyle V, Remedios V, Bchir A. Accuracy and quality of immunization information systems in forty-one low income countries. Trop Med Int Heal. 2009;14(1):2–10.

6. Ndabarora E, Chipps JA, Uys L. Systematic review of health data quality management and best practices at community and district levels in LMIC. Inf Dev. 2014;30(2):103–20.

7. Mate KS, Bennett B, Mphatswe W, Barker P, Rollins N. Challenges for routine health system data management in a large public programme to prevent mother-to-child HIV transmission in South Africa. PLoS One. 2009;4(5):e5483. pmid:19434234

8. Mpimbaza A, Miles M, Sserwanga A, Kigozi R, Wanzira H, Rubahika D, et al. Short Report: Comparison of Routine Health Management Information System Versus Enhanced Inpatient Malaria Surveillance for Estimating the Burden of Malaria Among Children Admitted to Four Hospitals in Uganda. Am J Trop Med Hyg. 2015;92(1):18–21. pmid:25422396

9. Amoakoh-Coleman M, Kayode GA, Brown-Davies C, Agyepong IA, Grobbee DE, Klipstein-Grobusch K, et al. Completeness and accuracy of data transfer of routine maternal health services data in the greater Accra region. BMC Res Notes. 2015;8(1):e114.

10. Gimbel S, Micek M, Lambdin B, Lara J, Karagianis M, Cuembelo F, et al. An assessment of routine primary care health information system data quality in Sofala Province, Mozambique. Popul Health Metr. 2011;9:e12.

11. Manya A, Nielsen P. Reporting practices and data quality in health information systems in developing countries: An exploratory case study in Kenya. J Health Inform Dev Ctries. 2016;10(1):114–126.

12. Odhiambo-Otieno GW. Evaluation of existing District Health Management Information Systems. Int J Med Inform. 2005;74(9):733–44. pmid:15979937

13. Sharma A, Rana SK, Prinja S, Kumar R. Quality of Health Management Information System for Maternal & Child Health Care in Haryana State, India. PLoS One. 2016;11(2):e0148449. pmid:26872353

14. Vital Wave Consulting HIS. Health Information Systems in Developing Countries [Internet]. 2009 [cited 2017 Jun 14].

15. Ethiopian Public Health Institute, Federal Ministry of Health, World Health Organization. Ethiopia Health Data Quality Review: System Assessment and Data Verification 2016 [Internet]. Addis Ababa; 2016 http://www.ephi.gov.et/images/pictures/download2009/V6%20Final%20DV-SA%20Report%20Jan%202017.pdf

16. Abera E, Daniel K, Letta T, Tsegaw D. Utilization of Health Management Information System and Associated Factors in Hadiya Zone Health Centers, Southern Ethiopia. Res Heal Sci. 2016;1(2):e98.

17. Abajebel S, Jira C, Beyene W. Utilization of health information system at district level in Jimma zone, Oromia regional state, Southwest Ethiopia. Ethiop J Health Sci. 2011;21(Suppl 1):65–76. pmid:22435010

18. Gebrekidan M, Hajira M, Habtamu T, Negusu W, Dereje M, Nafo-Traoré F. Data quality and information use: A systematic review to improve evidence. Ethiopia Int J Intell Inf Syst. 2012;3(6):69–75.

19. Teklegiorgis K, Tadesse K, Mirutse G, Terefe W. Level of data quality from Health Management Information Systems in a resources limited setting and its associated factors, eastern Ethiopia. SA J Inf Manag. 2016;18(1):a612.

20. Belay H, Azim T, Kassahun H. Assessment of Health Management Information System (HMIS) Performance in SNNPR, Ethiopia SNNP Regional Health Bureau [Internet]. 2014. http://pdf.usaid.gov/pdf_docs/PA00K27K.pdf

21. Bhattacharyya S, Berhanu D, Taddesse N, Srivastava A, Wickremasinghe D, Schellenberg J, et al. District decision-making for health in low-income settings: a case study of the potential of public and private sector data in India and Ethiopia. Health Policy Plan. 2016;31(suppl 2):25–34.

22. Chen H, Hailey D, Wang N, Yu P. A Review of Data Quality Assessment Methods for Public Health Information Systems. Int J Environ Res Public Health. 2014 May 14;11(5):5170–207. pmid:24830450

23. World Health Organization. Guide to the health facility data quality report card [Internet]. 2014. http://www.who.int/healthinfo/DQRC_Indicators.pdf

24. Szklo M, Nieto J. Epidemiology: Beyond the basics. Jones and Bartlett Publishers; 2007.

25. Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986;1(8476):307–10. pmid:2868172

26. Bland JM, Altman DG. Measuring agreement in method comparison studies. Stat Methods Med Res. 1999;8(99):962–2802.

27. Bayisa R. Assessment of health management information system (HMIS) data quality and information use: The case of Yekatit 12 Hospital, Addis Ababa [Internet]. M.Sc. Thesis, Addis Ababa University; 2014. http://etd.aau.edu.et/bitstream/123456789/12866/1/RegasaBayisa.pdf

28. Tsedeke Mathewos. Community health management information system Performance and factors associated with at health post of Gurage zone, SNNPR, Ethiopia [Internet]. M.Sc. Thesis, University of Gondar and Addis Continental Institute of Public Health; 2015. https://www.researchgate.net/profile/Tsedeke_Mathewos/publication/304567082_Community_health_management_information_system_Performance_and_factors_associated_with_at_health_post_of_Gurage_zone_SNNPR_Ethiopia/links/5773798508aeef01a0b66949/Community-health

29. Ministry of Health Federal Democratic Republic of Ethiopia. HMIS Information Use Guide- Technical Standards Area 4: Version 2 [Internet]. 2013. https://www.jsi.com/JSIInternet/Inc/Common/_download_pub.cfm?id=13660&lid=3