Evaluation of the Performance of a Dengue Outbreak Detection Tool for China

Abstract

An outbreak detection and response system, using time series moving percentile method based on historical data, in China has been used for identifying dengue fever outbreaks since 2008. For dengue fever outbreaks reported from 2009 to 2012, this system achieved a sensitivity of 100%, a specificity of 99.8% and a median time to detection of 3 days, which indicated that the system was a useful decision tool for dengue fever control and risk-management programs in China.

About the Authors:

Honglong Zhang

Contributed equally to this work with: Honglong Zhang, Zhongjie Li

Affiliation: Key Laboratory of Surveillance and Early-warning on infectious Disease, Division of Infectious Diseases, Chinese Center for Disease Control and Prevention, Beijing, China

Zhongjie Li

Contributed equally to this work with: Honglong Zhang, Zhongjie Li

Affiliation: Key Laboratory of Surveillance and Early-warning on infectious Disease, Division of Infectious Diseases, Chinese Center for Disease Control and Prevention, Beijing, China

Shengjie Lai

Affiliation: Key Laboratory of Surveillance and Early-warning on infectious Disease, Division of Infectious Diseases, Chinese Center for Disease Control and Prevention, Beijing, China

Archie C. A. Clements

Affiliation: Research School of Population Health, College of Medicine, Biology and Environment, The Australian National University, Canberra, Australia

Liping Wang

Affiliation: Key Laboratory of Surveillance and Early-warning on infectious Disease, Division of Infectious Diseases, Chinese Center for Disease Control and Prevention, Beijing, China

Wenwu Yin

Affiliation: Key Laboratory of Surveillance and Early-warning on infectious Disease, Division of Infectious Diseases, Chinese Center for Disease Control and Prevention, Beijing, China

Hang Zhou

Affiliation: Key Laboratory of Surveillance and Early-warning on infectious Disease, Division of Infectious Diseases, Chinese Center for Disease Control and Prevention, Beijing, China

Hongjie Yu

Affiliation: Key Laboratory of Surveillance and Early-warning on infectious Disease, Division of Infectious Diseases, Chinese Center for Disease Control and Prevention, Beijing, China

Wenbiao Hu

* E-mail: [email protected] (WH); [email protected] (WY)

Affiliation: Infectious Disease Epidemiology Unit, School of Population Health, University of Queensland, Brisbane, Australia

Weizhong Yang

* E-mail: [email protected] (WH); [email protected] (WY)

Affiliation: Key Laboratory of Surveillance and Early-warning on infectious Disease, Division of Infectious Diseases, Chinese Center for Disease Control and Prevention, Beijing, China

Introduction

Detecting infectious disease outbreaks quickly is crucial for timely implementation of control measures, thereby minimizing morbidity and mortality. To automatically identify aberrations in disease incidence data at an early stage, some countries have established infectious disease surveillance and outbreak detection systems, such as the Early Aberration Reporting System (EARS) of the US Centre for Disease Control and Prevention [1], the Real-time Outbreak and Disease Surveillance (RODS) system of the University of Pittsburgh [2], the Electronic Surveillance System for the Early Notification of Community-Based Epidemics (ESSENCE) in the USA [3], and SurvNet@RKI in Germany [4].

Dengue fever (DF) is one of the world’s most important vector-borne diseases, with cases reported from more than 100 countries in Africa, America, Southeast Asia, the Western Pacific and Europe. The World Health Organization estimates that DF affects over 50 million people annually [5]. In 1978, DF re-emerged in Guangdong province, China, after having disappeared from the country for more than 30 years [6]. From 1978 to 2012 more than 650,000 cases of DF, and hundreds of deaths, were documented in China. DF epidemics had spread from Guangdong, Hainan and Guangxi provinces in the southern coastal regions to some relatively northern regions including Fujian and Zhejiang provinces [7]. It has been suggested that establishing an early outbreak detection system is one of the prerequisites for adequate preparedness and responses to DF epidemics [5], which can enable better engagement of the community in prevention and control activities, thereby reducing DF transmission and improving clinical outcomes.

To facilitate early detection of infectious disease outbreaks, the Chinese Centre for Disease Control and Prevention (China CDC) has developed the China Infectious Disease Automated-alert and Response System (CIDARS), implemented since April 2008 [8]. CIDARS automatically conduct the aberration detection from the reported data in the web-based Nationwide Notifiable Infectious Diseases Reporting Information System (NIDRIS). This was established in 2004 and is the largest direct infectious disease reporting system in the world, covering all general hospitals in the prefectures, and all hospitals in the counties and townships in China [9]. CIDARS has been operating in China for more than four years. This study aims to provide a preliminarily prospective evaluation of the performance of CIDARS for DF outbreak detection during the initial phase of real-world implementation nationwide.

Methods

As a notifiable disease in China, DF are diagnosed by clinicians according to the diagnostic criteria for DF enacted by the Chinese Ministry of Health, which includes suspected cases, clinically diagnosed cases and laboratory-confirmed cases. They are then reported to NIDRIS.

In CIDARS, a time-series moving-percentile method (MPM) was used to detect aberrations of DF occurrence at the county level for the 31 provinces by comparing the reported cases in the current observation period to those of the corresponding historical period for each county. Accounting for the day-of-week effect and the stability of data, we used the most recent seven-day period as the current observation period and the previous three years as the historical period [1], [10]. The number of cases in the current observation period is the sum of reported cases within the recent seven days. The corresponding historical period included, for each of the previous three years, the same seven-day period, the two preceding seven-day periods and the two following seven-day periods, resulting in 15 historical seven-day data blocks covering 105 days. The 50th percentile of the 15 blocks of historical data was set as the threshold value to trigger a signal in CIDARS. The current observation period and historical data blocks were dynamically moved forward day by day [8]. When the number of reported cases in the current observation period reached the defined threshold, CIDARS automatically generated a signal.

According to the operational proposal of CIDARS, action should be taken in response to the signals, which included two steps: (1) signal initial verification, and (2) signal field investigation. The signal initial verification was carried out by epidemiologists in local CDCs by reviewing the reported cases, conducting the rapid assessment of information from other surveillance sources or directly contacting the cases reporting agencies. If the signal denoted one suspected outbreak after the initial verification, then a field investigation was conducted to confirm whether an outbreak was occurring, otherwise this signal would be determined as a negative signal.

The nationwide reported outbreaks from 2009 to 2012 were adopted as the reference standard to evaluate the performance of outbreak detection by CIDARS. According to the national DF surveillance proposal, the definition of a reported DF outbreak was ≥3 cases occurring in a concentrated setting (e.g., community, school or village) within 15 days [11]. All outbreaks defined using this criteria should be reported to an information system in China CDC by the public health staff of local CDC. The start and end dates of an outbreak were defined as the dates when the first and last cases of the outbreak was reported. When CIDARS generated a signal during the period of outbreak occurrence, we considered the outbreak to have been detected.

In this study, we investigated the spatial and temporal patterns of reported DF cases and signals from January 1st, 2009 to December 31st, 2012. Sensitivity, specificity and time to detection (TTD) were employed to assess the performance of CIDARS for early detection of DF outbreaks during the study period [12]. Sensitivity was defined as the number of outbreaks during which at least one signal was triggered by CIDARS, divided by the total number of reported outbreaks. Specificity was defined as the number of non-outbreak days with no signal, divided by the total number of non-outbreak days. TTD was defined as the interval between the start of each outbreak and the first signal during the outbreak. If the signal was triggered on the first day of an outbreak, TTD was zero.

Results

Between 2009 and 2012, 1224 DF cases were reported in China. The temporal pattern showed seasonal variation, with 86.8% (1,062) of cases being reported from the 31^st week to the 49^th week of each year, which corresponds to late summer and autumn in China. The annual average incidence was 0.023 per 100,000 between 2009 and 2012, and the case numbers were higher in 2009 and 2012 (Figure 1). A total of 147 signals for DF outbreaks were generated in CIDARS, of which 91.8% (135) occurred from the 32^nd week to the 49^th week between 2009 and 2012 (Figure 1), which was consistent with the DF case occurrence. Twenty-six of China’s 31 provinces reported DF cases and signals were generated in 11 provinces (Figure 2). Most cases were reported in Guangdong (682, 55.72%) and Zhejiang (228, 18.63%). Accordingly, most signals were generated in Guangdong (121, 82.31%) and Zhejiang (7, 4.76%), respectively. Overall, there were 30 outbreaks reported from 4 provinces located in the south-eastern coastal regions, including 20 outbreaks in Guangdong, 7 in Zhejiang, 2 in Guangxi and 1 in Fujian (Figure 2), lasting 405 days and including 325 cases. CIDARS successfully detected all the 30 outbreaks, with a sensitivity of 100% and a specificity of 99.8%. The median time to detection (TTD) of all outbreaks was 3 days, with an interquartile range of 4 days (Table 1).

[Figure omitted. See PDF.]

Figure 1. The time distribution of dengue fever cases and signals between 2009 and 2012 in China.

https://doi.org/10.1371/journal.pone.0106144.g001

[Figure omitted. See PDF.]

Figure 2. The spatial distribution of dengue fever cases and signals between 2009 and 2012 in China.

https://doi.org/10.1371/journal.pone.0106144.g002

[Figure omitted. See PDF.]

Table 1. Dengue fever outbreaks between 2009 and 2012 in China.

https://doi.org/10.1371/journal.pone.0106144.t001

Discussion

Our study found that CIDARS successfully detected all the DF outbreaks and had a high specificity and timeliness, by adopting a simple and convenient algorithm to automatically generate the early warning signals when aberrations of cases occurred.

In the field of outbreak detection system evaluation, as the true outbreak is nearly impossible to obtain in the real world, the reported outbreaks with high reporting quality are commonly adopted as the reference standards to evaluate the system [4], [13]–[15]. In China, as DF is a notifiable and concerning disease, once getting information on potential outbreaks reported from clinical institutes, media, and community, etc, the public health staff of local CDC immediately launches verification and investigation. Then local CDC reports all the confirmed outbreaks to China CDC according to the outbreak definition issued by Chinese Ministry of Health. Therefore, we supposed the reported outbreaks were the outbreak data source most approximating to the true outbreaks situation, and adopted the reported outbreaks as the reference standard for calculating the sensitivity, specificity and timelines of outbreak detection. Certainly, in practice, it is possible that an outbreak occurred but no cases were reported, which lead to that outbreak would not be captured by CIDARS. Therefore, we regard the quality of case reporting and outbreak reporting as critical to evaluate the performance of CIDARS. By using the reported cases and outbreaks during the 3 years of 2009–2012 in this study, we found that CIDARS could detect all the reported outbreaks during the study period, which were currently regarded as 100% sensitivity of outbreak detection for CIDARS. We admit that, as the number of reported outbreaks during study period is limited, which may not represent the true situation of DF outbreaks in China, more data over a longer time period should be adopted to prospectively validate and evaluate the sensitivity of CIDARS for DF in the future.

As for all early detection tools, the trigger threshold for a signal should be set according to the practical requirements of outbreak detection and response under local conditions. Taking account of the high priority placed on control and prevention of DF in China, with the aim to catch as many outbreaks as possible in the preliminary phase of CIDARS operation, it was decided that the CIDARS needed a relatively low threshold leading to a high sensitivity of the algorithm. Following the advice of senior epidemiologists and the CIDARS research group, the 50th percentile (the median value) of historical data was defined as the trigger threshold, which means that when the current disease incidence level reached the median of corresponding historical baseline incidence level, local public health should be vigilant. Actually, this study was a prospective analysis of a real-world system rather than a retrospective analysis testing different theoretical thresholds. Future evaluation of CIDARS could make use of more cases and outbreaks data collected over a longer period to further assess the performance of CIDARS using different thresholds, helping us optimize the trigger threshold by ROC graph [16] and detection timeliness.

In this study, we found that the timeliness of outbreak detection by CIDARS varied distinctly among outbreaks with similar size, for example, both outbreak ID 9 and 27 have low total number of cases for the entire outbreaks and low number of cases reported on both the first and last days of outbreak, and TTD varied significantly between them. The major cause is that the generation of a signal by CIDARS was affected by many factors, including the current disease incidence level, the spatiotemporal distribution of reported cases, and the incidence level in historical baseline, which may lead to the variation on the timeliness of outbreak detection of CIDARS. We will consider further validating and improving the algorithms of outbreak detection by taking into account various level of dengue incidence.

In this study, all the dengue cases, including imported and local DF cases, were used to detect DF outbreaks. However, for the no local dengue transmission regions where only imported dengue cases occurred, or with few or no indigenous cases, the signals generated by CIDARS may only show the imported dengue occurring, not the dengue fever outbreak in these regions. That may be why the signals were generated in 11 provinces, while DF outbreaks just occurred in 4 provinces which are local dengue transmission regions. One limitation of CIDARS is that it is difficult to predict DF outbreaks before cases are diagnosed and reported, because this system is only based on numbers of notified DF cases. However, it is possible to incorporate vector surveillance data or data pertaining to environmental and social drivers of disease risk in CIDARS using spatiotemporal models, which could lead to a more rapid response than is possible with the current system [17]–[19]. In the future, an early detection system based on socio-environmental factors for DF should be developed and integrated into CIDARS using rigorously evaluated modelling methods and processes.

The findings of this study indicated that CIDARS had good sensitivity, specificity and timeliness of DF outbreak detection, and the system could act as a tool to assist early detection on outbreaks for the local public health staff in China. CIDARS automatically carried out abnormality detection on case occurrence data, and reminded the local CDC to verify aberrations by distributing signals to them. CIDARS assisted the local public health staff to detect potential outbreaks early, and even confirm some outbreaks which passive report did not detect. In future research, we will evaluate the impact of CIDARS on the number and size of dengue outbreaks with more long-term data, so as to show the significance of CIDARS on recognition and response to DF outbreaks.

Mr Honglong Zhang is a Research Intern of Epidemiology and Health Statistics in the Division of Infectious Diseases, Chinese Center for Disease Control and Prevention. He is mainly engaged in the research of disease surveillance and outbreak detection algorithms, and developing the application tools of outbreak early detection.

Author Contributions

Conceived and designed the experiments: WZY ZJL. Performed the experiments: HLZ SJL. Analyzed the data: ACAC LPW. Contributed reagents/materials/analysis tools: WWY HZ HJY WBH. Contributed to the writing of the manuscript: ZJL WBH. Designed the steps of analysis: HLZ ZJL.

Citation: Zhang H, Li Z, Lai S, Clements ACA, Wang L, Yin W, et al. (2014) Evaluation of the Performance of a Dengue Outbreak Detection Tool for China. PLoS ONE 9(8): e106144. https://doi.org/10.1371/journal.pone.0106144

References

1. Hutwagner LC, Thompson W, Seeman GM, Treadwell T (2003) The bioterrorism preparedness and response Early Aberration Reporting System (EARS). J Urban Health 80, i89–i96.

2. Tsui FC, Espino JU, Dato VM, Gesteland PH, Hutman J, et al. (2003) Technical description of RODS: a real-time public health surveillance system. J Am Med Inform Assoc 10, 399–408.

3. Lombardo J, Burkom H, Elbert E, Magruder S, Lewis SH, et al. (2003) A systems overview of the Electronic Surveillance System for the Early Notification of Community-Based Epidemics (ESSENCE II). J Urban Health 80, i32–i42.

4. Krause G, Altmann D, Faensen D, Porten K, Benzler J, et al. (2007) SurvNet electronic surveillance system for infectious disease outbreaks, Germany. Emerg Infect Dis 13, 1548–1555.

5. World Health organization. Special Programme for Research and Training in Tropical Disease. Dengue: guidelines for diagnosis, treatment, prevention and control (New edition 2009) Available: http://whqlibdoc.who.int/publications/2009/9789241547871_eng.pdf. Accessed 2014 April 7.

6. Zhao HL, Luo QH, Shen G (1981) Epidemiology of the dengue outbreak in Shiwanzhen, Nanhai County, Guangdong Province. Chin Med J 61, 466–469.

7. Wu JY, Lun ZR, James AA, Chen XG (2010) Review: Dengue Fever in mainland China. Am J Trop Med Hyg 83, 664–671.

8. Yang WZ, Li ZJ, Lan YJ, Wang JF, Ma JQ, et al. (2011) A nationwide web-based automated system for outbreak early detection and rapid response in China. WPSAR Vol 2, No 1. 2011 | doi:10.5365/wpsar.2010.1.1.009.

9. Wang Y (2013) The H7N9 Influenza Virus in China – Changes since SARS. N Engl J Med 368, 2348–2349.

10. Centers for Disease Control and Prevention (2009) Notifiable diseases/deaths in selected cities weekly information. MMWR Morb Mortal Wkly Rep 58, 1076–1087.

11. Ministry of Health of the People’s Republic of China. National dengue fever surveillance program [in Chinese] Available: www.moh.gov.cn/uploadfile/2005818141858129.doc. Accessed 2013 March 20.

12. Wang XL, Zeng D, Seale H, Li S, Cheng H, et al. (2010) Comparing early outbreak detection algorithms based on their optimized parameter values. J Biomed Inform 43, 97–103.

13. Watkins RE, Eagleson S, Veenendaal B, Wright G, Plant AJ (2008) Applying cusum-based methods for the detection of outbreaks of Ross River virus disease in Western Australia. BMC Med Inform Decis Mak 8, 37.

14. Zhang HL, Lai SJ, Wang LP, Zhao D, Zhou DL, et al. (2013) Improving the Performance of Outbreak Detection Algorithms by Classifying the Levels of Disease Incidence. PLoS ONE 8, e71803.

15. Li ZJ, Lai SJ, Buckeridge DL, Zhang HL, Lan YJ, et al. (2011) Adjusting Outbreak Detection Algorithms for Surveillance during Epidemic and Non-epidemic Periods. J Am Med Inform Assoc 2012 19: e51ee53

16. Lowe R, Bailey TC, Stephenson DB, Jupp TE, Graham RJ, et al. (2013) The development of an early warning system for climate-sensitive disease risk with a focus on dengue epidemics in Southeast Brazil. Statistics in Medicine 32: 864–883

17. Mohammad J, Dana F (2009) Early warning systems for dengue in Indonesia and Thailand. Berkala IlmuKedoktera, 41, 134–142.

18. Depradine C, Lovell E (2004) Climatological variables and the incidence of Dengue fever in Barbados. Int J Environ Health Res 14, 429–441.

19. Sithiprasasna R, Patpoparn S, Attatippaholkun W, Suvannadabba S, Srisuphanunt M (2004) The geographic information system as an epidemiological tool in the surveillance of dengue virus-infected Aedes mosquitos. SE ASIAN J TROP MED 35, 918–926.