Detection of Diabetic Retinopathy Using Bichannel Convolutional Neural Network

Abstract

Deep learning of fundus photograph has emerged as a practical and cost-effective technique for automatic screening and diagnosis of severer diabetic retinopathy (DR). The entropy image of luminance of fundus photograph has been demonstrated to increase the detection performance for referable DR using a convolutional neural network- (CNN-) based system. In this paper, the entropy image computed by using the green component of fundus photograph is proposed. In addition, image enhancement by unsharp masking (UM) is utilized for preprocessing before calculating the entropy images. The bichannel CNN incorporating the features of both the entropy images of the gray level and the green component preprocessed by UM is also proposed to improve the detection performance of referable DR by deep learning.

This work is licensed under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.

1. Introduction

Retinopathy often refers to retinal microvascular damage resulted from abnormal blood flow and may cause visual impairment. Frequently, retinopathy is an ocular manifestation of diabetes or hypertension. It is predicted that around 600 million people will have diabetes by 2040, with one-third estimated to have diabetic retinopathy (DR). Early detection of DR by regular clinical examination and prompt treatment are essential for the prevention of vision impairment and to raise living quality [1–4]. Fundus photography has been widely used worldwide to be an ophthalmologic screening tool utilized for detecting DR [5]. Retinal telescreening with remote interpretation by an expert for evaluation of DR may be useful in helping rural and medically underserved patients [6]. However, some diabetic patients cannot afford the cost of an ophthalmologist visit [7]. In addition, the assessment of DR severity needs specialized expertise, and the agreement of interpretation results may vary from the graders [8]. Automated assessment systems for DR image may provide clinically effective and cost-effective detection of retinopathy and therefore help the prevention of diabetic-associated blindness [9].

Artificial intelligence has the potential to revolutionize the traditional diagnosis method for eye disease and bring out a significant clinical impact on promoting ophthalmic health care service [10–13]. Automated DR detections have been previously studied [14–18]. Deep learning of fundus photograph has emerged as a practical technique for automatic screening and diagnosis of DR. The effective deep learning system is able to correctly and automatically identify severer DR with equal or better accuracy than the trained graders and retina specialists [19], and thus, it can benefit the patients in medically underserved areas that have limited numbers of ophthalmologists and rare medical resources. The convolutional neural network (CNN), a core model of deep learning in computer vision, has yielded impressive results in terms of prediction and diagnosis in medical image classification. The entropy image of luminance of fundus photograph, which involves measuring the complexity of a retinal image, has been demonstrated to increase the detection performance of referable DR for a CNN-based system in [18].

Several image processing methods have been discussed for the retinal image to meliorate microaneurysm detection in [20]. An image enhancement method is proposed to increase the contrast and improve the overall appearance of retinal image by taking the information of color models in [21]. The green component of the RGB retinal image is used for preprocessing of improved blood vessel and optic disc segmentation in [22]. In [23], the green component of retinal image is also used to train a network to segment the macular region. In this paper, we extract the green component of color fundus photograph and enhance the details by unsharp masking (UM), a classical tool for sharpness enhancement [24] and has been applied to fundus photograph [25, 26] and medical images [27, 28], before calculating the entropy images. The proposed bichannel CNN is trained by incorporating the features of both the entropy images from the gray level and the green component of fundus photograph preprocessed by UM to heighten the detection of referable DR.

2. Materials and Methods

2.1. Dataset and Grading

The total of 35,126 color fundus photographs with the sizes from 433 × 289 pixels to 5184 × 3456 pixels is obtained from the publicly available “Kaggle Diabetic Retinopathy” dataset [29–34], which is acquired by using various digital fundus cameras in several eye centers in California and around the United States. We select 21,123 color fundus photographs with good image quality. The experimental setup is the same with that in [18].

The retinal images obtained from the Kaggle dataset have been independently graded by well-trained clinicians according to the International Clinical Diabetic Retinopathy Disease Severity Scale: no apparent retinopathy (grade 0), mild nonproliferative DR (grade 1), moderate nonproliferative DR (grade 2), severe nonproliferative DR (grade 3), and proliferative DR (grade 4) [35]. The image numbers for grade 0, grade 1, grade 2, grade 3, and grade 4 are 16,500, 1,333, 2,000, 645, and 645, respectively. Referable DR is defined as the presence of severe DR grades 2–4 (3,290 images, 15.6%) that requires a referral to the eye specialist and used as the output for the deep learning in this paper.

2.2. Data Augmentation

The 21,123 eligible color fundus photographs are resized to a resolution of 100 × 100 pixels. The resized images are increased by data augmentation using flipping and rotation. The retinal images of grade 1–grade 4 are randomly selected from the augmented images to the numbers 4,375 (13.26%), 4,375 (13.26%), 3,875 (11.74%), and 3,875 (11.74%), respectively, for a total of 16,500 images balanced with grade 0 (16500, 50%). In total, 33,000 images are used for experiments. To compose a total of 30,000 retinal images in the training set, 15,000 images of grade 0 and 15,000 images of grade 1–grade 4 (4,000, 4,000, 3,500, and 3,500, respectively) are randomly chosen. Then, the remaining 1,500 images of grade 0 and the 1,500 images of grade 1–grade 4 (375, 375, 375, and 375, respectively) are utilized as the test set.

2.3. Preprocessing of Retinal Images

Figure 1 illustrates the dataflow of preprocessing for the retinal images in the proposed method. After resizing to a resolution of 100 × 100 from original retinal fundus photograph, the green component is extracted from the retinal image with the RGB color model. The luminance conversion is calculated from red (R), green (G), and blue (B) components of color retinal fundus photograph by equation (1) to obtain the gray level image as in [18].$$\begin{array}{}\text{(1)}& \text{Gray\hspace{0.17em}Level}=0.299\ast R+0.587\ast G+0.114\ast B.\end{array}$$

[figure omitted; refer to PDF]

In order to enhance of the details of the retinal image, the UM technique is utilized to amplify the high-frequency parts of the gray level (luminance) and the green component of the retinal image before computing the entropy images. An unsharp mask is obtained by subtracting a Gaussian blurred image from the original image. The unsharp mask contains high-frequency information associated with edges. Then, a scaled mask is added to the original image to create an enhanced image.

The entropy image of luminance of fundus photograph represents the complexity of the original retinal image and benefits the training of the CNN-based deep learning system [18]. The values in the entropy image are calculated locally from n x n blocks to measure the heterogeneity. The entropy is a function of the probability distribution of the local intensity. Equations (2) and (3) represent the values in the entropy images for the two inputs of the proposed bichannel CNN.$$\begin{array}{}\text{(2)}& E_{\text{gray}}=-{\displaystyle \sum }_i{P}_{\text{gray}\_\text{UM}}\left(i\right)\times {\log }_2{P}_{\text{gray}\_\text{UM}}\left(i\right),\text{(3)}& E_{\text{green}}=-{\displaystyle \sum }_i{P}_{\text{green}\_\text{UM}}\left(i\right)\times {\log }_2{P}_{\text{green}\_\text{UM}}\left(i\right),\end{array}$$where P_{gray_UM} (i) and P_{green_UM} (i) denote the relative frequencies associated with the i-th intensity within a n x n block in the gray level and the green component of retinal image, respectively, after processing by UM. Since the result of n = 9 reaches the maximal accuracy among the various block sizes as exhibited in [18], accordingly, n = 9 is also chosen to calculate the entropy images of the gray level and the green component of retinal image after applying UM in our experiments. The entropy images use the statistical characteristics of the local areas and present the local structural information of the retinal images. The pixels of the entropy image with intensities between 0 and 255 are rescaled to the values between 0 and 1 to be the CNN inputs.

2.4. Deep Learning by Bichannel Convolutional Neural Network

The convolutional neural network (CNN) is used for the feature learning of referable DR in this study. We construct a bichannel CNN model to simultaneously process the entropy images of luminance (gray level) and the green component after processing by UM as shown in Figure 2. For each channel, 4 convolutional layers are with 5 × 5 kernels, and the numbers of filters are 32, 64, 64, and 128 in successive layers. Maximum pooling, rectified linear unit activation function, and dropout (set to 0.3), to prevent overfitting, are used. After flattening from the two channels, the fully connected layers are linked to statistically determine the detection of referable DR. The proposed referable DR detection method is coded by TensorFlow software with Python. The cross-entropy loss function and the Adam algorithm with learning rate 0.0001 are adopted for training the network.

[figure omitted; refer to PDF]

3. Results and Discussion

Performance evaluation consists of several standard measurements including accuracy, sensitivity, specificity, and the area under the receiver-operating characteristic curve (AUC of the ROC curve) of the automatic screening for the presence of referable DR. We use the clinically defined referable DR in the Kaggle dataset as the benchmark to validate the proposed algorithm.

Table 1 compares the detection accuracy, sensitivity, and specificity for referable DR by various retinal image inputs to the CNN. As revealed in [18], the result of the entropy image of the gray level outperforms that of the original photograph. The proposed method utilizing the entropy image of the green component provides better accuracy and sensitivity than the entropy of luminance in the fundus photograph. Employing the entropy image of the green component can improve the accuracy and prevent under diagnosis by elevating sensitivity with a negligible loss of specificity. By applying the bichannel CNN with the two inputs of the entropy images from the gray level and green component, the performance is better than that of the single-channel CNN of individual input.

Table 1

Performance comparison of various input images of the CNN for referable DR.

Input images of the CNN	Accuracy (%)	Sensitivity (%)	Specificity (%)
Original photograph [18]	81.80	68.36	89.87
Entropy image of the gray level [18]	86.10	73.24	93.81
Entropy image of the green component	87.27	76.70	93.07
Bichannel, entropy images (gray and green)	87.37	76.93	93.57
Entropy images of unsharp masking gray level	86.87	75.09	93.86
Entropy image of unsharp masking green component	87.41	76.75	93.28
Bichannel, entropy images (unsharp masking gray level and green component)	87.83	77.81	93.88

All of accuracy, sensitivity, and specificity increase when the entropy image is obtained from the preprocessed gray level or green component by UM. Since the contrast is enhanced by UM, the corresponding bichannel CNN yields the best results for deep learning. The measurements of the proposed bichannel CNN model regarding accuracy, sensitivity, and specificity are 87.83%, 77.81%, and 93.88%, respectively, which are better than 86.10%, 73.24%, and 93.81%, respectively, of the previous study [18], which only implements the single-channel CNN by training the entropy image of the gray level of fundus photograph alone.

Furthermore, the AUC of the ROC curve is used as the integral performance index. As shown in Figure 3, the proposed bichannel CNN method obtains the AUC of 0.93, which is better than 0.87 of the CNN trained by original photograph.

[figure omitted; refer to PDF]

Figure 4 shows the color fundus photograph of DR grade 3 with the resolution of 2592 × 1944, the respective intensities of monochromatic R, G, and B components, and the histograms. Excluding the peak values for all of R, G, and B components with very low intensities in the histograms resulting from the dark background, broader distribution of the green component is observed from Figure 4(e). The histogram in Figure 4(e) can depict the amount of contrast, which is the measurement of brightness difference between light and dark regions in the retinal image. Broad histogram expresses the image with significant contrast, whereas narrow histogram represents less contrast.

[figures omitted; refer to PDF]

From the observation of red, green, and blue components for monochromatic fundus photograph in previous studies [36, 37], green light provides the best overall view of the retina and displays excellent contrast because the retinal pigmentations reflect green light more than blue light. Hence, green filter is utilized for enhancing the visibility of retinal vasculature, drusen, hemorrhage, and exudate. This finding motivates us to extract the green component from color fundus photograph before calculating the entropy image for the input of the CNN. It benefits the learning of the CNN to recognize lesions using the local features by entropy images.

Figure 5 illustrates the resized, preprocessed, and entropy images in the dataflow of the proposed system for Figure 4(a). Our proposed approach utilizes UM to increase contrast, which can display a significant visual impact by emphasizing texture in the retinal image. From Figures 5(f) and 5(h), more structural information is enhanced by UM than from Figures 5(b) and 5(d). Severer DR gives rise to higher heterogeneity than mild or no DR in a retinal image. To discriminate the characteristics of no or mild DR and severer DR, the complexities of the gray level and green component images are analyzed by computing local entropy. The low-entropy image has low complexity; instead, the high-entropy image represents high complexity among neighboring pixels. Severer DR images may contain neovascularization or the lesions more than just microaneurysms and thus have more heterogeneous areas with high local entropy values; on the contrary, no or mild DR images may have more homogenous regions with low local entropy values.

[figures omitted; refer to PDF]

Based on the training of the CNN by the entropy image of the gray level as shown in [18], the green component, UM, and bichannel CNN model are incorporated in this study to improve the detection performance and may assist ophthalmologists in evaluating retinal images for more accurate diagnoses.

4. Conclusions

A deep learning system can increase the accuracy for detecting or diagnosing retinal pathologies in patients with diabetes. The methodology of the proposed method first includes the green component of the RGB image. The entropy image of the green component can improve the accuracy and the sensitivity. Preprocessing by UM can provide better detection accuracy, sensitivity, and specificity. The bichannel CNN with the inputs of both the entropy images of the gray level and the green component preprocessed by UM further advances the detection of referable DR. The proposed deep learning technology can assist ophthalmologists in referable DR diagnosis and will be beneficial to the automated retinal image analysis system.

Acknowledgments

This work was supported by the Ministry of Science and Technology, Taiwan, R.O.C. (Grant number: MOST 107-2221-E-899-002-MY3).

References

[1] J. W. Y. Yau, S. L. Rogers, R. Kawasaki, "Global prevalence and major risk factors of diabetic retinopathy," Diabetes Care, vol. 35 no. 3, pp. 556-564, DOI: 10.2337/dc11-1909, 2012.

[2] D. S. W. Ting, G. C. M. Cheung, T. Y. Wong, "Diabetic retinopathy: global prevalence, major risk factors, screening practices, and public health challenges: a review," Clinical & Experimental Ophthalmology, vol. 44 no. 4, pp. 260-277, DOI: 10.1111/ceo.12696, 2016.

[3] G. M. Lin, S. Redline, R. Klein, "Sex-specific association of obstructive sleep apnea with retinal microvascular signs: the multiethnic study of atherosclerosis," Journal of the American Heart Association, vol. 5 no. 7, 2016.

[4] L. Lu, Y. Jiang, R. Jaganathan, "Current advances in pharmacotherapy and technology for diabetic retinopathy: a systematic review," Journal of Ophthalmology, vol. 2018,DOI: 10.1155/2018/5047142, 2018.

[5] M. M. Khansari, W. D. O’Neill, R. D. Penn, "Detection of subclinical diabetic retinopathy by the fine structure analysis of retinal images," Journal of Ophthalmology, vol. 2019,DOI: 10.1155/2019/5171965, 2019.

[6] P. D. Jani, L. Forbes, A. Choudhury, J. S. Preisser, A. J. Viera, S. Garg, "Evaluation of diabetic retinal screening and factors for ophthalmology referral in a telemedicine network," JAMA Ophthalmology, vol. 135 no. 7, pp. 706-714, DOI: 10.1001/jamaophthalmol.2017.1150, 2017.

[7] L. Z. Wang, C. Y. Cheung, R. J. Tapp, "Availability and variability in guidelines on diabetic retinopathy screening in Asian countries," British Journal of Ophthalmology, vol. 101 no. 10, pp. 1352-1360, DOI: 10.1136/bjophthalmol-2016-310002, 2017.

[8] L. Sellahewa, C. Simpson, P. Maharajan, J. Duffy, I. Idris, "Grader agreement, and sensitivity and specificity of digital photography in a community optometry-based diabetic eye screening program," Clinical Ophthalmology (Auckland, N.Z.), vol. 8, pp. 1345-1349, 2014.

[9] A. Tufail, C. Rudisill, C. Egan, "Automated diabetic retinopathy image assessment software," Ophthalmology, vol. 124 no. 3, pp. 343-351, DOI: 10.1016/j.ophtha.2016.11.014, 2017.

[10] W. Lu, Y. Tong, Y. Yu, "Applications of artificial intelligence in ophthalmology: general overview," Journal of Ophthalmology, vol. 2018,DOI: 10.1155/2018/5278196, 2018.

[11] L. Balyen, T. Peto, "Promising artificial intelligence-machine learning-deep learning algorithms in ophthalmology," Asia-Pacific Academy of Ophthalmology, vol. 8 no. 3, pp. 264-272, 2019.

[12] Y. Fujinami-Yokokawa, N. Pontikos, L. Yang, "Prediction of causative genes in inherited retinal disorders from spectral-domain optical coherence tomography utilizing deep learning techniques," Journal of Ophthalmology, vol. 2019, 2019.

[13] E. Vaghefi, S. Hill, H. M. Kersten, "Multimodal retinal image analysis via deep learning for the diagnosis of intermediate dry age-related macular degeneration: a feasibility study," Journal of Ophthalmology, vol. 2020, 2020.

[14] V. Gulshan, L. Peng, M. Coram, "Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs," JAMA, vol. 316 no. 22, pp. 2402-2410, DOI: 10.1001/jama.2016.17216, 2016.

[15] R. Gargeya, T. Leng, "Automated identification of diabetic retinopathy using deep learning," Ophthalmology, vol. 124 no. 7, pp. 962-969, DOI: 10.1016/j.ophtha.2017.02.008, 2017.

[16] M. D. Abràmoff, Y. Lou, A. Erginay, "Improved automated detection of diabetic retinopathy on a publicly available dataset through integration of deep learning," Investigative Opthalmology & Visual Science, vol. 57 no. 13, pp. 5200-5206, DOI: 10.1167/iovs.16-19964, 2016.

[17] D. S. W. Ting, C. Y.-L. Cheung, G. Lim, "Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes," JAMA, vol. 318 no. 22, pp. 2211-2223, DOI: 10.1001/jama.2017.18152, 2017.

[18] G.-M. Lin, M.-J. Chen, C.-H. Yeh, "Transforming retinal photographs to entropy images in deep learning to improve automated detection for diabetic retinopathy," Journal of Ophthalmology, vol. 2018,DOI: 10.1155/2018/2159702, 2018.

[19] V. Gulshan, R. P. Rajan, K. Widner, "Performance of a deep-learning algorithm vs manual grading for detecting diabetic retinopathy in India," JAMA Ophthalmology, vol. 137 no. 9, pp. 987-993, DOI: 10.1001/jamaophthalmol.2019.2004, 2019.

[20] B. Antal, A. Hajdu, "An ensemble-based system for microaneurysm detection and diabetic retinopathy grading," IEEE Transactions on Biomedical Engineering, vol. 59 no. 6, pp. 1720-1726, DOI: 10.1109/tbme.2012.2193126, 2012.

[21] M. Zhou, K. Jin, S. Wang, J. Ye, D. Qian, "Color retinal image enhancement based on luminosity and contrast adjustment," IEEE Transactions on Biomedical Engineering, vol. 65 no. 3, pp. 521-527, DOI: 10.1109/tbme.2017.2700627, 2018.

[22] S. Kumar, A. Adarsh, B. Kumar, "An automated early diabetic retinopathy detection through improved blood vessel and optic disc segmentation," Optics And Laser Technology, vol. 121,DOI: 10.1016/j.optlastec.2019.105815, 2020.

[23] R. Alais, P. Dokládal, A. Erginay, "Fast macula detection and application to retinal image quality assessment," Biomedical Signal Processing and Control, vol. 55,DOI: 10.1016/j.bspc.2019.101567, 2020.

[24] G. Deng, "A generalized unsharp masking algorithm," IEEE Transaction on Image Processing, vol. 20 no. 5, pp. 1249-1261, 2011.

[25] L. Frisen, W. F. Hoyt, "Unsharp masking in fundus photography," Investigate Ophthalmology & Visual Science, vol. 12 no. 6, pp. 461-464, 1973.

[26] C. J. Barry, R. L. Cooper, R. H. Eikelboom, "Simplification of unsharp masking in retinal nerve fibre layer photography," Australian and New Zealand Journal of Ophthalmology, vol. 18 no. 4, pp. 411-420, DOI: 10.1111/j.1442-9071.1990.tb01827.x, 1990.

[27] J. Rogowska, K. Preston, D. Sashin, "Evaluation of digital unsharp masking and local contrast stretching as applied to chest radiographs," IEEE Transactions on Biomedical Engineering, vol. 35 no. 10, pp. 817-827, DOI: 10.1109/10.7288, 1988.

[28] S. Chen, Y. Cai, "Enhancement of chest radiograph in emergency intensive care unit by means of reverse anisotropic diffusion-based unsharp masking model," Diagnostics, vol. 9 no. 2,DOI: 10.3390/diagnostics9020045, 2019.

[29] Kaggle, "Diabetic Retinopathy Detection," 2017. https://www.kaggle.com/c/diabetic-retinopathy-detection

[30] M. J. J. P. van Grinsven, B. van Ginneken, C. B. Hoyng, "Fast convolutional neural network training using selective data sampling: application to hemorrhage detection in color fundus images," IEEE Transactions on Medical Imaging, vol. 35 no. 5, pp. 1273-1284, DOI: 10.1109/tmi.2016.2526689, 2016.

[31] G. Quellec, K. Charrière, Y. Boudi, B. Cochener, M. Lamard, "Deep image mining for diabetic retinopathy screening," Medical Image Analysis, vol. 39, pp. 178-193, DOI: 10.1016/j.media.2017.04.012, 2017.

[32] C. Lam, C. Yu, L. Huang, D. Rubin, "Retinal lesion detection with deep learning using image patches," Investigative Opthalmology & Visual Science, vol. 59 no. 1, pp. 590-596, DOI: 10.1167/iovs.17-22721, 2018.

[33] A. Diaz-Pinto, A. Colomer, V. Naranjo, S. Morales, Y. Xu, A. F. Frangi, "Retinal image synthesis and semisupervised learning for glaucoma assessment," IEEE Transactions on Medical Imaging, vol. 38 no. 9, pp. 2211-2218, DOI: 10.1109/tmi.2019.2903434, 2019.

[34] S. Sengupta, A. Singha, H. A. Leopold, "Ophthalmic diagnosis using deep learning with fundus images:a critical review," Artificial Intelligence in Medicine, vol. 102,DOI: 10.1016/j.artmed.2019.101758, 2020.

[35] American Academy of Ophthalmology, "International clinical diabetic retinopathy disease severity scale detailed table," 2017. https://www.icoph.org/dynamic/attachments/resources/diabetic-retinopathy-detail.pdf

[36] T. J. Bennett, C. J. Barry, "Ophthalmic imaging today: an ophthalmic photographer’s viewpoint-a review," Clinical & Experimental Ophthalmology, vol. 37 no. 1,DOI: 10.1111/j.1442-9071.2008.01812.x, 2009.

[37] M. Badar, M. Harisa, A. Fatima, "Application of deep learning for retinal image analysis: a review," Computer Science Review, vol. 35, 2020.