Road Pavement Damage Detection Based on Local Minimum of Grayscale and Feature Fusion

Abstract

In this work, we propose a road pavement damage detection deep learning model based on feature points from a local minimum of grayscale. First, image blocks, consisting of the neighborhood of feature points, are cut from the image window to form an image block dataset. The image blocks are then input into a convolutional neural network (CNN) to train the model, extracting the image block features. In the testing process, the feature points as well as the image blocks are selected from a test image, and the trained CNN model can output the feature vectors for these feature image blocks. All the feature vectors will be combined to a composite feature vector as the feature descriptor of the test image. At last, the classifier of the model, constructed by a support vector machine (SVM), gives the classification as to whether the image window contains damaged areas or not. The experimental results suggest that the proposed pavement damage detection method based on feature-point image blocks and feature fusion is of high accuracy and efficiency. We believe that it has application potential in general road damage detection, and further investigation is desired in the future.

1. Introduction

Pavement detection and maintenance is supported by advanced technology, such as image sensing, big data and AI, to ensure the safety and transportation efficiency of the roads. Nowadays, with the rapid development of image acquisition techniques, a global shutter camera can realize fast imaging with a lower price than that of a ground-penetrating radar and laser system. However, the automatic processing algorithm of pavement images still remains challenging [1,2,3,4]. Considering the difference in the pavement circumstance and the economic level of various areas, there is practical significance to study adaptable pavement detection methods [5].

So far, road pavement detection technology has been developed, from traditional image processing methods to deep learning techniques [6]. The traditional methods for image segmentation, including threshold image segmentation, an edge detection algorithm, texture segmentation and multi-feature fusion methods, are realized by an artificially designed feature [7,8]. Therefore, they work out good results on specific datasets. Among them, Nayyeri et al. proposed a method for crack detection via a salient structure extraction from a textured background, which extracts strong edges and distinguishes them from strong textures in a local neighborhood [9]. Yousaf et al. employed and computed famous scale-invariant feature transform (SIFT) features to establish the visual vocabulary of words to represent the pavement surface and employed a support vector machine (SVM) for the training and testing of potholes in pavement images [10]. Jin et al. proposed a pavement crack detection method fused with a histogram of oriented gradients and a watershed algorithm of a range image [11]. Patel et al. evaluated two machine learning algorithms for surface roughness based on statistical parameters [12].

In recent years, several road damage detection methods based on a convolution neural network (CNN) have been proposed and demonstrated to have good effects [13], including ASINVOS net [14], VGG [15], FS-Net [16] and YOLO [17]. For instance, Xiang et al. proposed a pavement crack detection method based on an end-to-end trainable deep CNN [18]. A deep learning algorithm significantly reduces the cost of the feature description, extraction and recognition, which further improves the accuracy and versatility of pavement detection methods [19,20]. A deep learning neural network relies on its powerful fitting ability to realize pavement detection. However, training with high-resolution images, which contain much more information about the pavement, requires plenty of memory and computing power. The effective way is to locate the target of the high-resolution images before training the neural networks [21]. Nevertheless, the size of the damaged areas varies dramatically, which also have fuzzy boundaries. As a result, to pick up the damaged areas precisely from pavement images seems very difficult [22].

It is noticed that the grayscale of damaged areas is obviously lower than that of normal pavement, with apparent cracks and potholes. Dependent on this intuitive information, damaged areas could be separated roughly from the raw data and then be recognized by the existing detection methods. In this paper, we propose a road pavement damage detection deep learning model based on feature points from a local minimum of grayscale. First, the local minimum points of grayscale in an image window are selected to be the feature points according to the grayscale distribution. The image blocks, the neighborhood of these feature points, are then cut from the image window as individuals, because they contain the main information about the damage. The image blocks, with a small size, are collected as a dataset to train the CNN model, which could extract the deep feature of the damage. After that, the trained CNN model can be used to detect the practical pavement images. Similarly, based on the local minimum grayscale feature points, a few image blocks of testing images are selected and then input into the trained CNN model. The trained CNN model can output a feature vector for each image block. These feature vectors will be combined to a composite feature vector, which is the feature descriptor of the whole image window. Finally, the composite feature vector is input to the classifier constructed by the SVM, to detect whether there are damaged areas in the image window. The proposed detection method exhibits high accuracy with little cost of computing resources, and our study offers possible solutions to the demand for fast detection at mobile terminals in the future. The methodology used to detect the pavement damage is shown in Figure 1.

2. Methods

2.1. Feature Points Based on Local Minimum of Grayscale

Computer vision usually relies on the local analyses of feature points, such as corner, spot and edge, to achieve object recognition and image matching [23]. Therefore, there has to be enough feature points with high degree of discrimination, which can be located clearly. In the image of road pavement, the grayscale values of cracks and caves are significantly less than those of normal pavement. Feature points based on the local minimum of grayscale can be used for block segmentation. To obtain the feature points with local minimum of grayscale, one should first calculate gradient of each pixel point and find the zero-gradient points with positive and negative gradient around them. In practical cases, some methods have been proposed to extract these feature points. For instance, the local minimum of grayscale is identified by two-dimensional gradients at four directions based on bidimensional empirical mode decomposition (BEMD) [24]. Moreover, a morphological grayscale reconstruction method was used to search for the local maximum of grayscale [25].

In this work, we propose an improved method to obtain these feature points with local minimum of grayscale as follows. First, the pavement images are preprocessed via reversing the grayscale of each pixel point. Then, we search for the local maximum, namely the local minimum of grayscale, in all the image windows. In detail, an amplitude value of each image window is set to adjust various brightness by

(1)$P_{\mathrm{A}}=P_{\mathrm{th}}\times (G_{\max }-G_{\min })+G_{\min },$

where $P_{\mathrm{th}}=0.1$ is the detection threshold value, and $G_{\max }$ and $G_{\min }$ are the maximum and minimum of grayscale in image window. Then, the first-order difference in all the pixel points is calculated, and all the points of zero first-order difference with opposite signs of first-order difference around them are selected. Among these picked points, we choose the points whose grayscale is larger than the amplitude $P_{\mathrm{A}}$ as peak points. Further, all the peak points are ordered by their grayscale, and a minimum distance $P_{\mathrm{dist}}=16$ is set to remove redundant peak points. These ordered peak points are taken in turn as a center of circle with radius $P_{\mathrm{dist}}$, and the remaining peak points in this circle are removed. So far, we obtain a few distant peak points, which are called local extremum points. Figure 2a,c show two typical pavement image windows of different brightness and cracks and the local extremum points obtained by our method.

Next, feature points will be picked up from the local extremum points, based on the average and variance of grayscale in the neighborhood of these points. If the neighborhood of the i-th extremum points has the distribution of grayscale for N pixel points $G_i(x,y)$, the average and variance of the neighborhood are

(2)$E_i=\frac{1}{N}\sum _{(x,y)\in \left\{N\right\}}{G}_i(x,y)$

and

(3)${\sigma }_i=\sqrt{\frac{1}{N}\sum _{(x,y)\in \left\{N\right\}}{(G_i(x,y)-E_i)}^2}.$

To filter noise from normal pavement, we only retain the points whose grayscale average $E_i/E<95\%$ and ${\sigma }_i>0.05$ as feature points (E is the average grayscale of the image window). Figure 2b,d show the feature points picked up from extremum points of (a)/(c). After removing these noise points, we order the feature points by their grayscale. Based on these feature points, image windows can be cut into several blocks. These feature blocks can then be used for image recognition or feature matching.

2.2. Extraction of Block Feature by CNN

CNN has been widely applied in pavement detection for the efficient image recognition algorithm with feature extraction and classification [26]. In practice, filters in CNN perform convolutional and pooling calculations on the input image data, layer by layer, to extract data features [27]. In this work, the input data of our CNN model are image blocks around the feature points. These image blocks, namely the neighborhood of the feature points, are very small (32 × 32 pixels), which significantly reduces the complexity of the CNN model. According to this fact, we construct a CNN model for pavement damage detection, whose detailed information is displayed in Table 1.

2.3. SVM Classification

Considering that these image blocks are usually relevant, even belonging to the identical crack or cave as shown in Figure 3, it would be more efficient to combine the features of these image blocks. All the feature vectors of each feature point in the image window are combined to form a composite feature vector, which is able to describe the whole image window. Analyses based on these associated blocks, which is a kind of multi-feature fusion, would increase the accuracy of pavement detection [28,29].

These image blocks, proceeded by convolution and subsampling, are transformed to feature vectors by fully connected layer. The feature vectors are further combined to form a composite feature vector of the image window. As a descriptor of the image window, the composite feature vector can be classified by a machine learning algorithm, such as SVM, Adaboost, neural network and random forest. Among them, SVM is often used to replace the softmax classifier of CNN in many recognition and classification cases. SVM has advantages in nonlinear high-dimensional cases, where SVM could transfer low-dimensional linearly inseparable problems to high-dimensional separable ones, with kernel function used to avoid the dimension catastrophe. The composite feature vector, which is of high dimension, can be predicted whether there is pavement damage by SVM-constructed classifier. Therefore, the detection model in our work is of high accuracy and generalization, which could be very efficient even on a small dataset.

3. Results and Discussion

The GAPs v2 dataset is the most extensive dataset in the pavement distress domain that provides standardized, high-quality images [30]. The data of the pavement damage, collected by a Pulnix TM2030 camera on a S.T.I.E.R. car, contain cracks, pot slots, open joints, applied patches and inlaid patches. The GAPs v2 dataset includes a total of 2468 gray-valued images (8 bit), partitioned into 1417 training images, 51 validation images, 500 validation-test images and 500 test images. The image resolution is 1920 × 1080 pixels with a per pixel resolution of 1.2 × 1.2 mm. To facilitate deep learning research, they also created a smaller subset named 50 k for fast experiments, whose size is 160 × 160 pixels. This 50 k dataset also contains four pavement datasets, which are the training set, validation set, validation-test set and test set. In this work, the image blocks extracted from the dataset are used to train the CNN model rather than the dataset itself. To avoid confusion, we renamed these four datasets to be test0, test1, test2 and test3, and we test our model on each dataset independently. Each dataset is separated into a train set and a test set at a ratio of $1:1$.

As stated above, our model is based on the image blocks rather than the whole image window. We construct four training datasets of image blocks, extracting image blocks of 32 × 32 pixels individually from the four training sets, with 160 × 160 pixels in 50 k via the proposed feature-points method. These image blocks, marked with a serial number, are used to train the CNN model to extract the features. The image blocks of the damage field are labeled by a positive sample, and those of normal pavement are by a negative sample. After constructing the training datasets, we then train the CNN model, combine the features and test the model based on the 50 k dataset. The detailed algorithm flow chart of our experiment is shown in Figure 4.

As shown, the feature points based on the local minimum points of grayscale are picked up in the image window. Then, the image blocks of these feature points are cut into a square of 32 × 32 pixels and input into the trained CNN model. For each image block, the fully connected layer of our CNN model can output a 64-dimensional feature vector. These feature vectors are ordered by the grayscale of the corresponding feature points. The top m feature vectors are combined in series to form a composite $m\times 64$-dimensional feature vector. The composite feature vectors from the training set are input into the classifier constructed by an SVM [31], whose kernel is the Radial Basis Function (RBF), with the parameters $C=0.9$, $Tol=1\times {10}^{-3}$ and Gamma of default value in the scikit-learn library. Finally, the images in the test set are input into the proposed detection system to test the accuracy.

All the image processing, coding, compiling and training are conducted on a personal computer with Anaconda3 and OpenCV in a Windows10 operating system. Figure 5 shows the accuracy of our model for the four datasets, test0, test1, test2 and test3, as a function of the number of feature points m. As expected, the accuracy increases as the feature points increase for all four datasets and tends to saturation when $m>5$. The accuracy for all four datasets can reach beyond 0.883 for test3 and even 0.973 for test1 by selecting only 5∼7 feature points. Although the model constructing, training dataset and test dataset of our model are very different from the existing CNN model, we could still perform a rough comparison. In a previous report, the authors employed the train dataset of 50 k to train the ResNet34 model and test the model on the other three subsets [30]. The F1 index in their results shows 0.9284, 0.8709 and 0.8982 for the valid (test1 in this paper), test (test 2 in this paper) and valid-test (test3 in this paper) subsets, respectively. Our testing results are comparable to theirs, which indicates the validity and efficiency of the proposed model in various types of pavement damage. Nevertheless, there are still some limitations and challenges for the proposed methodology. First, the results of our model rely on the quality of the training set. The training set of image blocks is constructed artificially by choosing the image blocks before training the CNN model. The construction of the training set should be optimized in the future. Second, theoretical efforts should be made to confirm the empirical parameters such as the detection threshold value Pth and the number of feature points.

4. Conclusions

To summarize, we propose a road pavement damage detection method based on the local minimum of grayscale and the feature fusion of image blocks. By the image preprocessing of the grayscale, we confirm the feature points according to the local minimum of grayscale. Then, the image blocks consisting of the feature points neighborhood are input into the designed CNN model with an SVM classifier. The tested accuracy for four independent datasets increases as the number of feature points increases and can reach at least 88.3% when $m\ge 5$, which indicates that the proposed model is valid and efficient in identical pavement damage detection. The proposed training model is constructed based on image blocks, is of a small scale and has a small requirement for computing resources. Our model is privileged for cracks and potholes detection, and pavement damage always consists of cracks and potholes. Therefore, our model could be generalized to most damage detection. In practical pavement detection, one can build a dataset of the pavement to be detected with only a few pictures to train the model and obtain recognition results of a relatively high accuracy with little machine time, even at mobile terminals.

Author Contributions

Data curation, G.-H.C. and H.H.; formal analysis, Z.C., Y.-L.S., W.-H.I. and K.-L.Y.; funding acquisition, J.N.; investigation, G.-H.C., W.-W.J. and H.H.; software, Z.C., Y.-L.S. and W.-W.J.; validation, Z.C. and J.N.; writing—original draft, Z.C.; writing—review and editing, Y.-L.S.; discussion and future work, Y.-L.S., W.-H.I. and K.-L.Y. All authors have read and agreed to the published version of the manuscript.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

The data presented in this study are available in the article.

Acknowledgments

This work was supported in part by a grant from the Key Laboratory of Wireless Sensor Network & Communication, Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences (20190909). The work was supported in part by the Research Center for Deep Space Explorations of the Hong Kong Polytechnic University.

Conflicts of Interest

The funders had no role in the design of the study; in the collection, analyses or interpretation of data; in the writing of the manuscript; or in the decision to publish the results.

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Figures and Table

Figure 1. Flow chart of the methodology used in this work.

Figure 2. (a,c) Two typical pavement image windows of different brightness and cracks, and the local extremum points. (b,d) The feature points picked up from the local extremum points.

Figure 3. (a) The detected image blocks in the image window. (b) Most image blocks belong to the identical crack.

Figure 4. The detailed algorithm flow chart of our experiment.

Figure 5. Accuracy as a function of the number of feature points m for datasets test0, test1, test2 and test3.

Table 1

The detailed information of CNN model in this work.

Type	Filter Size	Stride	Output Shape	Parameters
in	N/A	N/A	(32,32,1)	N/A
conv	3 × 3(64)	(1,1)	(30,30,64)	640
conv	3 × 3(64)	(1,1)	(28,28,64)	36,928
pool	2 × 2	(2,2)	(14,14,64)
conv	3 × 3(128)	(1,1)	(12,12,128)	73,856
pool	2 × 2	(2,2)	(6,6,128)
conv	3 × 3(128)	(1,1)	(4,4,128)	147,584
flat	N/A	N/A	2048	N/A
fc	(128)	N/A	128	262,272
fc	(64)	N/A	64	8256
out	(2)	N/A	2	130
				Total parameters: 529,666.

References

1. Fang, W.; Zhu, C.; Yu, F.R.; Wang, K.; Zhang, W. Towards Energy-Efficient and Secure Data Transmission in AI-Enabled Software Defined Industrial Networks. IEEE Trans. Ind. Inform.; 2022; 18, pp. 4265-4274. [DOI: https://dx.doi.org/10.1109/TII.2021.3122370]

2. Fang, W.; Cui, N.; Chen, W.; Zhang, W.; Chen, Y. A Trust-based Security System for Data Collection in Smart City. IEEE Trans. Ind. Inform.; 2021; 17, pp. 4131-4140. [DOI: https://dx.doi.org/10.1109/TII.2020.3006137]

3. Fang, W.; Zhang, W.; Yang, W.; Li, Z.; Gao, W.; Yang, Y. Trust management-based and energy efficient hierarchical routing protocol in wireless sensor networks. Digit. Commun. Netw.; 2021; 7, pp. 470-478. [DOI: https://dx.doi.org/10.1016/j.dcan.2021.03.005]

4. Xu, Z.; Che, Y.; Li, J.; Zhao, X.; Pan, Y.; Wang, Z.; Wei, N.; Song, H. Research progress on automatic processing technology for pavement distress. J. Traffic Transp. Eng.; 2019; 19, pp. 172-190.

5. Habib, A.; Kunt, M.M. A framework for pavement crack detection and classification. IOP Conf. Ser. Mater. Sci. Eng.; 2020; 800, 012020. [DOI: https://dx.doi.org/10.1088/1757-899X/800/1/012020]

6. Chen, H. Deep Learning for Pavement Distress Detection. Ph.D. Thesis; Zhejiang University of Technology: Hangzhou, China, 2020.

7. Kheradmandi, N.; Mehranfar, V. A critical review and comparative study on image segmentation-based techniques for pavement crack detection. Constr. Build Mater.; 2022; 321, 126162. [DOI: https://dx.doi.org/10.1016/j.conbuildmat.2021.126162]

8. Zhang, H.; Wang, Q.; Wei, W. Pavement Distress Detection Based on Artificial Population. J. Nanjing Univ. Sci. Technol.; 2005; 29, 4.

9. Nayyeri, F.; Hou, L.; Zhou, J.; Guan, H. Foreground-background separation technique for crack detection. Comput.-Aided. Civ. Inf.; 2019; 34, pp. 457-470. [DOI: https://dx.doi.org/10.1111/mice.12428]

10. Yousaf, M.H.; Azhar, K.; Murtaza, F.; Hussain, F. Visual analysis of asphalt pavement for detection and localization of potholes. Adv. Eng. Inform.; 2018; 38, pp. 527-537. [DOI: https://dx.doi.org/10.1016/j.aei.2018.09.002]

11. Jin, H.; Wan, F.; Ye, Z. Pavement crack detection fused HOG and watershed algorithm of range image. J. Cent. China Norm. Univ.; 2017; 51, pp. 7415-7722.

12. Patel, D.R.; Thakker, H.; Kiran, M.B.; Vakharia, V. Surface roughness prediction of machined components using gray level co-occurrence matrix and Bagging Tree. FME Trans.; 2020; 48, pp. 468-475. [DOI: https://dx.doi.org/10.5937/fme2002468P]

13. Lu, K. Advances in deep learning methods for pavement surface crack detection and identification with visible light visual images. arXiv; 2020; arXiv: 2012.14704

14. Eisenbach, M.; Stricker, R.; Seichter, D.; Amende, K.; Debes, K.; Sesselmann, M.; Ebersbach, D.; Stöckert, U.; Gross, H.-M. How to Get Pavement Distress Detection Ready for Deep Learning? A Systematic Approach. Proceedings of the 2017 International Joint Conference on Neural Networks; Anchorage, AK, USA, 3 July 2017; pp. 2039-2047.

15. Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv; 2014; arXiv: 1409.1556

16. Hou, Y.; Dong, Y.; Zhang, Y.; Zhou, Z.; Tong, X.; Wu, Q.; Qian, Z.; Li, R. The Application of a Pavement Distress Detection Method Based on FS-Net. Sustainability; 2022; 14, 2715. [DOI: https://dx.doi.org/10.3390/su14052715]

17. Ukhwah, E.N.; Yuniarno, E.M.; Suprapto, Y.K. Asphalt pavement pothole detection using deep learning method based on YOLO neural network. Proceedings of the 2019 International Seminar on Intelligent Technology and Its Applications; Surabaya, Indonesia, 6 January 2020; pp. 35-40.

18. Xiang, X.; Zhang, Y.; Saddik, A.E. Pavement Crack Detection Network Based on Pyramid Structure and Attention Mechanism. IET Image Process.; 2020; 14, pp. 1580-1586. [DOI: https://dx.doi.org/10.1049/iet-ipr.2019.0973]

19. Chen, G.-H.; Ni, J.; Chen, Z.; Huang, H.; Sun, Y.-L.; Ip, W.H.; Yung, K.L. Detection of Highway Pavement Damage Based on a CNN Using Grayscale and HOG Features. Sensors; 2022; 22, 2455. [DOI: https://dx.doi.org/10.3390/s22072455]

20. Salama, M.; Kader, H.A.; Abdelwahab, A. An analytic framework for enhancing the performance of big heterogeneous data analysis. Int. J. Eng. Bus. Manag.; 2021; 13, 1847979021990523. [DOI: https://dx.doi.org/10.1177/1847979021990523]

21. Ibragimov, E.; Lee, H.J.; Lee, J.J.; Kim, N. Automated pavement distress detection using region based convolutional neural networks. Int. J. Pavement Eng.; 2022; 23, pp. 1981-1992. [DOI: https://dx.doi.org/10.1080/10298436.2020.1833204]

22. Li, Z.; Xu, G.; Cheng, Y.; Wang, Z.; Wu, Q. Pavement Crack Detection Using Progressive Curvilinear Structure Anisotropy Filtering and Adaptive Graph-Cuts. IEEE Access; 2020; 8, pp. 65020-65034. [DOI: https://dx.doi.org/10.1109/ACCESS.2020.2985216]

23. Laganiére, R. OpenCV2 Computer Vision Application Programming Cookbook; 2nd ed. Packt Publishing Ltd.: Birmingham, UK, 2014; pp. 167-188.

24. Chong, Y.; Ai, W.; Xu, G. Image Enhancement of Empirical Mode Decomposition Based on Local Gradient Extreme Point. Ordnance Ind. Autom.; 2020; 39, pp. 28-31.

25. Vincent, L. Morphological Grayscale Reconstruction: Definition, Efficient Algorithm and Applications in Image Analysis. Proceedings of the 1992 IEEE Computer Society Conference on Computer Vision and Pattern Recognition; Champaign, IL, USA, 6 August 2002; pp. 633-635.

26. Yang, X.; Li, H.; Yu, Y.; Luo, X.; Huang, T.; Yang, X. Automatic Pixel-Level crack detection and measurement using fully convolutional network. Comput.-Aided. Civ. Inf.; 2018; 33, pp. 1090-1109. [DOI: https://dx.doi.org/10.1111/mice.12412]

27. Yang, F.; Zhang, L.; Yu, S.; Prokhorov, D.; Mei, X.; Ling, H. Feature pyramid and hierarchical boosting network for pavement crack detection. IEEE Trans. Intell. Transp. Syst.; 2020; 21, pp. 1525-1535. [DOI: https://dx.doi.org/10.1109/TITS.2019.2910595]

28. Ding, M.; Li, L. CNN and HOG Dual-Path Feature Fusion for Face Expression Recognition. Inf. Control; 2020; 49, pp. 47-54.

29. Qu, Z.; Cao, C.; Liu, L.; Zhou, D. A Deeply Supervised Convolutional Neural Network for Pavement Crack Detection With Multiscale Feature Fusion. IEEE Trans. Neural Netw. Learn. Syst.; 2022; 33, pp. 4890-4899. [DOI: https://dx.doi.org/10.1109/TNNLS.2021.3062070] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/33720835]

30. Stricker, R.; Eisenbach, M.; Sesselmann, M.; Debes, K.; Gross, H.-M. Improving Visual Road Condition Assessment by Extensive Experiments on the Extended GAPs Dataset. Proceedings of the 2019 International Joint Conference on Neural Networks; Budapest, Hungary, 30 September 2019; pp. 1-8.

31. Wang, J.J.; Wang, Y.K.; Zhang, F.; Zhang, S.W.; Dai, Y.; Yu, X.D. Real-Time Detection for Eye Closure Feature of Fatigue Driving Based on CNN and SVM. Comput. Syst. Appl.; 2021; 30, pp. 118S-126S.