Full text

Turn on search term navigation

1. Introduction

1.1. Background

Underground sewerage systems (USSs) are a vital part of public infrastructure that contributes to collecting wastewater or stormwater from various sources and conveying it to storage tanks or sewer treatment facilities. A healthy USS with proper functionality can effectively prevent urban waterlogging and play a positive role in the sustainable development of water resources. However, sewer defects caused by different influence factors such as age and material directly affect the degradation of pipeline conditions. It was reported in previous studies that the conditions of USSs in some places are unsatisfactory and deteriorate over time. For example, a considerable proportion (20.8%) of Canadian sewers is graded as poor and very poor. The rehabilitation of these USSs is needed in the following decade in order to ensure normal operations and services on a continuing basis [1]. Currently, the maintenance and management of USSs have become challenging problems for municipalities worldwide due to the huge economic costs [2]. In 2019, a report in the United States of America (USA) estimated that utilities spent more than USD 3 billion on wastewater pipe replacements and repairs, which addressed 4692 miles of pipeline [3].

1.2. Defect Inspection Framework

Since it was first introduced in the 1960s [4], computer vision (CV) has become a mature technology that is used to realize promising automation for sewer inspections. In order to meet the increasing demands on USSs, a CV-based defect inspection system is required to identify, locate, or segment the varied defects prior to the rehabilitation process. As illustrated in Figure 1, an efficient defect inspection framework for underground sewer pipelines should cover five stages. In the data acquisition stage, there are many available techniques such as closed-circuit television (CCTV), sewer scanner and evaluation technology (SSET), and totally integrated sonar and camera systems (TISCITs) [5]. CCTV-based inspections rely on a remotely controlled tractor or robot with a mounted CCTV camera [6]. An SSET is a type of method that acquires the scanned data from a suite of sensor devices [7]. The TISCIT system utilizes sonar and CCTV cameras to obtain a 360° view of the sewer conditions [5]. As mentioned in several studies [6,8,9,10], CCTV-based inspections are the most widely used methods due to the advantages of economics, safety, and simplicity. Nevertheless, the performance of CCTV-based inspections is limited by the quality of the acquired data. Therefore, image-based learning methods require pre-processing algorithms to remove noise and enhance the resolution of the collected images. Many studies on sewer inspections have recently applied image pre-processing before examining the defects [11,12,13].

In the past few decades, many defect inspection strategies and algorithms have been presented based on CCTV cameras. Manual inspections by humans are inefficient and error-prone, so several studies attempted to adopt conventional machine learning (ML) approaches in order to diagnose the defects based on morphological, geometrical, or textural features [14,15,16]. With the elevation and progress of ML, deep learning (DL) methods have been widely applied to enhance the overall performance in recent studies on sewer inspections. Previous investigations have reviewed and summarized different kinds of inspections, which mainly include manual inspections [17,18] and automatic inspections based on the conventional machine learning algorithms [15,19] and deep learning algorithms [9,20].

In the attempt to evaluate the infrastructure conditions, some researchers have developed risk assessment approaches using different image post-processing algorithms [21,22,23]. For instance, a defect segmentation method was proposed to separate the cracks from the background, and post-processing was subsequently used to calculate the morphological features of the cracks [22]. In another study, a method based on a fully convolutional network and post-processing was introduced to detect and measure cracks [21]. Nevertheless, the existing risk assessment methods are limited to the feature analysis of cracks only, and there is no further research and exploration of each specific category.

1.3. Previous Survey Papers

Table 1 lists the major contributions of five survey papers, which considered different aspects of defect inspection and condition assessment in underground sewer pipelines. In 2019, an in-depth survey was presented to analyze different inspection algorithms [24]. However, it only focused on defect detection, and defect segmentation was not involved in this study. Several surveys [7,10,20] were conducted one year later to discuss the previous studies on sewer defects. Moreover, the recent studies associated with image-based construction applications are discussed in [8]. In these relevant surveys, the authors of each paper put efforts into emphasizing a particular area. A more comprehensive review of the latest research on defect inspection and condition assessment is significant for the researchers who are interested in integrating the algorithms into real-life sewer applications. In addition, the detailed and well-arranged list tables for the existing defect inspection methods according to the different categories are not provided in these papers.

1.4. Contributions

In order to address the above issues, a survey that covers various methods regarding sewer defect inspection and condition assessment is conducted in this study. The main contributions are as follows. This survey creates a comprehensive review of the vision-based algorithms about defect inspection and condition assessment from 2000 to the present. Moreover, we divide the existing algorithms into three categories, which include defect classification, detection, and segmentation. In addition, different datasets and evaluation metrics are summarized. Based on the investigated papers, the research focuses and tendencies in previous studies are analyzed. Meanwhile, the limitations of the existing approaches as well as the future research directions are indicated.

The rest of this survey is divided into four sections. Section 2 presents the methodology used in this survey. Section 3 discusses the image-based defect inspection algorithms that cover classification, detection, and segmentation. Section 4 analyzes the dataset and the evaluation metrics that are used from 2000 onwards. In Section 5, the challenges and future needs are indicated. Conclusions of previous studies and suggestions for future research are provided in Section 6.

2. Survey Methodology

A thorough search of the academic studies was conducted by using the Scopus journal database. It automatically arranges the results from different publishers, which include Elsevier, Springer Link, Wiley online library, IEEE Xplore, ASCE Library, MDPI, SACG, preprint, Taylor & Francis Group, and others. Figure 2 shows the distribution of the academic journals reviewed in diverse databases. The journals in the other databases include SPIE Digital Library, Korean Science, Easy Chair, and Nature. In order to highlight the advances in vision-based defect inspection and condition assessment, the papers of these fields that were published between 2000 and 2022 are investigated. The search criterion of this survey is to use an advanced retrieval approach by selecting high-level keywords like (“vision-based sensor” OR “video” OR “image”) AND (“automatic sewer inspection” OR “defect classification” OR “defect detection” OR “defect segmentation” OR “condition assessment”). Since there is no limitation on a certain specific construction material or pipe typology, the research on any sewer pipeline that can be entered and that obtained visual data is covered in this survey. Nevertheless, the papers that focus on some topics, which do not relate to the vision-based sewer inspection, are not included in this paper. For example, the quality assessment for sewer images [25], pipe reconstruction, internal pipe structure, wall thickness measurement, and sewer inspections based on other sensors such as depth sensors [26,27], laser scanners [28,29], or acoustic sensors [30,31] are considered irrelevant topics. Figure 3 represents the number of articles including journals and conference papers in different time periods from 2000 to 2022. By manually scanning the title and abstract sections, a total of 124 papers that includes both journals (95) and conferences (29) in English was selected to examine the topic’s relevancy. In addition to these papers, four books and three websites were also used to construct this survey. After that, the filtered papers were classified in terms of the employed methods and application areas. Finally, the papers in each category were further studied by analyzing their weaknesses and strengths.

3. Defect Inspection

In this section, several classic algorithms are illustrated, and the research tendency is analyzed. Figure 4 provides a brief description of the algorithms in each category. According to the literature review, the existing studies about sewer inspection are summarized in three tables. Table 2, Table 3 and Table 4 show the recent studies about defect classification (Section 3.1), detection (Section 3.2), and segmentation (Section 3.3) algorithms. In order to comprehensively analyze these studies, the publication time, title, utilized methodology, advantages, and disadvantages for each study are covered. Moreover, the specific proportion of each inspection algorithm is computed in Figure 5. It is clear that the defect classification accounts for the most significant percentages in all the investigated studies.

3.1. Defect Classification

Due to the recent advancements in ML, both the scientific community and industry have attempted to apply ML-based pattern recognition in various areas, such as agriculture [32], resource management [33], and construction [34]. At present, many types of defect classification algorithms have been presented for both binary and multi-class classification tasks. The commonly used algorithms are described below.

3.1.1. Support Vector Machines (SVMs)

SVMs have become one of the most typical and robust ML algorithms because they are not sensitive to the overfitting problem compared with other ML algorithms [35,36,37]. The principal objective of an SVM is to perfectly divide the training data into two or more classes by optimizing the classification hyperplane [38,39]. A classification hyperplane equation can be normalized in order to form a two-dimensional sample set that satisfies Equation (1).

(1)yi(wTx+b)≥1, i=1, …,n.

where xi∈ℝ2 and yi∈(+1,−1); w is the optimal separator and b is the bias. As shown in Figure 6, the circles and triangles indicate two classes of training samples. The optimal hyperplane is represented as H, and the other two parallel hyperplanes are represented as H1 and H2. On the premise of correctly separating samples, the maximum margin between the two hyperplanes (H1 and H2) is conducive to gaining the optimal hyperplane (H).

Despite classifying various types of defects with high accuracy, the SVM algorithm cannot be applied to end-to-end classification problems [40]. As demonstrated in [41], Ye et al. established a sewer image diagnosis system where a variety of image pre-processing algorithms, such as Hu invariant moments [42] and lateral Fourier transform [43] were used for the feature extraction, and the SVM was then used as the classifier. The accuracy of the SVM classifier reached 84.1% for seven predefined classes, and the results suggested that the training sample number is positively correlated with the final accuracy. In addition to this study, Zuo et al. applied the SVM algorithm that is based on a specific histogram to categorize three different cracks at the sub-class level [11]. Before the classification process, bilateral filtering [44,45] was applied in image pre-processing in order to denoise input images and keep the edge information. Their proposed method obtained a satisfactory average accuracy of 89.6%, whereas it requires a series of algorithms to acquire 2D radius angular features before classifying the defects.

3.1.2. Convolutional Neural Networks (CNNs)

A CNN was first proposed in 1962 [46], and it has demonstrated excellent performances in multiple domains. Due to its powerful generalization ability, CNN-based classifiers that automatically extract features from input images are superior to the classifiers that are based on the pre-engineered features [47]. Consequently, numerous researchers have applied CNNs to handle the defect classification problem in recent years. Kumar et al. presented an end-to-end classification method using several binary CNNs in order to identify the presence of three types of commonly encountered defects in sewer images [48]. In their proposed framework, the extracted frames were inputted into networks that contained two convolutional layers, two pooling layers, two fully connected layers, and one output layer. The classification results achieved high values in terms of average accuracy (0.862), precision (0.877), and recall (0.906), but this work was limited to the classification of ubiquitous defects.

Meijer et al. reimplemented the network proposed in [48], and they compared the performances based on a more realistic dataset introduced in [49]. They used a single CNN to deal with multi-label classification problems, and their classifier outperformed the method presented by Kumar et al. In another work, several image pre-processing approaches, which included histogram equalization [50] and morphology operations [51], were used for noise removal. After that, a fine-tuned defect classification model was used to extract informative features based on highly imbalanced data [52]. Their presented model architecture was based on the VGG network, which achieved first place in the ILSVRC-2014 [53]. As illustrated in Figure 7, the model structure in the first 17 layers is frozen, and the other sections are trainable; also, two convolutional layers and one batch normalization were added to enhance the robustness of the modified network.

Table 2

Academic studies in vision-based defect classification algorithms.

Time Methodology Advantage Disadvantage Ref.
2000 Back-propagation algorithm Perform well for classification Slow learning speed [54]
2002 Neuro-fuzzy algorithm Good classification efficiency Weak feature extraction scheme [55]
2006 Neuro-fuzzy classifier ● Combines neural network and fuzzy logic concepts● Screens data before network training to improve efficiency Not an end-to-end model [56]
2009 Rule-based classifier Recognize defects under the realistic sewer condition No real-time recognition [57]
2009 Rule-based classifier Addresses realistic defect detection and recognition Unsatisfactory classification result [58]
2009 Radial basis network (RBN) Overall classification accuracy is high Heavily relies on the pre-engineered results [59]
2012 Self-organizing map (SOM) Suitable for large-scale real applications High computation complexities [60]
2013 Ensemble classifiers ● High practicability● Reliable classification result Feature extraction and classification are separately implemented [61]
2016 Random forests Dramatically reduces the processing time Processing speed can be improved [62]
2017 Random forest classifier Automated fault classification Poor performance [63]
2017 Hidden Markov model (HMM) ● Efficient for numerous patterns of defects● Real time Low classification accuracy [64]
2018 One-class SVM (OCSVM) Available for both still images and video sequences Cannot achieve a standard performance [65]
2018 Multi-class random forest Poor classification accuracy Real-time prediction [66]
2018 Multiple binary CNNs ● Good generalization capability● Can be easily re-trained ● Do not support sub-defects classification● Cannot localize defects in pipeline [48]
2018 CNN ● High detection accuracy● Strong scene adaptability in realistic scenes Poor performance for the unnoticeable defects [67]
2018 HMM and CNN Automatic defect detection and classification in videos Poor performance [68]
2019 Single CNN ● Outperforms the SOTA● Allows multi-label classification Weak performance for fully automatic classification [49]
2019 Two-level hierarchical CNNs Can identify the sewer images into different classes Cannot classify multiple defects in the same image simultaneously [69]
2019 Deep CNN ● Classifies defects at different levels● Performs well in classifying most classes There exists a extremely imbalanced data problem (IDP) [70]
2019 CNN Accurate recognition and localization for each defect Classifies only one defect with the highest probability in an image [71]
2019 SVM Reveals the relationship between training data and accuracy Requires various steps for feature extraction [41]
2020 SVM ● Classifies cracks at a sub-category level● High recall and fast processing speed Limited to only three crack patterns [11]
2020 CNN ● Image pre-processing used for noisy removal and image enhancement● High classification accuracy Can classify one defect only [72]
2020 CNN Shows great ability for defect classification under various conditions Limited to recognize the tiny and narrow cracks [73]
2021 CNN Is robust against the IDP and noisy factors in sewer images No multi-label classification [52]
2021 CNN Covers defect classification, detection, and segmentation Weak classification results [74]

3.2. Defect Detection

Rather than the classification algorithms that merely offer each defect a class type, object detection is conducted to locate and classify the objects among the predefined classes using rectangular bounding boxes (BBs) as well as confidence scores (CSs). In recent studies, object detection technology has been increasingly applied in several fields, such as intelligent transportation [75,76,77], smart agriculture [78,79,80], and autonomous construction [81,82,83]. The generic object detection consists of the one-stage approaches and the two-stage approaches. The classic one-stage detectors based on regression include YOLO [84], SSD [85], CornerNet [86], and RetinaNet [87]. The two-stage detectors are based on region proposals, including Fast R-CNN [88], Faster R-CNN [89], and R-FCN [90]. In this survey, the one-stage and two-stage methods that were employed in sewer inspection studies are both discussed and analyzed as follows.

3.2.1. You Only Look Once (YOLO)

YOLO is a one-stage algorithm that maps directly from image pixels to BBs and class probabilities. In [84], object detection was addressed as a single regression problem using a simple and unified pipeline. Due to its advantages of robustness and efficiency, an updated version of YOLO, which is called YOLOv3 [91], was explored to locate and classify defects in [9]. YOLOv3 outperformed the previous YOLO algorithms in regard to detecting the objects with small sizes because the YOLOv3 model applies a 3-scale mechanism that concatenates the feature maps of three scales [92,93]. Figure 8 illustrates how the YOLOv3 architecture implements the 3-scale prediction operation. The prediction result with a scale of 13 × 13 is obtained in the 82nd layer by down-sampling and convolution operations. Then, the result in the 79th layer is concatenated with the result of the 61st layer after up-sampling, and the prediction result with 26 × 26 is generated after several convolution operations. The result of 52 × 52 is generated at layer 106 using the same method. The predictions at different scales have different receptive fields that determine the appropriate sizes of the detection objects in the image. As a result, YOLOv3 with a 3-scale mechanism is capable of detecting more fine-grained features.

Based on the detection model developed by [9], a video interpretation algorithm was proposed to build an autonomous assessment framework in sewer pipelines [94]. The assessment system verified how the defect detector can be put to use with realistic infrastructure maintenance and management. A total of 3664 images extracted from 63 videos were trained by the YOLOv3 model, which achieved a high mean average precision (mAP) of 85.37% for seven defects and also obtained a fast detection speed for real-time applications.

3.2.2. Single Shot Multibox Detector (SSD)

Similarly, another end-to-end detector that is named SSD was first introduced for multiple object classes in [85]. Several experiments were conducted to analyze the detection speed and accuracy based on different public datasets. The results suggest that the SSD model (input size: 300 × 300) obtained faster speed and higher accuracy than the YOLO model (input size: 448 × 448) on the VOC2007 test. As shown in Figure 9, the SSD method first extracts features in the base network (VGG16 [53]). It then predicts the fixed-size bounding boxes and class scores for each object instance using a feed-forward CNN [95]. After that, a non-maximum suppression (NMS) algorithm [96] is used to refine the detections by removing the redundant boxes.

Moreover, the SSD method was utilized to detect defects for CCTV images in a condition assessment framework [97]. Several image pre-processing algorithms were used to enhance the input images prior to the feature extraction process. Then three state-of-the-art (SOTA) detectors (YOLOv3 [91], SSD [85], and faster-RCNN [89]) based on DLs were tested and compared on the same dataset. The defect severity was rated in the end from different aspects in order to assess the pipe condition. Among three experimental models, YOLOv3 demonstrated that it obtained a relatively balanced performance between speed and accuracy. The SSD model achieved the fastest speed (33 ms per image), indicating the feasibility of real-time defect detection. However, the detection accuracy of SSD was the lowest, which was 28.6% lower than the accuracy of faster R-CNN.

3.2.3. Faster Region-Based CNN (Faster R-CNN)

The faster R-CNN model was introduced to first produce candidate BBs and then refine the generated BB proposals [89]. Figure 10 shows the architecture of faster R-CNN developed by [98] in a defect detection system. First of all, the multiple CNN layers in the base network were used for feature extraction. Then, the region proposal network (RPN) created numerous proposals based on the generated feature maps. Finally, these proposals were sent to the detector for further classification and localization. Compared with the one-stage frameworks, the region proposal-based methods require more time in handling different model components. However, the faster R-CNN model that trains RPN and fast R-CNN detector separately is more accurate than other end-to-end training models, such as YOLO and SSD [99]. As a result, the faster R-CNN was explored in many studies for more precise detection of sewer defects.

In [98], 3000 CCTV images were fed into the faster R-CNN model, and the trained model was then utilized to detect four categories of defects. This research indicated that the data size, training scheme, network structure, and hyper-parameter are important impact factors for the final detection accuracy. The results show the modified model achieved a high mAP of 83%, which was 3.2% higher than the original model. In another work [99], a defect tracking framework was firstly built by using a faster R-CNN detector and learning discriminative features. In the defect detection process, a mAP of 77% was obtained for detecting three defects. At the same time, the metric learning model was trained to reidentify defects. Finally, the defects in CCTV videos were tracked based on detection information and learned features.

Table 3

Academic studies in vision-based defect detection algorithms.

Time Methodology Advantage Disadvantage Ref.
2004 Genetic algorithm (GA) and CNN High average detection rate Can only detect one type of defect [100]
2014 Histograms of oriented gradients (HOG) and SVM Viable and robust algorithm Complicated image processing steps before detecting defective regions [101]
2018 Faster R-CNN ● Explores the influences of several factors for the model performance● Provides references to applied DL in autonomous construction Limited to the still images [98]
2018 Faster R-CNN Addresses similar object detection problems in industry Long training time and slow detection speed [102]
2018 Rule-based detection algorithm ● Based on image processing techniques● No need training process● Requires less images Low detection performance [103]
2019 YOLO End-to-end detection workflow Cannot detect defect at the sub-classes [104]
2019 YOLOv3 ● High detection rate● Real-time defect detection● Efficient input data manipulation process Weak function of output frames [9]
2019 SSD, YOLOv3, and Faster R-CNN Automatic detection for the operational defects Cannot detect the structural defects [105]
2019 Rule-based detection algorithm Performs well on the low-resolution images Requires multiple digital image processing steps [106]
2019 Kernel-based detector Promising and reliable results for anomaly detection Cannot get the true position inside pipelines [107]
2019 CNN and YOLO Obtained a considerable reduction in processing speed Can detect only one type of structural defect [108]
2020 Faster R-CNN Can assess the defect severity as well as the pipe condition Cannot run in real time [97]
2020 Faster R-CNN ● Can obtain the number of defects● First work for sewer defect tracking Requires training two models separately, not an end-to-end framework [99]
2020 SSD, YOLOv3, and Faster R-CNN Automated defect detection Structural defect detection and severity classification are not available [105]
2021 YOLOv3 ● Covers defect detection, video interpretation, and text recognition● Detect defect in real time The ground truths (GTs) are not convincing [94]
2021 CNN and non-overlapping windows Outperformed existing models in terms of detection accuracy Deeper CNN model with better performance requires longer inference time [109]
2021 Strengthened region proposal network (SRPN) ● Effectively locate defects● Accurately assess the defect grade ● Cannot be applied for online processing● Cannot identify if the defect is mirrored [110]
2021 YOLOv2 Covers defect classification, detection, and segmentation Weak detection results [74]
2022 Transformer-based defect detection (DefectTR) ● Does not require prior knowledge● Can generalize well with limited parameters The robustness and efficiency can be improved for real-world applications [111]

3.3. Defect Segmentation

Defect segmentation algorithms can predict defect categories and pixel-level location information with exact shapes, which is becoming increasingly significant for the research on sewer condition assessment by re-coding the exact defect attributes and analyzing the specific severity of each defect. The previous segmentation methods were mainly based on mathematical morphology [112,113]. However, the morphology segmentation approaches were inefficient compared to the DL-based segmentation methods. As a result, the defect segmentation methods based on DL have been recently explored in various fields. The studies related to sewer inspection are described as follows.

3.3.1. Morphology Segmentation

Morphology-based defect segmentation contains many methods, such as closing bottom-hat operation (CBHO), opening top-hat operation (OTHO), and morphological segmentation based on edge detection (MSED). By evaluating and comparing the segmentation performances of different methods, MSED was verified as being useful to detect cracks, and OTHO was verified as being useful to detect open joints [113]. They also indicated the removal of the text on the CCTV images is necessary to further improve the detection accuracy. Similarly, MSED was applied to segment eight categories of typical defects, and it outperformed another popular approach called OTHO [112]. In addition, some morphology features, including area, axis length, and eccentricity, were also measured, which is of great significance to assist inspectors in judging and assessing defect severity. Although the morphology segmentation methods showed good segment results, they need multiple image pre-processing steps before the segmentation process.

3.3.2. Semantic Segmentation

Automatic localization of the sewer defect’s shape and the boundary was first proposed by Wang et al. using a semantic segmentation network called DilaSeg [114]. In order to improve the segmentation accuracy, an updated network named DilaSeg-CRF was introduced by combining the CNN with a dense conditional random field (CRF) [115,116]. Their updated network improved the segmentation accuracy considerably in terms of the mean intersection over union (mIoU), but the single data feature and the complicated training process reflect that the DilaSe-CRF is not suitable to be applied in real-life applications.

Recently, the fully convolutional network (FCN) has been explored for the pixels-to-pixels segmentation task [117,118,119,120]. Meanwhile, some other network architectures that are similar to an FCN have emerged in large numbers, including U-Net [121]. Pan et al. proposed a semantic segmentation network called PipeUNet, in which the U-Net was used as the backbone due to its fast convergence speed [122]. As shown in Figure 11, the encoder and decoder on both sides form a symmetrical architecture. In addition, three FRAM blocks were added before the skip connections to improve the ability of feature extraction and reuse. Besides, the focal loss was demonstrated, which is useful for handling the imbalanced data problem (IDP). Their proposed PipeUNet achieved a high mIoU of 76.3% and a fast speed of 32 frames per second (FPS).

Table 4

Academic studies in vision-based defect segmentation algorithms.

Time Methodology Advantage Disadvantage Ref.
2005 Mathematical morphology-based Segmentation ● Automated segmentation based on geometry image modeling● Perform well under various environments ● Can only segment cracks● Complicated and multiple steps [123]
2014 Mathematical morphology-based Segmentation Requires less data and computing resources to achieve a decent performance ● Challenging to detect cracks● Various processing steps [113]
2019 DL-based semantic segmentation (DilaSeg-CRF) ● End-to-end trainable model● Fair inference speed Long training time [116]
2020 DilaSeg-CRF ● Promising segmentation accuracy● The defect severity grade is presented Complicated workflow [23]
2020 DL-based semantic segmentation(PipeUNet) ● Enhances the feature extraction capability● Resolves semantic feature differences● Solves the IDP Still exists negative segmentation results [122]
2021 Feature pyramid networks (FPN) and CNN Covers defect classification, detection, and segmentation Weak segmentation results [74]
2022 DL-based defect segmentation (Pipe-SOLO) ● Can segment defect at the instance level● Is robust against various noises from natural scenes Only suitable for still sewer images [124]

4. Dataset and Evaluation Metric

The performances of all the algorithms were tested and are reported based on a specific dataset using specific metrics. As a result, datasets and protocols were two primary determining factors in the algorithm evaluation process. The evaluation results are not convincing if the dataset is not representative, or the used metric is poor. It is challenging to judge what method is the SOTA because the existing methods in sewer inspections utilize different datasets and protocols. Therefore, benchmark datasets and standard evaluation protocols are necessary to be provided for future studies.

4.1. Dataset

4.1.1. Dataset Collection

Currently, many data collection robotic systems have emerged that are capable of assisting workers with sewer inspection and spot repair. Table 5 lists the latest advanced robots along with their respective information, including the robot’s name, company, pipe diameter, camera feature, country, and main strong points. Figure 12 introduces several representative robots that are widely utilized to acquire images or videos from underground infrastructures. As shown in Figure 12a, LETS 6.0 is a versatile and powerful inspection system that can be quickly set up to operate in 150 mm or larger pipes. A representative work (Robocam 6) of the Korean company TAP Electronics is shown in Figure 12b. Robocam 6 is the best model to increase the inspection performance without the considerable cost of replacing the equipment. Figure 12c is the X5-HS robot that was developed in China, which is a typical robotic crawler with a high-definition camera. In Figure 12d, Robocam 3000, sold by Japan, is the only large-scale system that is specially devised for inspecting pipes ranging from 250 mm to 3000 mm. It used to be unrealistic to apply the crawler in huge pipelines in Korea.

4.1.2. Benchmarked Dataset

Open-source sewer defect data is necessary for academia to promote fair comparisons in automatic multi-defect classification tasks. In this survey, a publicly available benchmark dataset called Sewer-ML [125] for vision-based defect classification is introduced. The Sewer-ML dataset, acquired from Danish companies, contains 1.3 million images labeled by sewer experts with rich experience. Figure 13 shows some sample images from the Sewer-ML dataset, and each image includes one or more classes of defects. The recorded text in the image was redacted using a Gaussian blur kernel to protect private information. Besides, the detailed information of the datasets used in recent papers is described in Table 6. This paper summarizes 32 datasets from different countries in the world, of which the USA has 12 datasets, accounting for the largest proportion. The largest dataset contains 2,202,582 images, whereas the smallest dataset has only 32 images. Since the images were acquired by various types of equipment, the collected images have varied resolutions ranging from 64 × 64 to 4000 × 46,000.

4.2. Evaluation Metric

The studied performances are ambiguous and unreliable if there is no suitable metric. In order to present a comprehensive evaluation, multitudinous methods are proposed in recent studies. Detailed descriptions of different evaluation metrics are explained in Table 7. Table 8 presents the performances of the investigated algorithms on different datasets in terms of different metrics.

As shown in Table 8, accuracy is the most commonly used metric in the classification tasks [41,48,52,54,56,57,58,61,65,66,67,69,70,71,73]. In addition to this, other subsidiary metrics such as precision [11,48,67,69,73], recall [11,48,57,67,69,73], and F1-score [69,73] are also well supported. Furthermore, AUROC and AUPR are calculated in [49] to measure the classification results, and FAR is used in [57,58] to check the false alarm rate in all the predictions. In contrast to classification, mAP is a principal metric for detection tasks [9,98,99,105,110]. In another study [97], precision, recall, and F1-score are reported in conjunction to provide a comprehensive estimation for defect detection. Heo et al. [106] assessed the model performance based on the detection rate and the error rate. Kumar and Abraham [108] report the average precision (AP), which is similar to the mAP but for each class. For the segmentation tasks, the mIoU is considered as an important metric that is used in many studies [116,122]. Apart from the mIoU, the per-class pixel accuracy (PA), mean pixel accuracy (mPA), and frequency-weighted IoU (fwIoU) are applied to evaluate the segmented results at the pixel level.

5. Challenges and Future Work

This part first discusses the main challenges in recent studies, and some potential methods are then indicated to address these difficulties in the future work. Since a few surveys have already mentioned the partial limitations, a more complete summary of the existing challenges and future research direction are presented in this survey.

5.1. Data Analysis

During the data acquisition process, vision-based techniques such as the traditional CCTV are the most popular because of their cost-effective characteristics. Nevertheless, it is challenging for the visual equipment to inspect all the defects whenever they are below the water level or behind pipelines. As a result, the progress in hybrid devices has provided a feasible approach to acquire unavailable defects [126]. For example, the SSET methods [31,127,128] have been applied to collect quality data and evaluate the detected defects that are hard to deduce based on the visual data. In addition, the existing sewer inspection studies mainly focus on the concrete pipeline structures. The inspection and assessment for the traditional masonry sewer system that are still ubiquitous in most of the European cities become cumbersome for inspectors in practice. As for this issue, several automated diagnostic techniques (CCTV, laser scanning, ultrasound, etc.) for brick sewers are analyzed and compared in detail by enumerating the specific advantages and disadvantages [129,130]. Furthermore, varied qualities of the exiting datasets under distinct conditions and discontinuous backgrounds require image preprocessing prior to the inspection process to enhance the image quality and then improve the final performance [131].

Moreover, the current work concentrates on the research of structural defects such as cracks, joints, breaks, surface damage, lateral protrusion, and deformation, whereas there is less concern about the operation and maintenance defects (roots, infiltration, deposits, debris, and barriers). As mentioned in Section 4.1.2, there are 32 datasets investigated in this survey. Figure 14 shows the previous studies on sewer inspections of different classes of defects. We listed 12 classes of common defects in underground sewer pipelines. In addition to this, other defects that are rare and at the sub-class level are also included. According to the statistics for common defects, the proportion (50.5%) of structural defects is 20.3% higher than the proportion (30.2%) of operation and maintenance defects, which reflects that future research needs more available data for operation and maintenance defects.

5.2. Model Analysis

Although defect severity analysis methods have been proposed in several papers in order to assess the risk of the detected cracks, approaches for the analysis of other defects are limited. As for cracks, the risk levels can be assessed by measuring morphological features such as the crack length, mean width, and area to judge the corresponding severity degree. In contrast, it is difficult to comprehensively analyze the severity degrees for other defects because only the defect area is available for other defects. Therefore, researchers should explore more features that are closely related to the defect severity, which is significant for further condition assessment.

In addition, the major defect inspection models rely on effective supervised learning methods that cost much time in the manual annotation process for training [10]. The completely automated systems that include automatic labeling tools need to be developed for more efficient sewer inspections. On the other hand, most of the inspection approaches that demand long processing times only test based on still images, so these methods cannot be practiced in real-time applications for live inspections. More efforts should be made in future research to boost the inference speed in CCTV sewer videos.

6. Conclusions

Vision-based automation in construction has attracted increasing interest of researchers from different fields, especially with image processing and pattern recognition. The main outcomes of this paper include (1) an exhaustive review of diverse research approaches presented in more than 120 studies through a scientific taxonomy, (2) an analytical discussion of various algorithms, datasets, and evaluation protocols, and (3) a compendious summary of the existing challenges and future needs. Based on the current research situation, this survey outlines several suggestions that can facilitate future research on vision-based sewer inspection and condition assessment. Firstly, classification and detection have become a topic of great interest in the past several decades, which has attracted a lot of researchers’ attention. Compared with them, defect segmentation at the pixel level is a more significant task to assist the sewer inspectors in evaluating the risk level of the detected defects. However, it has the lowest proportion of the research of overall studies. Hence, automatic defect segmentation should be given greater focus considering its research significance. Secondly, we suggest that a public dataset and source code be created to support replicable research in the future. Thirdly, the evaluation metrics should be standardized for a fair performance comparison. Since this review presents clear guidelines for subsequent research by analyzing the concurrent studies, we believe it is of value to readers and practitioners who are concerned with sewer defect inspection and condition assessment.

Author Contributions

Conceptualization, Y.L. and H.W.; methodology, Y.L.; validation, L.M.D.; investigation, H.W. and H.-K.S.; writing—original draft preparation, Y.L.; writing—review and editing, H.W. and L.M.D.; supervision, H.M. All authors have read and agreed to the published version of the manuscript.

 Funding

This research received no external funding.

 Institutional Review Board Statement

Not applicable.

 Informed Consent Statement

Not applicable.

 Data Availability Statement

Not applicable.

 Acknowledgments

This work was supported by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (2020R1A6A1A03038540) and National Research Foundation of Korea (NRF) grant funded by the Korea government, Ministry of Science and ICT (MSIT) (2021R1F1A1046339) and by a grant(20212020900150) from “Development and Demonstration of Technology for Customers Bigdata-based Energy Management in the Field of Heat Supply Chain” funded by Ministry of Trade, Industry and Energy of Korean government.

Conflicts of Interest

There is no conflict of interest.

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Figures and Tables
View Image - Figure 1. There are five stages in the defect inspection framework, which include (a) the data acquisition stage based on various sensors (CCTV, sonar, or scanner), (b) the data processing stage for the collected data, (c) the defect inspection stage containing different algorithms (defect classification, detection, and segmentation), (d) the risk assessment for detected defects using image post-processing, and (e) the final report generation stage for the condition evaluation.Enlarge this image.

Figure 1. There are five stages in the defect inspection framework, which include (a) the data acquisition stage based on various sensors (CCTV, sonar, or scanner), (b) the data processing stage for the collected data, (c) the defect inspection stage containing different algorithms (defect classification, detection, and segmentation), (d) the risk assessment for detected defects using image post-processing, and (e) the final report generation stage for the condition evaluation.

View Image - Figure 2. Number of journal publications investigated in different databases.Enlarge this image.

Figure 2. Number of journal publications investigated in different databases.

View Image - Figure 3. Number of publications investigated in different time periods (from 2000 to 2022).Enlarge this image.

Figure 3. Number of publications investigated in different time periods (from 2000 to 2022).

View Image - Figure 4. The classification map of the existing algorithms for each category. The dotted boxes represent the main stages of the algorithms.Enlarge this image.

Figure 4. The classification map of the existing algorithms for each category. The dotted boxes represent the main stages of the algorithms.

View Image - Figure 5. Proportions of the investigated studies using different inspection algorithms.Enlarge this image.

Figure 5. Proportions of the investigated studies using different inspection algorithms.

View Image - Figure 6. Optimal separation hyperplane.Enlarge this image.

Figure 6. Optimal separation hyperplane.

View Image - Figure 7. A fine-tuned network architecture used for defect classification.Enlarge this image.

Figure 7. A fine-tuned network architecture used for defect classification.

View Image - Figure 8. The YOLOv3 architecture with the 3-scale prediction mechanism.Enlarge this image.

Figure 8. The YOLOv3 architecture with the 3-scale prediction mechanism.

View Image - Figure 9. The model architecture of the SSD model.Enlarge this image.

Figure 9. The model architecture of the SSD model.

View Image - Figure 10. An architecture of the faster R-CNN developed for defect detection. ‘CNN’ refers to convolutional neural network. ‘ROI’ means region of interest. ‘FC’ is fully connected layer.Enlarge this image.

Figure 10. An architecture of the faster R-CNN developed for defect detection. ‘CNN’ refers to convolutional neural network. ‘ROI’ means region of interest. ‘FC’ is fully connected layer.

View Image - Figure 11. An architecture of PipeUNet proposed for semantic segmentation.Enlarge this image.

Figure 11. An architecture of PipeUNet proposed for semantic segmentation.

View Image - Figure 12. Representative inspection robots for data acquisition. (a) LETS 6.0, (b) Robocam 6, (c) X5-HS, and (d) Robocam 3000.Enlarge this image.

Figure 12. Representative inspection robots for data acquisition. (a) LETS 6.0, (b) Robocam 6, (c) X5-HS, and (d) Robocam 3000.

View Image - Figure 13. Sample images from the Sewer-ML dataset that has a wide diversity of materials and shapes.Enlarge this image.

Figure 13. Sample images from the Sewer-ML dataset that has a wide diversity of materials and shapes.

View Image - Figure 14. Studies on sewer inspections of different classes of defects.Enlarge this image.

Figure 14. Studies on sewer inspections of different classes of defects.

Major contributions of the previous review papers on defect inspection and condition assessment. ‘√’ indicates the research areas (defect inspection or condition assessment) are involved. ‘×’ means the research areas (defect inspection or condition assessment) are not involved.

ID Ref. Time Defect Inspection Condition Assessment Contributions
1 [24] 2019 √ √

  • Analyze the status of practical defect detection and condition assessment technologies.

  • Compare the benefits and drawbacks of the reviewed work.
2 [10] 2020 √ ×

  • Introduce defect inspection methods that are suitable for different materials.

  • Provide a taxonomy of various defects.

  • List the state-of-the-art (SOTA) methods for the classification and detection.
3 [7] 2020 √ ×

  • Create a brief overview of the defect inspection algorithms, datasets, and evaluation metrics.

  • Indicate three recommendations for the future research.
4 [20] 2020 × √

  • Investigate different models for the condition assessment.

  • Analyze the influence factors of the reviewed models on the sewer conditions.
5 [8] 2021 √ √

  • Present a review for main applications, advantages, and possible research areas.

The detailed information of the latest robots for sewer inspection.

Name Company Pipe Diameter Camera Feature Country Strong Point
CAM160 (https://goolnk.com/YrYQob accessed on 20 February 2022) Sewer Robotics 200–500 mm NA USA ● Auto horizon adjustment● Intensity adjustable LED lighting● Multifunctional
LETS 6.0 (https://ariesindustries.com/products/ accessed on 20 February 2022) ARIES INDUSTRIES 150 mm or larger Self-leveling lateral camera or a Pan and tilt camera USA ● Slim tractor profile● Superior lateral camera● Simultaneously acquire mainline and lateral videos
wolverine® 2.02 ARIES INDUSTRIES 150–450 mm NA USA ● Powerful crawler to maneuver obstacles● Minimum set uptime● Camera with lens cleaning technique
X5-HS (https://goolnk.com/Rym02W accessed on 20 February 2022) EASY-SIGHT 300–3000 mm ≥2 million pixels China ● High-definition● Freely choose wireless and wired connection and control● Display and save videos in real time
Robocam 6 (https://goolnk.com/43pdGA accessed on 20 February 2022) TAP Electronics 600 mm or more Sony 130-megapixel Exmor 1/3-inch CMOS Korea ● High-resolution● All-in-one subtitle system
RoboCam Innovation4 TAP Electronics 600 mm or more Sony 130-megapixel Exmor 1/3-inch CMOS Korea ● Best digital record performance● Super white LED lighting● Cableless
Robocam 30004 TAP Electronics’ Japanese subsidiary 250–3000 mm Sony 1.3-megapixel Exmor CMOS color Japan ● Can be utilized in huge pipelines● Optical 10X zoom performance

Research datasets for sewer defects in recent studies.

ID Defect Type Image Resolution Equipment Number of Images Country Ref.
1 Broken, crack, deposit, fracture, hole, root, tap NA NA 4056 Canada [9]
2 Connection, crack, debris, deposit, infiltration, material change, normal, root 1440 × 720–320 × 256 RedZone®Solo CCTV crawler 12,000 USA [48]
3 Attached deposit, defective connection, displaced joint, fissure, infiltration, ingress, intruding connection, porous, root, sealing, settled deposit, surface 1040 × 1040 Front-facing and back-facing camera with a 185∘ wide lens 2,202,582 The Netherlands [49]
4 Dataset 1: defective, normal NA NA 40,000 China [69]
Dataset 2: barrier, deposit, disjunction, fracture, stagger, water 15,000
5 Broken, deformation, deposit, other, joint offset, normal, obstacle, water 1435 × 1054–296 × 166 NA 18,333 China [70]
6 Attached deposits, collapse, deformation, displaced joint, infiltration, joint damage, settled deposit NA NA 1045 China [41]
7 Circumferential crack, longitudinal crack, multiple crack 320 × 240 NA 335 USA [11]
8 Debris, joint faulty, joint open, longitudinal, protruding, surface NA Robo Cam 6 with a 1/3-in. SONY Exmor CMOS camera 48,274 South Korea [71]
9 Broken, crack, debris, joint faulty, joint open, normal, protruding, surface 1280 × 720 Robo Cam 6 with a megapixel Exmor CMOS sensor 115,170 South Korea [52]
10 Crack, deposit, else, infiltration, joint, root, surface NA Remote cameras 2424 UK [66]
11 Broken, crack, deposit, fracture, hole, root, tap NA NA 1451 Canada [104]
12 Crack, deposit, infiltration, root 1440 × 720–320 × 256 RedZone® Solo CCTV crawler 3000 USA [98]
13 Connection, fracture, root 1507 × 720–720 × 576 Front facing CCTV cameras 3600 USA [99]
14 Crack, deposit, root 928 × 576–352 × 256 NA 3000 USA [97]
15 Crack, deposit, root 512 × 256 NA 1880 USA [116]
16 Crack, infiltration, joint, protruding 1073 × 749–296 × 237 NA 1106 China [122]
17 Crack, non-crack 64 × 64 NA 40,810 Australia [109]
18 Crack, normal, spalling 4000 × 46,000–3168 × 4752 Canon EOS. Tripods and stabilizers 294 China [73]
19 Collapse, crack, root NA SSET system 239 USA [61]
20 Clean pipe, collapsed pipe, eroded joint, eroded lateral, misaligned joint, perfect joint, perfect lateral NA SSET system 500 USA [56]
21 Cracks, joint, reduction, spalling 512 × 512 CCTV or Aqua Zoom camera 1096 Canada [54]
22 Defective, normal NA CCTV (Fisheye) 192 USA [57]
23 Deposits, normal, root 1507 × 720–720 × 576 Front-facing CCTV cameras 3800 USA [72]
24 Crack, non-crack 240 × 320 CCTV 200 South Korea [106]
25 Faulty, normal NA CCTV 8000 UK [65]
26 Blur, deposition, intrusion, obstacle NA CCTV 12,000 NA [67]
27 Crack, deposit, displaced joint, ovality NA CCTV (Fisheye) 32 Qatar [103]
29 Crack, non-crack 320 × 240–20 × 20 CCTV 100 NA [100]
30 Barrier, deposition, distortion, fraction, inserted 600 × 480 CCTV and quick-view (QV) cameras 10,000 China [110]
31 Fracture NA CCTV 2100 USA [105]
32 Broken, crack, fracture, joint open NA CCTV 291 China [59]

Overview of the evaluation metrics in the recent studies.

Metric Description Ref.
Precision The proportion of positive samples in all positive prediction samples [9]
Recall The proportion of positive prediction samples in all positive samples [48]
Accuracy The proportion of correct prediction in all prediction samples [48]
F1-score Harmonic mean of precision and recall [69]
FAR False alarm rate in all prediction samples [57]
True accuracy The proportion of all predictions excluding the missed defective images among the entire actual images [58]
AUROC Area under the receiver operator characteristic (ROC) curve [49]
AUPR Area under the precision-recall curve [49]
mAP mAP first calculates the average precision values for different recall values for one class, and then takes the average of all classes [9]
Detection rate The ratio of the number of the detected defects to total number of defects [106]
Error rate The ratio of the number of mistakenly detected defects to the number of non-defects [106]
PA Pixel accuracy calculating the overall accuracy of all pixels in the image [116]
mPA The average of pixel accuracy for all categories [116]
mIoU The ratio of intersection and union between predictions and GTs [116]
fwIoU Frequency-weighted IoU measuring the mean IoU value weighing the pixel frequency of each class [116]

Performances of different algorithms in terms of different evaluation metrics.

ID Number of Images Algorithm Task Performance Ref.
Accuracy (%) Processing Speed
1 3 classes Multiple binary CNNs Classification Accuracy: 86.2Precision: 87.7Recall: 90.6 NA [48]
2 12 classes Single CNN Classification AUROC: 87.1AUPR: 6.8 NA [48]
3 Dataset 1: 2 classes Two-level hierarchical CNNs Classification Accuracy: 94.5Precision: 96.84Recall: 92F1-score: 94.36 1.109 h for 200 videos [69]
Dataset 2: 6 classes Accuracy: 94.96Precision: 85.13Recall: 84.61F1-score: 84.86
4 8 classes Deep CNN Classification Accuracy: 64.8 NA [70]
5 6 classes CNN Classification Accuracy: 96.58 NA [71]
6 8 classes CNN Classification Accuracy: 97.6 0.15 s/image [52]
7 7 classes Multi-class random forest Classification Accuracy: 71 25 FPS [66]
8 7 classes SVM Classification Accuracy: 84.1 NA [41]
9 3 classes SVM Classification Recall: 90.3Precision: 90.3 10 FPS [11]
10 3 classes CNN Classification Accuracy: 96.7Precision: 99.8Recall: 93.6F1-score: 96.6 15 min 30 images [73]
11 3 classes RotBoost and statistical feature vector Classification Accuracy: 89.96 1.5 s/image [61]
12 7 classes Neuro-fuzzy classifier Classification Accuracy: 91.36 NA [56]
13 4 classes Multi-layer perceptions Classification Accuracy: 98.2 NA [54]
14 2 classes Rule-based classifier Classification Accuracy: 87FAR: 18Recall: 89 NA [57]
15 2 classes OCSVM Classification Accuracy: 75 NA [65]
16 4 classes CNN Classification Recall: 88Precision: 84Accuracy: 85 NA [67]
17 2 class Rule-based classifier Classification Accuracy: 84FAR: 21True accuracy: 95 NA [58]
18 4 classes RBN Classification Accuracy: 95 NA [59]
19 7 classes YOLOv3 Detection mAP: 85.37 33 FPS [9]
20 4 classes Faster R-CNN Detection mAP: 83 9 FPS [98]
21 3 classes Faster R-CNN Detection mAP: 77 110 ms/image [99]
22 3 classes Faster R-CNN Detection Precision: 88.99Recall: 87.96F1-score: 88.21 110 ms/image [97]
23 2 classes CNN Detection Accuracy: 96Precision: 90 0.2782 s/image [109]
24 3 classes Faster R-CNN Detection mAP: 71.8 110 ms/image [105]
SSD mAP: 69.5 57 ms/image
YOLOv3 mAP: 53 33 ms/image
25 2 classes Rule-based detector Detection Detection rate: 89.2Error rate: 4.44 1 FPS [106]
26 2 classes GA and CNN Detection Detection rate: 92.3 NA [100]
27 5 classes SRPN Detection mAP: 50.8Recall: 82.4 153 ms/image [110]
28 1 class CNN and YOLOv3 Detection AP: 71 65 ms/image [108]
29 3 classes DilaSeg-CRF Segmentation PA: 98.69mPA: 91.57mIoU: 84.85fwIoU: 97.47 107 ms/image [116]
30 4 classes PipeUNet Segmentation mIoU: 76.37 32 FPS [122]

References

1. The 2019 Canadian Infrastructure Report Card (CIRC). 2019; Available online: http://canadianinfrastructure.ca/downloads/canadian-infrastructure-report-card-2019.pdf (accessed on 20 February 2022).

2. Tscheikner-Gratl, F.; Caradot, N.; Cherqui, F.; Leitão, J.P.; Ahmadi, M.; Langeveld, J.G.; Le Gat, Y.; Scholten, L.; Roghani, B.; Rodríguez, J.P. et al. Sewer asset management–state of the art and research needs. Urban Water J.; 2019; 16, pp. 662-675. [DOI: https://dx.doi.org/10.1080/1573062X.2020.1713382]

3. 2021 Report Card for America’s Infrastructure 2021 Wastewater. Available online: https://infrastructurereportcard.org/wp-content/uploads/2020/12/Wastewater-2021.pdf (accessed on 20 February 2022).

4. Spencer, B.F., Jr.; Hoskere, V.; Narazaki, Y. Advances in computer vision-based civil infrastructure inspection and monitoring. Engineering; 2019; 5, pp. 199-222. [DOI: https://dx.doi.org/10.1016/j.eng.2018.11.030]

5. Duran, O.; Althoefer, K.; Seneviratne, L.D. State of the art in sensor technologies for sewer inspection. IEEE Sens. J.; 2002; 2, pp. 73-81. [DOI: https://dx.doi.org/10.1109/JSEN.2002.1000245]

6. 2021 Global Green Growth Institute. 2021; Available online: http://gggi.org/site/assets/uploads/2019/01/Wastewater-System-Operation-and-Maintenance-Guideline-1.pdf (accessed on 20 February 2022).

7. Haurum, J.B.; Moeslund, T.B. A Survey on image-based automation of CCTV and SSET sewer inspections. Autom. Constr.; 2020; 111, 103061. [DOI: https://dx.doi.org/10.1016/j.autcon.2019.103061]

8. Mostafa, K.; Hegazy, T. Review of image-based analysis and applications in construction. Autom. Constr.; 2021; 122, 103516. [DOI: https://dx.doi.org/10.1016/j.autcon.2020.103516]

9. Yin, X.; Chen, Y.; Bouferguene, A.; Zaman, H.; Al-Hussein, M.; Kurach, L. A deep learning-based framework for an automated defect detection system for sewer pipes. Autom. Constr.; 2020; 109, 102967. [DOI: https://dx.doi.org/10.1016/j.autcon.2019.102967]

10. Czimmermann, T.; Ciuti, G.; Milazzo, M.; Chiurazzi, M.; Roccella, S.; Oddo, C.M.; Dario, P. Visual-based defect detection and classification approaches for industrial applications—A survey. Sensors; 2020; 20, 1459. [DOI: https://dx.doi.org/10.3390/s20051459]

11. Zuo, X.; Dai, B.; Shan, Y.; Shen, J.; Hu, C.; Huang, S. Classifying cracks at sub-class level in closed circuit television sewer inspection videos. Autom. Constr.; 2020; 118, 103289. [DOI: https://dx.doi.org/10.1016/j.autcon.2020.103289]

12. Dang, L.M.; Hassan, S.I.; Im, S.; Mehmood, I.; Moon, H. Utilizing text recognition for the defects extraction in sewers CCTV inspection videos. Comput. Ind.; 2018; 99, pp. 96-109. [DOI: https://dx.doi.org/10.1016/j.compind.2018.03.020]

13. Li, C.; Lan, H.-Q.; Sun, Y.-N.; Wang, J.-Q. Detection algorithm of defects on polyethylene gas pipe using image recognition. Int. J. Press. Vessel. Pip.; 2021; 191, 104381. [DOI: https://dx.doi.org/10.1016/j.ijpvp.2021.104381]

14. Sinha, S.K.; Fieguth, P.W. Segmentation of buried concrete pipe images. Autom. Constr.; 2006; 15, pp. 47-57. [DOI: https://dx.doi.org/10.1016/j.autcon.2005.02.007]

15. Yang, M.-D.; Su, T.-C. Automated diagnosis of sewer pipe defects based on machine learning approaches. Expert Syst. Appl.; 2008; 35, pp. 1327-1337. [DOI: https://dx.doi.org/10.1016/j.eswa.2007.08.013]

16. McKim, R.A.; Sinha, S.K. Condition assessment of underground sewer pipes using a modified digital image processing paradigm. Tunn. Undergr. Space Technol.; 1999; 14, pp. 29-37. [DOI: https://dx.doi.org/10.1016/S0886-7798(00)00021-3]

17. Dirksen, J.; Clemens, F.; Korving, H.; Cherqui, F.; Le Gauffre, P.; Ertl, T.; Plihal, H.; Müller, K.; Snaterse, C. The consistency of visual sewer inspection data. Struct. Infrastruct. Eng.; 2013; 9, pp. 214-228. [DOI: https://dx.doi.org/10.1080/15732479.2010.541265]

18. van der Steen, A.J.; Dirksen, J.; Clemens, F.H. Visual sewer inspection: Detail of coding system versus data quality?. Struct. Infrastruct. Eng.; 2014; 10, pp. 1385-1393. [DOI: https://dx.doi.org/10.1080/15732479.2013.816974]

19. Hsieh, Y.-A.; Tsai, Y.J. Machine learning for crack detection: Review and model performance comparison. J. Comput. Civ. Eng.; 2020; 34, 04020038. [DOI: https://dx.doi.org/10.1061/(ASCE)CP.1943-5487.0000918]

20. Hawari, A.; Alkadour, F.; Elmasry, M.; Zayed, T. A state of the art review on condition assessment models developed for sewer pipelines. Eng. Appl. Artif. Intell.; 2020; 93, 103721. [DOI: https://dx.doi.org/10.1016/j.engappai.2020.103721]

21. Yang, X.; Li, H.; Yu, Y.; Luo, X.; Huang, T.; Yang, X. Automatic pixel-level crack detection and measurement using fully convolutional network. Comput.-Aided Civ. Infrastruct. Eng.; 2018; 33, pp. 1090-1109. [DOI: https://dx.doi.org/10.1111/mice.12412]

22. Li, G.; Ma, B.; He, S.; Ren, X.; Liu, Q. Automatic tunnel crack detection based on u-net and a convolutional neural network with alternately updated clique. Sensors; 2020; 20, 717. [DOI: https://dx.doi.org/10.3390/s20030717]

23. Wang, M.; Luo, H.; Cheng, J.C. Severity Assessment of Sewer Pipe Defects in Closed-Circuit Television (CCTV) Images Using Computer Vision Techniques. Proceedings of the Construction Research Congress 2020: Infrastructure Systems and Sustainability; Tempe, AZ, USA, 8–10 March 2020; American Society of Civil Engineers: Reston, VA, USA, pp. 942-950.

24. Moradi, S.; Zayed, T.; Golkhoo, F. Review on computer aided sewer pipeline defect detection and condition assessment. Infrastructures; 2019; 4, 10. [DOI: https://dx.doi.org/10.3390/infrastructures4010010]

25. Yang, M.-D.; Su, T.-C.; Pan, N.-F.; Yang, Y.-F. Systematic image quality assessment for sewer inspection. Expert Syst. Appl.; 2011; 38, pp. 1766-1776. [DOI: https://dx.doi.org/10.1016/j.eswa.2010.07.103]

26. Henriksen, K.S.; Lynge, M.S.; Jeppesen, M.D.; Allahham, M.M.; Nikolov, I.A.; Haurum, J.B.; Moeslund, T.B. Generating synthetic point clouds of sewer networks: An initial investigation. Proceedings of the International Conference on Augmented Reality, Virtual Reality and Computer Graphics; Lecce, Italy, 7–10 September 2020; Springer: Berlin/Heidelberg, Germany, 2020; pp. 364-373.

27. Alejo, D.; Caballero, F.; Merino, L. RGBD-based robot localization in sewer networks. Proceedings of the 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS); Vancouver, BC, Canada, 24–28 September 2017; IEEE: Piscataway, NJ, USA, 2017; pp. 4070-4076.

28. Lepot, M.; Stanić, N.; Clemens, F.H. A technology for sewer pipe inspection (Part 2): Experimental assessment of a new laser profiler for sewer defect detection and quantification. Autom. Constr.; 2017; 73, pp. 1-11. [DOI: https://dx.doi.org/10.1016/j.autcon.2016.10.010]

29. Duran, O.; Althoefer, K.; Seneviratne, L.D. Automated pipe defect detection and categorization using camera/laser-based profiler and artificial neural network. IEEE Trans. Autom. Sci. Eng.; 2007; 4, pp. 118-126. [DOI: https://dx.doi.org/10.1109/TASE.2006.873225]

30. Khan, M.S.; Patil, R. Acoustic characterization of pvc sewer pipes for crack detection using frequency domain analysis. Proceedings of the 2018 IEEE International Smart Cities Conference (ISC2); Kansas City, MO, USA, 16–19 September 2018; IEEE: Piscataway, NJ, USA, 2018; pp. 1-5.

31. Iyer, S.; Sinha, S.K.; Pedrick, M.K.; Tittmann, B.R. Evaluation of ultrasonic inspection and imaging systems for concrete pipes. Autom. Constr.; 2012; 22, pp. 149-164. [DOI: https://dx.doi.org/10.1016/j.autcon.2011.06.013]

32. Li, Y.; Wang, H.; Dang, L.M.; Sadeghi-Niaraki, A.; Moon, H. Crop pest recognition in natural scenes using convolutional neural networks. Comput. Electron. Agric.; 2020; 169, 105174. [DOI: https://dx.doi.org/10.1016/j.compag.2019.105174]

33. Wang, H.; Li, Y.; Dang, L.M.; Ko, J.; Han, D.; Moon, H. Smartphone-based bulky waste classification using convolutional neural networks. Multimed. Tools Appl.; 2020; 79, pp. 29411-29431. [DOI: https://dx.doi.org/10.1007/s11042-020-09571-5]

34. Hassan, S.I.; Dang, L.-M.; Im, S.-H.; Min, K.-B.; Nam, J.-Y.; Moon, H.-J. Damage Detection and Classification System for Sewer Inspection using Convolutional Neural Networks based on Deep Learning. J. Korea Inst. Inf. Commun. Eng.; 2018; 22, pp. 451-457.

35. Scholkopf, B. Support vector machines: A practical consequence of learning theory. IEEE Intell. Syst.; 1998; 13, 4. [DOI: https://dx.doi.org/10.1041/X4018s-1998]

36. Li, X.; Wang, L.; Sung, E. A study of AdaBoost with SVM based weak learners. Proceedings of the 2005 IEEE International Joint Conference on Neural Networks; Montreal, QC, Canada, 31 July–4 August 2005; IEEE: Piscataway, NJ, USA, 2005; pp. 196-201.

37. Han, H.; Jiang, X. Overcome support vector machine diagnosis overfitting. Cancer Inform.; 2014; 13, pp. 145-158. [DOI: https://dx.doi.org/10.4137/CIN.S13875]

38. Shao, Y.-H.; Chen, W.-J.; Deng, N.-Y. Nonparallel hyperplane support vector machine for binary classification problems. Inf. Sci.; 2014; 263, pp. 22-35. [DOI: https://dx.doi.org/10.1016/j.ins.2013.11.003]

39. Cervantes, J.; Garcia-Lamont, F.; Rodríguez-Mazahua, L.; Lopez, A. A comprehensive survey on support vector machine classification: Applications, challenges and trends. Neurocomputing; 2020; 408, pp. 189-215. [DOI: https://dx.doi.org/10.1016/j.neucom.2019.10.118]

40. Xu, Y.; Yang, J.; Zhao, S.; Wu, H.; Sawan, M. An end-to-end deep learning approach for epileptic seizure prediction. Proceedings of the 2020 2nd IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS); Genoa, Italy, 31 August–2 September 2020; IEEE: Piscataway, NJ, USA, 2020; pp. 266-270.

41. Ye, X.; Zuo, J.; Li, R.; Wang, Y.; Gan, L.; Yu, Z.; Hu, X. Diagnosis of sewer pipe defects on image recognition of multi-features and support vector machine in a southern Chinese city. Front. Environ. Sci. Eng.; 2019; 13, 17. [DOI: https://dx.doi.org/10.1007/s11783-019-1102-y]

42. Hu, M.-K. Visual pattern recognition by moment invariants. IRE Trans. Inf. Theory; 1962; 8, pp. 179-187.

43. Quiroga, J.A.; Crespo, D.; Bernabeu, E. Fourier transform method for automatic processing of moiré deflectograms. Opt. Eng.; 1999; 38, pp. 974-982. [DOI: https://dx.doi.org/10.1117/1.602138]

44. Tomasi, C.; Manduchi, R. Bilateral filtering for gray and color images. Proceedings of the Sixth International Conference on Computer Vision (IEEE Cat. No. 98CH36271); Bombay, IN, USA, 7 January 1998; IEEE: Piscataway, NJ, USA, 1998; pp. 839-846.

45. Gavaskar, R.G.; Chaudhury, K.N. Fast adaptive bilateral filtering. IEEE Trans. Image Process.; 2018; 28, pp. 779-790. [DOI: https://dx.doi.org/10.1109/TIP.2018.2871597]

46. Hubel, D.H.; Wiesel, T.N. Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. J. Physiol.; 1962; 160, pp. 106-154. [DOI: https://dx.doi.org/10.1113/jphysiol.1962.sp006837]

47. LeCun, Y.; Bengio, Y.; Hinton, G. Deep learning. Nature; 2015; 521, pp. 436-444. [DOI: https://dx.doi.org/10.1038/nature14539]

48. Kumar, S.S.; Abraham, D.M.; Jahanshahi, M.R.; Iseley, T.; Starr, J. Automated defect classification in sewer closed circuit television inspections using deep convolutional neural networks. Autom. Constr.; 2018; 91, pp. 273-283. [DOI: https://dx.doi.org/10.1016/j.autcon.2018.03.028]

49. Meijer, D.; Scholten, L.; Clemens, F.; Knobbe, A. A defect classification methodology for sewer image sets with convolutional neural networks. Autom. Constr.; 2019; 104, pp. 281-298. [DOI: https://dx.doi.org/10.1016/j.autcon.2019.04.013]

50. Cheng, H.-D.; Shi, X. A simple and effective histogram equalization approach to image enhancement. Digit. Signal Process.; 2004; 14, pp. 158-170. [DOI: https://dx.doi.org/10.1016/j.dsp.2003.07.002]

51. Salazar-Colores, S.; Ramos-Arreguín, J.-M.; Echeverri, C.J.O.; Cabal-Yepez, E.; Pedraza-Ortega, J.-C.; Rodriguez-Resendiz, J. Image dehazing using morphological opening, dilation and Gaussian filtering. Signal Image Video Process.; 2018; 12, pp. 1329-1335. [DOI: https://dx.doi.org/10.1007/s11760-018-1286-9]

52. Dang, L.M.; Kyeong, S.; Li, Y.; Wang, H.; Nguyen, T.N.; Moon, H. Deep Learning-based Sewer Defect Classification for Highly Imbalanced Dataset. Comput. Ind. Eng.; 2021; 161, 107630. [DOI: https://dx.doi.org/10.1016/j.cie.2021.107630]

53. Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv; 2014; arXiv: 1409.1556

54. Moselhi, O.; Shehab-Eldeen, T. Classification of defects in sewer pipes using neural networks. J. Infrastruct. Syst.; 2000; 6, pp. 97-104. [DOI: https://dx.doi.org/10.1061/(ASCE)1076-0342(2000)6:3(97)]

55. Sinha, S.K.; Karray, F. Classification of underground pipe scanned images using feature extraction and neuro-fuzzy algorithm. IEEE Trans. Neural Netw.; 2002; 13, pp. 393-401. [DOI: https://dx.doi.org/10.1109/72.991425] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/18244440]

56. Sinha, S.K.; Fieguth, P.W. Neuro-fuzzy network for the classification of buried pipe defects. Autom. Constr.; 2006; 15, pp. 73-83. [DOI: https://dx.doi.org/10.1016/j.autcon.2005.02.005]

57. Guo, W.; Soibelman, L.; Garrett, J., Jr. Visual pattern recognition supporting defect reporting and condition assessment of wastewater collection systems. J. Comput. Civ. Eng.; 2009; 23, pp. 160-169. [DOI: https://dx.doi.org/10.1061/(ASCE)0887-3801(2009)23:3(160)]

58. Guo, W.; Soibelman, L.; Garrett, J., Jr. Automated defect detection for sewer pipeline inspection and condition assessment. Autom. Constr.; 2009; 18, pp. 587-596. [DOI: https://dx.doi.org/10.1016/j.autcon.2008.12.003]

59. Yang, M.-D.; Su, T.-C. Segmenting ideal morphologies of sewer pipe defects on CCTV images for automated diagnosis. Expert Syst. Appl.; 2009; 36, pp. 3562-3573. [DOI: https://dx.doi.org/10.1016/j.eswa.2008.02.006]

60. Ganegedara, H.; Alahakoon, D.; Mashford, J.; Paplinski, A.; Müller, K.; Deserno, T.M. Self organising map based region of interest labelling for automated defect identification in large sewer pipe image collections. Proceedings of the 2012 International Joint Conference on Neural Networks (IJCNN); Brisbane, Australia, 10–15 June 2012; IEEE: Piscataway, NJ, USA, 2012; pp. 1-8.

61. Wu, W.; Liu, Z.; He, Y. Classification of defects with ensemble methods in the automated visual inspection of sewer pipes. Pattern Anal. Appl.; 2015; 18, pp. 263-276. [DOI: https://dx.doi.org/10.1007/s10044-013-0355-5]

62. Myrans, J.; Kapelan, Z.; Everson, R. Automated detection of faults in wastewater pipes from CCTV footage by using random forests. Procedia Eng.; 2016; 154, pp. 36-41. [DOI: https://dx.doi.org/10.1016/j.proeng.2016.07.416]

63. Myrans, J.; Kapelan, Z.; Everson, R. Automatic detection of sewer faults using continuous CCTV footage. Computing & Control for the Water Industry; 2017; Available online: https://easychair.org/publications/open/ZQH3 (accessed on 20 February 2022).

64. Moradi, S.; Zayed, T. Real-time defect detection in sewer closed circuit television inspection videos. Pipelines 2017; ASCE: Reston, VA, USA, 2017; pp. 295-307.

65. Myrans, J.; Kapelan, Z.; Everson, R. Using automatic anomaly detection to identify faults in sewers. WDSA/CCWI Joint Conference Proceedings; 2018; Available online: https://ojs.library.queensu.ca/index.php/wdsa-ccw/article/view/12030 (accessed on 20 February 2022).

66. Myrans, J.; Kapelan, Z.; Everson, R. Automatic identification of sewer fault types using CCTV footage. EPiC Ser. Eng.; 2018; 3, pp. 1478-1485.

67. Chen, K.; Hu, H.; Chen, C.; Chen, L.; He, C. An intelligent sewer defect detection method based on convolutional neural network. Proceedings of the 2018 IEEE International Conference on Information and Automation (ICIA); Wuyishan, China, 11–13 August 2018; IEEE: Piscataway, NJ, USA, 2018; pp. 1301-1306.

68. Moradi, S.; Zayed, T.; Golkhoo, F. Automated sewer pipeline inspection using computer vision techniques. Pipelines 2018: Condition Assessment, Construction, and Rehabilitation; American Society of Civil Engineers: Reston, VA, USA, 2018; pp. 582-587.

69. Xie, Q.; Li, D.; Xu, J.; Yu, Z.; Wang, J. Automatic detection and classification of sewer defects via hierarchical deep learning. IEEE Trans. Autom. Sci. Eng.; 2019; 16, pp. 1836-1847. [DOI: https://dx.doi.org/10.1109/TASE.2019.2900170]

70. Li, D.; Cong, A.; Guo, S. Sewer damage detection from imbalanced CCTV inspection data using deep convolutional neural networks with hierarchical classification. Autom. Constr.; 2019; 101, pp. 199-208. [DOI: https://dx.doi.org/10.1016/j.autcon.2019.01.017]

71. Hassan, S.I.; Dang, L.M.; Mehmood, I.; Im, S.; Choi, C.; Kang, J.; Park, Y.-S.; Moon, H. Underground sewer pipe condition assessment based on convolutional neural networks. Autom. Constr.; 2019; 106, 102849. [DOI: https://dx.doi.org/10.1016/j.autcon.2019.102849]

72. Khan, S.M.; Haider, S.A.; Unwala, I. A Deep Learning Based Classifier for Crack Detection with Robots in Underground Pipes. Proceedings of the 2020 IEEE 17th International Conference on Smart Communities: Improving Quality of Life Using ICT, IoT and AI (HONET); Charlotte, NC, USA, 14–16 December 2020; IEEE: Piscataway, NJ, USA, 2020; pp. 78-81.

73. Chow, J.K.; Su, Z.; Wu, J.; Li, Z.; Tan, P.S.; Liu, K.-f.; Mao, X.; Wang, Y.-H. Artificial intelligence-empowered pipeline for image-based inspection of concrete structures. Autom. Constr.; 2020; 120, 103372. [DOI: https://dx.doi.org/10.1016/j.autcon.2020.103372]

74. Klusek, M.; Szydlo, T. Supporting the Process of Sewer Pipes Inspection Using Machine Learning on Embedded Devices. Proceedings of the International Conference on Computational Science; Krakow, Poland, 16–18 June 2021; Springer: Berlin/Heidelberg, Germany, 2021; pp. 347-360.

75. Sumalee, A.; Ho, H.W. Smarter and more connected: Future intelligent transportation system. Iatss Res.; 2018; 42, pp. 67-71. [DOI: https://dx.doi.org/10.1016/j.iatssr.2018.05.005]

76. Veres, M.; Moussa, M. Deep learning for intelligent transportation systems: A survey of emerging trends. IEEE Trans. Intell. Transp. Syst.; 2019; 21, pp. 3152-3168. [DOI: https://dx.doi.org/10.1109/TITS.2019.2929020]

77. Li, Y.; Wang, H.; Dang, L.M.; Nguyen, T.N.; Han, D.; Lee, A.; Jang, I.; Moon, H. A deep learning-based hybrid framework for object detection and recognition in autonomous driving. IEEE Access; 2020; 8, pp. 194228-194239. [DOI: https://dx.doi.org/10.1109/ACCESS.2020.3033289]

78. Yahata, S.; Onishi, T.; Yamaguchi, K.; Ozawa, S.; Kitazono, J.; Ohkawa, T.; Yoshida, T.; Murakami, N.; Tsuji, H. A hybrid machine learning approach to automatic plant phenotyping for smart agriculture. Proceedings of the 2017 International Joint Conference on Neural Networks (IJCNN); Anchorage, AK, USA, 14–19 May 2017; IEEE: Piscataway, NJ, USA, 2017; pp. 1787-1793.

79. Boukhris, L.; Abderrazak, J.B.; Besbes, H. Tailored Deep Learning based Architecture for Smart Agriculture. Proceedings of the 2020 International Wireless Communications and Mobile Computing (IWCMC); Limassol, Cyprus, 15–19 June 2020; IEEE: Piscataway, NJ, USA, 2020; pp. 964-969.

80. Wu, D.; Lv, S.; Jiang, M.; Song, H. Using channel pruning-based YOLO v4 deep learning algorithm for the real-time and accurate detection of apple flowers in natural environments. Comput. Electron. Agric.; 2020; 178, 105742. [DOI: https://dx.doi.org/10.1016/j.compag.2020.105742]

81. Melenbrink, N.; Rinderspacher, K.; Menges, A.; Werfel, J. Autonomous anchoring for robotic construction. Autom. Constr.; 2020; 120, 103391. [DOI: https://dx.doi.org/10.1016/j.autcon.2020.103391]

82. Lee, D.; Kim, M. Autonomous construction hoist system based on deep reinforcement learning in high-rise building construction. Autom. Constr.; 2021; 128, 103737. [DOI: https://dx.doi.org/10.1016/j.autcon.2021.103737]

83. Tan, Y.; Cai, R.; Li, J.; Chen, P.; Wang, M. Automatic detection of sewer defects based on improved you only look once algorithm. Autom. Constr.; 2021; 131, 103912. [DOI: https://dx.doi.org/10.1016/j.autcon.2021.103912]

84. Redmon, J.; Divvala, S.; Girshick, R.; Farhadi, A. You only look once: Unified, real-time object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; Honolulu, HI, USA, 21–26 July 2016; pp. 779-788.

85. Liu, W.; Anguelov, D.; Erhan, D.; Szegedy, C.; Reed, S.; Fu, C.-Y.; Berg, A.C. Ssd: Single shot multibox detector. Proceedings of the European Conference on Computer Vision; Munich, Germany, 8–14 September 2016; Springer: Berlin/Heidelberg, Germany, 2016; pp. 21-37.

86. Law, H.; Deng, J. Cornernet: Detecting objects as paired keypoints. Proceedings of the European Conference on Computer Vision (ECCV); Munich, Germany, 8–14 September 2018; pp. 734-750.

87. Lin, T.-Y.; Goyal, P.; Girshick, R.; He, K.; Dollár, P. Focal loss for dense object detection. Proceedings of the IEEE International Conference on Computer Vision; Venice, Italy, 22–29 October 2017; pp. 2980-2988.

88. Girshick, R. Fast r-cnn. Proceedings of the IEEE International Conference on Computer Vision; Santiago, Chile, 7–13 December 2015; pp. 1440-1448.

89. Ren, S.; He, K.; Girshick, R.; Sun, J. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell.; 2016; 39, pp. 1137-1149. [DOI: https://dx.doi.org/10.1109/TPAMI.2016.2577031] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/27295650]

90. Dai, J.; Li, Y.; He, K.; Sun, J. R-fcn: Object detection via region-based fully convolutional networks. Proceedings of the Advances in Neural Information Processing Systems; Barcelona, Spain, 5–10 December 2016; pp. 379-387.

91. Redmon, J.; Farhadi, A. Yolov3: An incremental improvement. arXiv; 2018; arXiv: 1804.02767

92. Won, J.-H.; Lee, D.-H.; Lee, K.-M.; Lin, C.-H. An improved YOLOv3-based neural network for de-identification technology. Proceedings of the 2019 34th International Technical Conference on Circuits/Systems, Computers and Communications (ITC-CSCC); Jeju, Korea, 23–26 June 2019; IEEE: Piscataway, NJ, USA, 2019; pp. 1-2.

93. Zhao, L.; Li, S. Object detection algorithm based on improved YOLOv3. Electronics; 2020; 9, 537. [DOI: https://dx.doi.org/10.3390/electronics9030537]

94. Yin, X.; Ma, T.; Bouferguene, A.; Al-Hussein, M. Automation for sewer pipe assessment: CCTV video interpretation algorithm and sewer pipe video assessment (SPVA) system development. Autom. Constr.; 2021; 125, 103622. [DOI: https://dx.doi.org/10.1016/j.autcon.2021.103622]

95. Murugan, P. Feed forward and backward run in deep convolution neural network. arXiv; 2017; arXiv: 1711.03278

96. Neubeck, A.; Van Gool, L. Efficient non-maximum suppression. Proceedings of the 18th International Conference on Pattern Recognition (ICPR’06); Hong Kong, China, 20–24 August 2006; IEEE: Piscataway, NJ, USA, 2006; pp. 850-855.

97. Wang, M.; Luo, H.; Cheng, J.C. Towards an automated condition assessment framework of underground sewer pipes based on closed-circuit television (CCTV) images. Tunn. Undergr. Space Technol.; 2021; 110, 103840. [DOI: https://dx.doi.org/10.1016/j.tust.2021.103840]

98. Cheng, J.C.; Wang, M. Automated detection of sewer pipe defects in closed-circuit television images using deep learning techniques. Autom. Constr.; 2018; 95, pp. 155-171. [DOI: https://dx.doi.org/10.1016/j.autcon.2018.08.006]

99. Wang, M.; Kumar, S.S.; Cheng, J.C. Automated sewer pipe defect tracking in CCTV videos based on defect detection and metric learning. Autom. Constr.; 2021; 121, 103438. [DOI: https://dx.doi.org/10.1016/j.autcon.2020.103438]

100. Oullette, R.; Browne, M.; Hirasawa, K. Genetic algorithm optimization of a convolutional neural network for autonomous crack detection. Proceedings of the 2004 Congress on Evolutionary Computation (IEEE Cat. No. 04TH8753); Portland, OR, USA, 19–23 June 2004; IEEE: Piscataway, NJ, USA, 2004; pp. 516-521.

101. Halfawy, M.R.; Hengmeechai, J. Automated defect detection in sewer closed circuit television images using histograms of oriented gradients and support vector machine. Autom. Constr.; 2014; 38, pp. 1-13. [DOI: https://dx.doi.org/10.1016/j.autcon.2013.10.012]

102. Wang, M.; Cheng, J.C. Development and improvement of deep learning based automated defect detection for sewer pipe inspection using faster R-CNN. Workshop of the European Group for Intelligent Computing in Engineering; Springer: Berlin/Heidelberg, Germany, 2018; pp. 171-192.

103. Hawari, A.; Alamin, M.; Alkadour, F.; Elmasry, M.; Zayed, T. Automated defect detection tool for closed circuit television (cctv) inspected sewer pipelines. Autom. Constr.; 2018; 89, pp. 99-109. [DOI: https://dx.doi.org/10.1016/j.autcon.2018.01.004]

104. Yin, X.; Chen, Y.; Zhang, Q.; Bouferguene, A.; Zaman, H.; Al-Hussein, M.; Russell, R.; Kurach, L. A neural network-based application for automated defect detection for sewer pipes. Proceedings of the 2019 Canadian Society for Civil Engineering Annual Conference, CSCE 2019; Montreal, QC, Canada, 12–15 June 2019.

105. Kumar, S.S.; Wang, M.; Abraham, D.M.; Jahanshahi, M.R.; Iseley, T.; Cheng, J.C. Deep learning–based automated detection of sewer defects in CCTV videos. J. Comput. Civ. Eng.; 2020; 34, 04019047. [DOI: https://dx.doi.org/10.1061/(ASCE)CP.1943-5487.0000866]

106. Heo, G.; Jeon, J.; Son, B. Crack automatic detection of CCTV video of sewer inspection with low resolution. KSCE J. Civ. Eng.; 2019; 23, pp. 1219-1227. [DOI: https://dx.doi.org/10.1007/s12205-019-0980-7]

107. Piciarelli, C.; Avola, D.; Pannone, D.; Foresti, G.L. A vision-based system for internal pipeline inspection. IEEE Trans. Ind. Inform.; 2018; 15, pp. 3289-3299. [DOI: https://dx.doi.org/10.1109/TII.2018.2873237]

108. Kumar, S.S.; Abraham, D.M. A deep learning based automated structural defect detection system for sewer pipelines. Computing in Civil Engineering 2019: Smart Cities, Sustainability, and Resilience; American Society of Civil Engineers: Reston, VA, USA, 2019; pp. 226-233.

109. Rao, A.S.; Nguyen, T.; Palaniswami, M.; Ngo, T. Vision-based automated crack detection using convolutional neural networks for condition assessment of infrastructure. Struct. Health Monit.; 2021; 20, pp. 2124-2142. [DOI: https://dx.doi.org/10.1177/1475921720965445]

110. Li, D.; Xie, Q.; Yu, Z.; Wu, Q.; Zhou, J.; Wang, J. Sewer pipe defect detection via deep learning with local and global feature fusion. Autom. Constr.; 2021; 129, 103823. [DOI: https://dx.doi.org/10.1016/j.autcon.2021.103823]

111. Dang, L.M.; Wang, H.; Li, Y.; Nguyen, T.N.; Moon, H. DefectTR: End-to-end defect detection for sewage networks using a transformer. Constr. Build. Mater.; 2022; 325, 126584. [DOI: https://dx.doi.org/10.1016/j.conbuildmat.2022.126584]

112. Su, T.-C.; Yang, M.-D.; Wu, T.-C.; Lin, J.-Y. Morphological segmentation based on edge detection for sewer pipe defects on CCTV images. Expert Syst. Appl.; 2011; 38, pp. 13094-13114. [DOI: https://dx.doi.org/10.1016/j.eswa.2011.04.116]

113. Su, T.-C.; Yang, M.-D. Application of morphological segmentation to leaking defect detection in sewer pipelines. Sensors; 2014; 14, pp. 8686-8704. [DOI: https://dx.doi.org/10.3390/s140508686] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/24841247]

114. Wang, M.; Cheng, J. Semantic segmentation of sewer pipe defects using deep dilated convolutional neural network. Proceedings of the International Symposium on Automation and Robotics in Construction; Banff, AB, Canada, 21–24 May 2019; IAARC Publications 2019; pp. 586-594. Available online: https://www.proquest.com/openview/9641dc9ed4c7e5712f417dcb2b380f20/1 (accessed on 20 February 2022).

115. Zhang, L.; Wang, L.; Zhang, X.; Shen, P.; Bennamoun, M.; Zhu, G.; Shah, S.A.A.; Song, J. Semantic scene completion with dense CRF from a single depth image. Neurocomputing; 2018; 318, pp. 182-195. [DOI: https://dx.doi.org/10.1016/j.neucom.2018.08.052]

116. Wang, M.; Cheng, J.C. A unified convolutional neural network integrated with conditional random field for pipe defect segmentation. Comput.-Aided Civ. Infrastruct. Eng.; 2020; 35, pp. 162-177. [DOI: https://dx.doi.org/10.1111/mice.12481]

117. Long, J.; Shelhamer, E.; Darrell, T. Fully convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; Boston, MA, USA, 7–12 June 2015; pp. 3431-3440.

118. Fayyaz, M.; Saffar, M.H.; Sabokrou, M.; Fathy, M.; Klette, R.; Huang, F. STFCN: Spatio-temporal FCN for semantic video segmentation. arXiv; 2016; arXiv: 1608.05971

119. Bi, L.; Feng, D.; Kim, J. Dual-path adversarial learning for fully convolutional network (FCN)-based medical image segmentation. Vis. Comput.; 2018; 34, pp. 1043-1052. [DOI: https://dx.doi.org/10.1007/s00371-018-1519-5]

120. Lu, Y.; Chen, Y.; Zhao, D.; Chen, J. Graph-FCN for image semantic segmentation. Proceedings of the International Symposium on Neural Networks; Moscow, Russia, 10–12 July 2019; Springer: Berlin/Heidelberg, Germany, 2019; pp. 97-105.

121. Ronneberger, O.; Fischer, P.; Brox, T. U-net: Convolutional networks for biomedical image segmentation. Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention; Munich, Germany, 5–9 October 2015; Springer: Berlin/Heidelberg, Germany, 2015; pp. 234-241.

122. Pan, G.; Zheng, Y.; Guo, S.; Lv, Y. Automatic sewer pipe defect semantic segmentation based on improved U-Net. Autom. Constr.; 2020; 119, 103383. [DOI: https://dx.doi.org/10.1016/j.autcon.2020.103383]

123. Iyer, S.; Sinha, S.K. A robust approach for automatic detection and segmentation of cracks in underground pipeline images. Image Vis. Comput.; 2005; 23, pp. 921-933. [DOI: https://dx.doi.org/10.1016/j.imavis.2005.05.017]

124. Li, Y.; Wang, H.; Dang, L.M.; Piran, M.J.; Moon, H. A Robust Instance Segmentation Framework for Underground Sewer Defect Detection. Measurement; 2022; 190, 110727. [DOI: https://dx.doi.org/10.1016/j.measurement.2022.110727]

125. Haurum, J.B.; Moeslund, T.B. Sewer-ML: A Multi-Label Sewer Defect Classification Dataset and Benchmark. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; Nashville, TN, USA, 19–25 June 2021; pp. 13456-13467.

126. Mashford, J.; Rahilly, M.; Davis, P.; Burn, S. A morphological approach to pipe image interpretation based on segmentation by support vector machine. Autom. Constr.; 2010; 19, pp. 875-883. [DOI: https://dx.doi.org/10.1016/j.autcon.2010.06.001]

127. Iyer, S.; Sinha, S.K.; Tittmann, B.R.; Pedrick, M.K. Ultrasonic signal processing methods for detection of defects in concrete pipes. Autom. Constr.; 2012; 22, pp. 135-148. [DOI: https://dx.doi.org/10.1016/j.autcon.2011.06.012]

128. Lei, Y.; Zheng, Z.-P. Review of physical based monitoring techniques for condition assessment of corrosion in reinforced concrete. Math. Probl. Eng.; 2013; 2013, 953930. [DOI: https://dx.doi.org/10.1155/2013/953930]

129. Makar, J. Diagnostic techniques for sewer systems. J. Infrastruct. Syst.; 1999; 5, pp. 69-78. [DOI: https://dx.doi.org/10.1061/(ASCE)1076-0342(1999)5:2(69)]

130. Myrans, J.; Everson, R.; Kapelan, Z. Automated detection of fault types in CCTV sewer surveys. J. Hydroinform.; 2019; 21, pp. 153-163. [DOI: https://dx.doi.org/10.2166/hydro.2018.073]

131. Guo, W.; Soibelman, L.; Garrett, J.H. Imagery enhancement and interpretation for remote visual inspection of aging civil infrastructure. Tsinghua Sci. Technol.; 2008; 13, pp. 375-380. [DOI: https://dx.doi.org/10.1016/S1007-0214(08)70177-9]

Word count: 11944

Show less

© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.

Abstract

Due to the advantages of economics, safety, and efficiency, vision-based analysis techniques have recently gained conspicuous advancements, enabling them to be extensively applied for autonomous constructions. Although numerous studies regarding the defect inspection and condition assessment in underground sewer pipelines have presently emerged, we still lack a thorough and comprehensive survey of the latest developments. This survey presents a systematical taxonomy of diverse sewer inspection algorithms, which are sorted into three categories that include defect classification, defect detection, and defect segmentation. After reviewing the related sewer defect inspection studies for the past 22 years, the main research trends are organized and discussed in detail according to the proposed technical taxonomy. In addition, different datasets and the evaluation metrics used in the cited literature are described and explained. Furthermore, the performances of the state-of-the-art methods are reported from the aspects of processing accuracy and speed.

Details

Title
Vision-Based Defect Inspection and Condition Assessment for Sewer Pipes: A Comprehensive Survey
Author
Li, Yanfen 1 ; Wang, Hanxiang 1 ; Dang, L Minh 2 ; Song, Hyoung-Kyu 2   VIAFID ORCID Logo  ; Moon, Hyeonjoon 1   VIAFID ORCID Logo 

1  Department of Computer Science and Engineering, Sejong University, Seoul 05006, Korea; [email protected] (Y.L.); [email protected] (H.W.) 
2  Department of Information and Communication Engineering and Convergence Engineering for Intelligent Drone, Sejong University, Seoul 05006, Korea; [email protected] (L.M.D.); [email protected] (H.-K.S.) 
First page
2722
Publication year
2022
Publication date
2022
Publisher
MDPI AG
e-ISSN
14248220
Source type
Scholarly Journal
Language of publication
English
DOI
https://doi.org/10.3390/s22072722
ProQuest document ID
2649064815
Copyright
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.