Bolt Loosening and Preload Loss Detection Technology Based on Machine Vision

Abstract

Steel bridges often experience bolt loosening and even fatigue fracture due to fatigue load, forced vibration, and other factors during operation, affecting structural safety. This study proposes a high-precision bolt key point positioning and recognition method based on deep learning to address the high cost, low efficiency, and poor safety of current bolt loosening identification methods. Additionally, a bolt loosening angle recognition method is proposed based on digital image processing technology. Using image recognition data, the angle-preload curve is revised. The established correlation between loosening angle and pretension for commonly utilized high-strength bolts provides a benchmark for identifying loosening angles. This finding lays a theoretical foundation for defining effective detection intervals in future bolt loosening recognition systems. Consequently, it enhances the system’s ability to deliver timely warnings, facilitating swift manual inspections and repairs.

1. Introduction

High-strength bolt connections are a crucial means of joining elements in steel structure bridges [1]. These bolts have rapidly replaced traditional ordinary bolts in steel structures due to their high load-bearing capacity, excellent fatigue resistance, ease of fabrication and installation, and convenience for dismantling and maintenance [2]. In steel structure bridges, high-strength bolts serve various roles, including anchoring attachment structures, connecting trusses, and linking concrete slabs to steel beams. However, during service, high-strength bolts may experience a reduction in preload due to the effects of fatigue loads, vibration impacts, forced vibrations, and harsh environmental conditions. This reduction can lead to bolt loosening or even fracture, posing significant safety risks to the bridge’s structural integrity [3]. Bolt failure can result in disastrous consequences and significant economic losses [4,5,6,7]. According to a survey by a highway company, of the steel bridges with identified issues, approximately one-third of the problems stem from bolt damage, with loosening being the primary cause. From 2010 to 2020, several bridge collapses occurred due to issues such as bolt loosening, falling off, and untimely maintenance, causing substantial economic and human losses [8]. Consequently, bolt loosening and preload loss in steel structures are common issues in bridge monitoring [9].

Currently, traditional bolt loosening detection methods involve maintenance personnel conducting regular inspections of bridges. Inspectors primarily assess bolt conditions using visual inspection, torque wrench methods, scribing methods, magnetic tape application, and small hammer tapping [10]. Using a torque wrench for visual inspection is the most common method for detecting bolt loosening, as shown in Figure 1. The torque wrench measures the torque of a bolt in a fastened state and estimates the preload to evaluate the bolt’s connection quality. However, when using a torque wrench, it is difficult to accurately determine the friction loss between the nut and bolt, leading to potential errors in value acquisition. Furthermore, employing torque wrenches is labor-intensive, making it challenging to meet the current large-scale bolt inspection demands. As shown in Figure 2, the scribed line method involves drawing a straight line at the junction of the nut and bolt shank after fastening. Inspectors visually check for line misalignment to determine the degree of bolt looseness. The magnetic strip attachment method employs magnets and rings; when looseness reaches a certain level, the magnets detach due to insufficient magnetic force, and the detachment location indicates the structural looseness. Although these two methods are simple to execute, they present significant limitations when applied to complex structures or confined environments and are unsuitable for cases where the bolt shank has axial deformation [11]. The small hammer tapping method is a more traditional approach. Inspectors touch the bolt with their fingers and tap the bolt head with a hammer; abnormal vibrations felt in the fingers indicate bolt looseness. This method is relatively simple but relies on the inspector’s experience and touch sensitivity, and it suffers from low detection efficiency. From the methods discussed above, traditional detection approaches have several drawbacks: First, some bolt locations are difficult for inspectors to access, compromising their safety. Second, determining bolt looseness largely depends on the personal experience of maintenance staff, yielding low reliability. Third, manual inspection incurs high costs [12].

Due to the limitations of manual inspections and with the continuous advancement of information and electronic technologies, some researchers have proposed using contact sensors for bolt looseness detection. These methods mainly include impedance-based, modal-based, and guided wave-based approaches [13]. In the impedance-based method, piezoelectric patches are adhered to the bolt joints, and high-frequency voltage is applied to excite the patches [14,15]. This method captures impedance characteristics reflecting local dynamics to assess the degree of bolt looseness. Changes in bolt looseness affect internal modal parameters, and modal-based detection methods utilize these changes. The structural modal parameters when the bolt is fastened serve as baseline parameters. By comparing these with the test results, the bolt connection state can be assessed [16,17]. Since post-loosening vibrations are nonlinear, assuming linear elasticity for mode identification introduces limitations. In guided wave-based detection, two piezoelectric patches are adhered to different sides of a bolt joint: one generates acoustic waves and the other collects the waves produced. Changes in bolt torque affect wave propagation; thus, analyzing wave changes can diagnose bolt looseness [18,19]. Theoretically, guided wave methods can monitor all bolts at structural joints and are better suited to complex bridge structures. From the methods discussed, it is evident that contact sensor-based approaches have their respective advantages in identifying bolt looseness and are continually developing and improving. However, most sensor-based methods require high-precision instruments, costly measurement channels, and algorithms to compensate for environmental effects. This makes implementation challenging for bridge structures with a large number of bolts.

When a bolt loosens, it rotates to a certain angle, resulting in a deviation from its initial state. The machine vision-based method captures images of the bolt in both fastened and loosened states using a camera. These images are analyzed using image processing and machine vision techniques to assess the bolt’s connection status [10]. Compared to the aforementioned detection methods, the vision-based approach offers several advantages: (1) Higher measurement accuracy and efficiency, adaptable to various complex engineering environments; (2) minimal impact from environmental factors such as temperature and humidity; (3) the ability to visually detect many structural damages more intuitively. Park et al. [20] proposed a bolt looseness detection method based on image processing. This method involves segmenting nuts and splice plates using image processing techniques and identifying the rotation angle of the nut using Hough transform and Canny edge detection. By comparing the angle before and after bolt loosening, bolt looseness can be detected. Zhao et al. [21] used a single-shot multiBox detector (SSD) to identify bolts, with a minimum recognition angle of ${10}^{\circ }$. Kong and Li [22] performed geometric feature matching and gray value matching on images from initial and vibration states, respectively. However, this method is cumbersome and requires stringent camera positioning at both capturing instances. Huynh et al. [23] proposed a bolt looseness detection algorithm based on region-based convolutional neural networks (RCNN), achieving an accuracy of 93% and validating the accuracy on real bridges. However, the algorithm performed poorly on painted bolt groups, showing a lack of generalization. Pham et al. [24] introduced a method using synthetic image software to generate datasets for training models, leveraging virtual software to synthesize bolt samples under various environmental conditions, thereby reducing time and cost in collecting high-quality sample sets and accelerating the application of synthetic datasets in bolt looseness detection. Wu [25] proposed a method similar to Kong’s, using image registration techniques and feature point matching algorithms to determine angle values based on the inverse of the transformation matrix, though the accuracy was inferior to edge extraction methods. Huynh [26] further improved their research, introducing a Faster RCNN-based bolt looseness detection algorithm, enhancing detection accuracy to 98.85%. This method also showed high adaptability to low-light images, suggesting significant potential for field applications. Yang et al. [27] proposed a two-stage detection framework combining traditional manual torque methods with deep learning models, using virtual software to synthesize images for training deep models and assessing fastening status by analyzing changes in bolt lines. Tests showed that the proposed method achieved a comprehensive advantage in detection accuracy and speed, indicating a promising practical application prospect. Cha et al. [28] created a dataset using the horizontal and vertical lengths of bolt heads as features to determine bolt connection status based on the protruding length of the bolt shank. Ramana et al. [29] combined the Viola-Jones algorithm with support vector machines, presenting a vision-based automatic detection method with recognition accuracy reaching 91%, although it requires manual feature design. Wang et al. [30] suggested a method using image processing technology to calculate the horizontal and vertical lengths of bolt heads, needing extensive manual bolt labeling, resulting in low efficiency for mass bolt detection. Zhang et al. [31] developed a deep learning algorithm based on Faster RCNN, defining bolt looseness by the exposed length of the shank, with a recognition accuracy of 95.03%, capable of identifying looseness in bolts with an exposed length of just 0.5 cm. However, this method performs poorly in detecting bolts with varying exposure lengths on a large scale. Gong et al. [32] combined deep learning with geometric imaging theory, proposing a method to detect bolt looseness. Utilizing Faster RCNN to locate exposed bolts and Cascaded Pyramid Network to detect key points of exposed bolts, the average detection error was 0.61 mm, yet the method was ineffective in detecting slight bolt loosening.

The above indicates that prior research has demonstrated the feasibility of combining deep learning and machine vision for bolt looseness detection. With the advancement of computer hardware and deep learning theories, digital image processing-based damage detection technologies have overcome the drawbacks of manual inspection, such as being time-consuming and lacking precision. Utilizing deep-learning-based image processing techniques to replace the human eye in identifying image features offers advantages of high reliability, low cost, and rapid speed. However, currently transforming vision-based methods into practical health monitoring for bridge bolt connections faces many challenges. As shown in Figure 3, current methods are mainly divided into two types: one approach is to detect the bolt angle by recognizing the six edges of the bolt in the image, but achieving high precision with this method is challenging; the other approach detects the exposed length of the bolt shank, which requires high-quality images and is less practical. Both methods show significant deficiencies in terms of detection accuracy, speed, robustness, and generalization, making them unable to meet the large-scale detection demands required in engineering applications.

This paper applies an integrated approach using the theories of deep learning and digital image processing, along with advanced technologies like computer vision, to study rapid identification methods for detecting loosening of bolts in steel structure bridges under complex environmental conditions. The proposed method in this paper is distinctly different from the aforementioned methods: it first identifies the six corner points in the image and then uses mathematical methods to calculate the relative angle changes of certain corner points in the image under two different states to determine the fastening status of the bolt. Comparatively, this model achieves higher accuracy in key point recognition. Figure 4 illustrates the research framework of this paper. The process begins with the deep-learning-based bolt key point recognition, which includes three main steps: (1) Create a training graphic database; (2) Train the deep learning model for bolt detection; (3) Test the performance of the deep learning network model. Once the bolt key points have been obtained, bolt loosening angle calculation based on image processing is conducted, comprising five main steps: (1) Classification and numbering of bolt key points; (2) Perspective correction; (3) Calculation of bolt loosening angle; (4) Laboratory environment testing; (5) Field environment testing. When combined with the current demands for large-scale bridge inspection, this approach addresses the challenges of monitoring linear, dense, towering bolts and critical node bolts, significantly enhancing the efficiency of detecting bolt looseness in bridges. This has crucial engineering significance in promoting the informatization, intelligence, and standardization of bridge bolt maintenance.

2. Key Point Identification of Bolts Based on Deep Learning

Bolt key point recognition based on deep learning mainly consists of three steps: First, a large number of bolt images are collected and preprocessed to form a training set. Next, the training set is input into the deep learning model for model training. Finally, newly acquired bolt images that were not part of the training set are input into the deep learning model to output the coordinates of key points in the images.

2.1. Dataset Construction

2.1.1. Annotation of True Positions of Bolt Key Points

All original samples in this study’s dataset were derived from several steel truss bridges in Zhejiang Province, China. A large number of bolt images were obtained using equipment such as DSLR cameras, smartphones, industrial cameras, and drones from on-site photography, as well as numerous samples under different environments taken in laboratory settings. Low-quality images due to blurring, excessive shooting angles, and occlusions were discarded, resulting in a final set of approximately 2000 image samples. Prior to model training, the collected raw images were preprocessed to extract the regions of interest (ROI). In addition, to enhance the training speed, the image sizes were normalized, with all images uniformly cropped and scaled to $1024\times 768$ pixels, stored in BMP format. The annotation software Labelme was employed to mark the true positions of corner and center points in the cropped sample images. Different labels were assigned to center and corner points, and the annotation information was converted into XML format to create the sample set. Figure 5 shows an example of key point annotation using the software.

2.1.2. Data Augmentation

A dataset consisting of 2000 original images is insufficient for training a deep learning model effectively. To enhance the size of the training dataset for deep learning models, ensure the complexity and diversity of bolt samples during training, and prevent overfitting, data augmentation techniques are employed to expand the existing dataset [33]. Based on the 2000 labeled bolt images, data augmentation methods referenced in [34] are utilized to expand the dataset. The data augmentation techniques applied to the bolt samples include five main methods: image flipping, translation, arbitrary contrast adjustment, blur processing, and random image erasure. Images captured in actual engineering applications often contain various types of noise, which are random and independent pixels. To increase the uncertainty of the samples, random Gaussian noise is introduced into the processed bolt images [35]. After data augmentation, the number of sample images increased by a factor of 6.8, expanding from 2000 to 13,647 images.

2.2. Bolt Keypoint Localization Method

2.2.1. Algorithm Principle

The bolt keypoint localization network is used to detect the precise location of bolt keypoints in an image, providing accurate positional information for subsequent angle calculations. In this paper, a method based on heatmaps is used instead of the traditional method of directly regressing keypoint coordinates to predict the positions of bolt keypoints. This method is currently widely applied in the fields of facial landmark detection [36,37], crowd counting [38], and pothole detection [39]. After evaluating the comprehensive performance of various networks, the bolt keypoint localization network was designed by referring to a network structure originally used for the Simple Baseline, making improvements on that basis. Compared to the Hourglass and CPMs models, which are currently the main heatmap-based coordinate regression models, the structure of this keypoint localization network is relatively simple [37]. The entire structure primarily consists of two subnetworks: a feature extraction module and an upsampling module, lacking repetitive modules, which effectively avoids code redundancy and complexity. The feature extraction module is primarily used for extracting features from the input bolt images and can be based on architectures like VGG, ResNet, and YOLO. Compared to shallow networks like VGG, ResNet50 is able to build a deeper network structure, making it more suitable for extracting complex image features. YOLO, on the other hand, does not perform as finely in fine-grained feature extraction as ResNet50. Therefore, the feature extraction module is based on the ResNet50 model [40]. The upsampling module is responsible for enlarging the feature maps, i.e., restoring the feature image dimensions. The upsampling module includes several deconvolution layers, with one additional layer added following Reference [38], resulting in a total of four deconvolutional layers. Each deconvolution layer comprises 256 filters with a kernel size of $4\times 4$, padding of 1, and a stride of 2.

The process flow of the bolt keypoint localization algorithm is shown in Figure 6. The image resolution for input to the model is set to $1024\times 768$. After four extraction stages by the feature extraction module, the final output feature map from the C5 module of ResNet-50 is downsized by a factor of 64 relative to the input. Subsequently, four deconvolution layers [41], each equipped with batch normalization and ReLU activation functions [42], are employed to upsample the image, restoring the low-resolution feature map obtained from the feature extraction module back to a high-resolution feature map. Finally, a convolutional layer is added at the end of the network to generate k bolt corner point prediction heatmaps and a prediction heatmap for the central point. The specific parameters of the model structure are detailed in Table 1. Due to the nature of heatmap regression-based bolt keypoint detection, the network outputs heatmaps, necessitating post-processing of the network output to convert the heatmap into numerical coordinates of the bolt keypoints. The method employed herein is the use of the argmax function, assuming the location with the highest probability in the heatmap as the predicted bolt keypoint location by the network. The two-dimensional coordinates of all keypoints with maximum probability are output as follows:

(1)${\mathrm{pos}}_{(x,y)}={\mathrm{argmax}}_{\mathrm{p}}{G}_{\mathrm{k}}\left(p\right)$

where ${\mathrm{pos}}_{(x,y)}$ denotes the coordinates at the location of the maximum value for the $p\mathrm{th}$ keypoint and $G_{\mathrm{k}}\left(p\right)$ represents the $p\mathrm{th}$ keypoint in the heatmap. It should be noted that if the model does not detect a keypoint, the original image is directly output without further processing.

2.2.2. Model Training Process

The entire model is trained to regress the positions of bolt keypoints. This paper employs a convolutional neural network-based heatmap regression approach to predict bolt keypoint locations. Given the relatively regular distribution of bolt keypoints in images, a two-dimensional Gaussian distribution can be utilized to generate the true heatmap of the keypoints. The probability density function for a two-dimensional Gaussian distribution is computed as follows:

(2)$f(x,y)={\displaystyle \frac{1}{2\pi {\sigma }^2}}exp\left({\displaystyle \frac{{(x-{\mu }_1)}^2+{(y-{\mu }_2)}^2}{2{\sigma }^2}}\right)$

where $f(x,y)$ is the Gaussian value, $(x,y)$ are the coordinates in the two-dimensional Gaussian distribution, ${\mu }_1$ and ${\mu }_2$ are the means, and $\sigma$ is the standard deviation.

By applying the two-dimensional Gaussian distribution function, a Gaussian curve can be plotted and projected onto a two-dimensional plane, where each pixel value represents a probability, forming the heatmap. All the bolt samples collected in this study require two Gaussian heatmaps to be generated: a corner heatmap and a center heatmap, as illustrated in Figure 7. Define the input image to the model as L, and let T be the set of coordinates for all keypoints’ centers in the image. The ground truth heatmap $N(t,T,{\sigma }^{2}L)$ is then defined as:

(3)$D_L\left(t\right)=\bigcup _{L}N(t;T;{\sigma }^{2}L)$

where ⋃ represents the union operation of pixels in space corresponding to the maximum value and $N(t;T;{\sigma }^{2}L)$ denotes the two-dimensional Gaussian distribution centered at the keypoint.

During the model training process, a supervised learning approach is adopted, utilizing pre-initialization with the ImageNet dataset to accelerate training speed and enhance training performance. Following the methodology described in Reference [43], data augmentation techniques such as random rotation ($-{30}^{\circ }$, ${30}^{\circ }$), random translation (−40, 40), and random scaling (0.75, 1.5) are applied to crop input images during training. The mean squared error (MSE) is chosen as the loss function, defined as:

(4)$G{\left(W_{\mathrm{a}}\right)}_{\mathrm{des}}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\left(G_{\mathrm{a}}(L_i;W_{\mathrm{a}})-D_i\left(t\right)\right)}^2$

where $G_{\mathrm{a}}(L_i;W_{\mathrm{a}})$ is the predicted heatmap from the keypoint localization network, $W_{\mathrm{a}}$ represents the set of training parameters, $L_i$ is the input image, N is the total number of training samples, and $D_i\left(t\right)$ corresponds to the actual heatmap distribution for the keypoints.

To achieve a model with better accuracy and generalization capabilities, it is necessary to monitor the loss on both the training and validation sets, visualizing the model’s loss function. The loss curves for the training and validation sets during the training process are shown in Figure 8. As depicted in Figure 8, the loss function decreases most rapidly during the first 20 epochs and begins to converge at around 40 epochs. With further iterations, the loss values for both the training and validation sets gradually decrease and stabilize around 140 epochs. The loss value for the training set reduces to 0.0018 at the last iteration, indicating that the model has fully converged, and its recognition performance has also stabilized. Therefore, under the stipulated training epochs, the developed model demonstrates effective capability in recognizing bolt keypoints.

In the detection of bolt keypoint localization, the percentage of correct keypoints (PCK) [44] and accuracy (ACC) are selected as evaluation metrics to assess the model’s precision and capability in recognizing keypoints. Additionally, the detection time t for a single image is used to evaluate the model’s recognition speed. PCK and ACC are calculated as follows:

(5)$\mathrm{PCK}={\displaystyle \frac{{\sum }_i\delta \left(\frac{d_i}{d_i^{\mathrm{def}}}<T_{\mathrm{k}}\right)\delta (v_i>0)}{{\sum }_i\delta (v_i>0)}}$

(6)$\mathrm{ACC}={\displaystyle \frac{S_{\mathrm{correct}}}{S_{\mathrm{total}}}}$

where i is the index of keypoints; $\delta$ is the Kronecker function; $d_i$ denotes the Euclidean distance between the i-th keypoint detected by the model and the true i-th keypoint; $d_i^{\mathrm{def}}$ is the length of the input image; $T_{\mathrm{k}}$ is the threshold set to 0.005 in this study; $v_i$ is the visibility of the i-th keypoint; $S_{\mathrm{correct}}$ represents the number of bolts for which all 7 keypoints are successfully detected; and $S_{\mathrm{total}}$ is the total number of bolts.

2.3. Model Performance Testing

In order to comprehensively evaluate the model’s performance, two experiments were designed: the keypoint localization test and the robustness test under different lighting conditions.

2.3.1. Keypoint Localization Testing

A total of 100 images of bolts were collected both in the laboratory and on-site. The newly collected images were not involved in the model’s training or validation and primarily include variations in coatings, different lighting conditions, and both front and back views of the bolts. The keypoint detection model was utilized to test the newly acquired images. To visually indicate the positions of the keypoints, the identified images were superimposed over the original images. The detection results for keypoints in the bolt section are shown in Figure 9. Each image from left to right displays the original image of the bolt, the corner point mapping, and the center point mapping. As seen in Figure 9, the algorithm performs well, accurately correlating both the center and corner points with their actual positions.

To further evaluate the model’s localization capabilities, manual annotations of the keypoints were made for comparison with the model’s detected positions. The coordinates of the manually annotated keypoints were extracted and organized in Table 2. It can be observed that the model’s detected positions show only minor differences from the true positions in terms of pixel values. The pixel distances between marked points and detected points were calculated and summarized in Table 3. From the table, it is evident that the distance for corner points ranges between 0.85 pixels and 1.89 pixels, indicating a small margin of error that can be virtually overlooked. However, the distances for the center points were larger, ranging from 4.59 pixels to 6.28 pixels. Overall, the model demonstrates superior localization capability for corner points as compared to center points, likely due to the more distinctive features associated with corner points, while the features of center points are less pronounced, leading to some discrepancies in the training set annotations.

2.3.2. Light Comparison Test

Lighting is one of the critical factors affecting the robustness of models. To investigate the impact of illumination changes on localization performance, this section sets up four different lighting intensities in a laboratory setting. For each condition, 20 images were captured by slightly moving within a small range at a distance of 1.5 m using an industrial camera. The model’s performance in locating keypoints is evaluated using two metrics: PCK and ACC. The recognition results are illustrated in Figure 10, where it can be observed that the model successfully identifies all corners. However, under strong lighting conditions, fluctuations in the keypoint map are noted, as shown in Figure 10a.

To further assess the model’s performance accurately, the images were manually annotated, and a comparison was made with the model’s predictions. The results of corner detection are summarized in Table 4. From Table 4, it is evident that the performance of the keypoint localization algorithm is significantly impacted by variations in lighting. Under normal lighting conditions, the two metrics reached their highest values of 97.5% and 95.6%, while in strong lighting situations, the performance decreased to 96.2% and 93.4%. Nonetheless, the indicators remain within an applicable range.

3. Bolt Loosening Angle Calculation Based on Digital Image Processing

After the collected images are processed through bolt key point recognition based on deep learning, the key point coordinates of the bolt can be obtained. Subsequently, through digital image processing, the bolt’s key points are classified and numbered, the image is perspective-corrected, and the bolt’s loosening angle is calculated using a formula.

3.1. Bolt Key Point Classification

The detection of the bolt loosening angle involves the identification of characteristic images of bolts that have been rotated by a certain angle. The analysis only requires consideration of the changes in specific common features of the bolt within a fixed coordinate system. In the keypoint localization model, two types of keypoints—centers and corners—are outputted, where the center point primarily provides the localization information of the bolt and is not used for calculation.

To group the seven points of each bolt, a center-corner clustering algorithm [45] is employed. The center-corner algorithm transforms the clustering problem of a dataset into a clustering problem of two datasets, using the position of the center point to locate the corner points. The distances between all center points and all corner points are calculated sequentially. After computing the distances from all corner points to the center points, the six corner points with the smallest distance values are selected to form a group corresponding to each bolt.

However, in the actual captured images of bolts, situations may arise where only half of a bolt is visible, as shown in Figure 11. Therefore, to avoid incorrect grouping of corner points after distance calculations, an outlier handling procedure is introduced: if a group of bolt corner points contains an outlier, the bolt group corresponding to that point is discarded. The threshold for outlier detection is set as 0.8 times the sum of the distances of two adjacent corner points to the center point, which can be calculated using the following formula:

(7)$t=(d_{(n+1)i}+d_{(n+2)i})\times 0.8$

with the outlier judgment determined as follows:

(8)$d_{ni}\ge t$

where t is the threshold for outliers, and if $d_{ni}\ge t$, then the n-th point is classified as an outlier in the set of corner points, where n takes on values from 1 to 6.

3.2. Calculation Principle of Initial Angle of the Bolt

Figure 12 illustrates the principle behind the calculation of the initial angle of the bolt. The point closest to the X-axis in the first quadrant is designated as the first corner point of the bolt. A vector is formed between the center point of the bolt and the first corner point, which then creates an angle with the positive direction of the X-axis. This angle is defined as the initial angle of the bolt, denoted by $\theta$, and can be calculated using the following formula:

(9)$\theta =arctan\left({\displaystyle \frac{y_2-y_1}{x_2-x_1}}\right)$

where $\theta$ is the initial angle of the bolt, and $(x_1,y_1)$ are the coordinates of the bolt’s center point, $(x_2,y_2)$ are the coordinates of the corner point in the first quadrant.

To improve the accuracy of the angle, the angles ${\theta }_1$ and ${\theta }_2$ formed by corner points 2 and 3 with the X-axis are also calculated. After subtracting ${180}^{\circ }$ from these three values, their average is computed to determine the final initial value for each bolt.

After calculating the initial angle for each bolt, the bolts in the image are numbered in a specific sequence. Figure 13 illustrates the principle of bolt position numbering. Using the corrected center point of each bolt as a reference, the 2D coordinates of all center points are mapped to a 1D coordinate system. By comparing the sizes of these 1D coordinates, the bolts are numbered from the top left to the bottom right.

3.3. Image Perspective Correction

The captured images may exhibit certain perspective distortion. It is necessary to correct the images by projecting points from the image plane to the world plane. As shown in Figure 14, this process is accomplished through a linear transformation defined by a non-singular $3\times 3$ matrix. Given a homography matrix M, a point $p_j$ on the image plane can be linearly transformed into $q_j$ in homogenous coordinates, as described by the following equation:

(10)$p_j=M{q}_j$

where M is defined as:

(11)$M=\left[\begin{array}{ccc}{t}_{11}& t_{12}& t_{13}\\ t_{21}& t_{22}& t_{23}\\ t_{31}& t_{32}& 1\end{array}\right]$

To perform the projection transformation on the image, it is necessary to determine the homography matrix M. Based on the coordinates of four pairs of corresponding points from the distorted image $p_j$ and the reference image $q_j$, the homography matrix M can be calculated. Once the homography matrix M is obtained, all points in the distorted image can be transformed into the corrected image using the inverse of the calculated homography matrix.

3.4. Calculation of Loosening Angle

In a practical operational environment, when a bolt is tightened, the initial angle is recorded immediately. Subsequently, the angle after the bolt loosens is also recorded. The loosening angle is calculated by comparing the angle changes between the two images. The principle behind calculating the bolt loosening angle is illustrated in Figure 15. The loosening angle of the bolt can be calculated using the following formula:

(12)$\Delta \theta =arctan\left({\displaystyle \frac{y_3-y_1}{x_3-x_1}}\right)-arctan\left({\displaystyle \frac{y_2-y_1}{x_2-x_1}}\right)$

where $\Delta \theta$ is the bolt loosening angle, $(x_1,y_1)$ are the coordinates of the bolt’s center point, $(x_2,y_2)$ are the coordinates of the corner point before rotation, and $(x_3,y_3)$ are the coordinates of the corner point after rotation.

The absolute value of the loosening angle is compared to a defined threshold $\mathrm{UCL}$. If it exceeds this threshold, the bolt is classified as loose, as expressed in the following equation:

(13)$\mathrm{UCL}=\mu +3\sigma \mathrm{and}|\Delta \theta |>\mathrm{UCL}$

where $\mu$ and $\sigma$ are the mean and standard deviation of the bolt loosening angles, respectively. $\mathrm{UCL}$ is the threshold defined by three standard deviations $\sigma$ from the mean $\mu$, which corresponds to a 99.7% confidence level.

It is important to note that due to the inherent nature of hexagonal bolts, if the rotation angle exceeds ${60}^{\circ }$ or its integer multiples, the estimation of bolt rotation may become inaccurate. However, in practical engineering situations, bolt loosening is a gradual process that takes a considerable amount of time to occur. Abrupt changes of more than ${60}^{\circ }$ certainly cannot happen in a short period; in fact, bolts tend to fail even after loosening by around ${10}^{\circ }$. Therefore, the bolt loosening detection method employed in this study remains effective, and utilizing real-time video monitoring based on visual inspection for bolt connections can also mitigate this issue.

3.5. Experimental Validation in Laboratory Environment

3.5.1. Cluster Algorithm Validation Experiment

This paper utilizes the centroid-corner algorithm to identify the loosening angles of bolts. Before the experiment, 200 representative bolt images (including various factors such as different numbers of bolts, perspectives, lighting conditions, etc.) were preprocessed using a keypoint localization model. The keypoint images of the bolts were then binarized to serve as the original images for testing the algorithm, establishing a bolt clustering test set. Figure 16 shows three processed binary images of bolt keypoints, where the white points represent the bolt keypoints after binarization. After clustering, to more intuitively present the clustering results of the bolts, an envelope algorithm was employed to group the clustered bolt keypoints, which were then merged with the original images to facilitate an intuitive observation of the bolt clustering. Figure 17 illustrates experimental results of the clustering effect.

This paper evaluates the performance of the bolt clustering method using accuracy ($\mathrm{ACC}$) and recall rate (R) as metrics:

(14)$R={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}}$

where $\mathrm{TP}$ is the number of correctly clustered bolts, and $\mathrm{FN}$ is the number of bolts incorrectly recognized as correct. The calculated accuracy ($\mathrm{ACC}$) of the centroid-corner algorithm is $0.94$, and the recall rate (R) is $0.87$. The computer is an HP desktop workstation equipped with a 40-core Intel Xeon Silver 4210R processor, 64GB of RAM, a 1TB solid-state drive, and two Nvidia GeForce GTX 3090 dedicated graphics cards, with a main frequency of 4.0 GHz. We conducted processing tests on all the collected bolt sample images to evaluate the average time consumption of the center-point-to-corner-point detection algorithm. The experimental results show that the average processing time per image is 0.54 s. Based on the hardware configuration and processing speed analysis, the system can meet the real-time processing requirements of large-scale monitoring tasks.

3.5.2. Angle Accuracy Test

To verify the accuracy of the angle detection algorithm, images of both the front and back of the bolt connection plate were taken from a distance of 1.5 m using a smartphone, with horizontal and perspective angles both less than $5^{\circ }$. The image contains 65 bolts, and the algorithm developed in this paper was utilized to identify the current angles. Figure 18 shows the results of angle identification, from which it can be seen that the angle of each bolt is identified. A comparison with manually measured values is presented in Figure 19. The figure indicates a high correlation between the algorithm’s identified values and the manually measured values, with a correlation coefficient of 0.998. An analysis of the errors between the identified and measured values is presented in Table 5, showing a maximum error of $2^{\circ }$, an average error of $0.{87}^{\circ }$, and a minimum error of $0^{\circ }$. The error results indicate a high accuracy of the algorithm in recognizing angles, with the identified values closely matching the measured values.

3.5.3. Loosening Angle Test

To evaluate the feasibility of the proposed method for detecting loose bolts, an experimental verification was conducted using a connection plate with 18 bolts in a laboratory setting, as shown in Figure 20. The test images were captured from a distance of 1.0 to 1.2 m, maintaining horizontal and vertical angles at $0^{\circ }$. An image was taken with the bolts fully tightened as a control, then bolts 1, 3, 10, 12, 15, and 17 were simultaneously loosened with loosening angles of $5^{\circ }$, ${16}^{\circ }$, ${21}^{\circ }$, ${12}^{\circ }$, ${25}^{\circ }$, and ${30}^{\circ }$, respectively. The experiment was repeated six times. The root-mean-square error (RMSE) between actual loosening values and algorithm-predicted values was used as a metric to evaluate the algorithm’s performance in identifying looseness.

The identification results are shown in Table 6, from which it can be observed that the loosened angles identified by the algorithm closely match the actual applied angles. The RMSE for the six loosening trials ranged from $0.{67}^{\circ }$ to $1.{83}^{\circ }$, indicating high accuracy of the proposed algorithm. This represents a significant improvement in precision compared to the study by Reference [26], which also used rotational angle as a metric. Additionally, the proposed method processes images of bolts at laboratory scale in approximately 0.5 s, which is about twice as fast as the method in Reference [26].

3.6. Experimental Validation in Field Environment

The primary objective of field testing is to incorporate environmental factors and consider possible working conditions that may occur in actual engineering scenarios. Building on the laboratory environment tests, field tests were conducted to simulate various factors such as different numbers of bolts, bolt corrosion, different coatings, and loosening scenarios.

3.6.1. Testing with Different Numbers of Bolts

In real bridges, the number of bolts at each connection varies, ranging from a few dozen to over a hundred. To verify that the proposed algorithm maintains good performance despite variations in bolt numbers on-site, a test was conducted on a bridge site in Hangzhou. This test included six scenarios with different numbers of bolts: 24, 32, 40, 56, 120, and 150 bolts, respectively. An industrial camera was used to capture images for each scenario from a distance of 1.5 m, maintaining a horizontal line of sight and adjusting the focus to fill the image frame with bolts in each scenario. The collected bolt images were then processed for recognition to verify the algorithm’s accuracy under varying numbers of bolts, with the results shown in Figure 21.

As shown in Figure 21, the algorithm accurately identified all bolts in the images for each scenario without any missed or false detections. A manual measurement of the actual angles was compared against the angles identified by the algorithm, with the results presented in Table 7. From Table 7, it is evident that errors increase with the number of bolts. The analysis reveals that as the number of bolts in an image increases, the pixel ratio per bolt decreases, leading to reduced keypoint localization accuracy. As a result, the algorithm’s identified positions deviate from the actual positions, increasing angle calculation errors and thereby affecting overall detection performance. Therefore, in practical applications, to ensure detection accuracy, pre-cropping of input images is recommended to limit the number of bolts to within 120, and the proportion of the bolt region of interest should be increased.

3.6.2. Bolt Corrosion Testing

Bolts exposed to external structural elements experience corrosion and aging over time. To verify the algorithm’s effectiveness in detecting bolts with varying degrees of corrosion, a smartphone was used at a bridge site to capture three bolt connection plates exhibiting different levels of corrosion: initial corrosion, moderate corrosion, and severe corrosion. The smartphone was positioned 1.5 m away from the bolt groups with normal lighting conditions. All bolt plates were coated in white, and the numbers of bolts were 79, 78, and 29, respectively. The captured images of the three scenarios were processed to identify the bolt angles, with the recognition results presented in Figure 22.

As shown in Figure 22, the algorithm accurately identifies and displays the angle of each bolt for all three corrosion scenarios. To precisely evaluate the algorithm’s accuracy under different corrosion conditions, manual measurements of the bolt angles were compared with the angles recognized by the algorithm. The comparison results are shown in Table 8. It is evident that as the degree of corrosion on the bolt connection plate increases, the error also increases. The analysis indicates that the rise in error is due to characteristic blurring of corner points caused by corrosion, leading to greater deviation in corner point localization. Overall, the detection method’s accuracy is affected by the aging of the bolt connection plates. The primary reason is the limited number of corroded samples in the original images, which prevents the algorithm from accurately recognizing the features of corroded bolts. Increasing the number of corrosion samples in future datasets could address this issue.

3.6.3. Effects of Bolt Surface Coating

The coatings of each bridge vary according to practical requirements, and with increasing service time, the surface may suffer varying degrees of damage. To examine the algorithm’s generalization capability under different coating conditions, bolt images were collected from a bridge site under four scenarios: white coating, black coating, coating peeling, and surface contamination. The bolt angles were recognized, and the recognition results are depicted in Figure 23.

The recognition outcomes are summarized in Table 9. As shown in Table 9, each coating condition resulted in high recognition accuracy. The test results indicate that the proposed algorithm possesses strong generalization capabilities for bolts with different coating effects.

4. Relationship Study Between High-Strength Bolt Pretension and Loosening Angle

In the image recognition detection system, setting a maximum threshold for bolt loosening angles facilitates a scientifically effective alarm mechanism. This system assists construction personnel in timely locating bolt defects and enables quick maintenance of defective bolts. In bolt connections, the level of bolt pretension determines the specific state of bolt fastening. Therefore, this section establishes a connection between the pretension change and loosening angle during the bolt loosening process based on factual data and experimental data analysis. Through scientific mathematical derivation, a theoretical basis is provided for setting the maximum loosening angle threshold in image recognition.

An electric torque wrench was used to perform tightening-loosening work on bolts installed on an axial force tester, and a digital axial force gauge was employed to measure the bolt pretension in real-time. This process linked the change in bolt pretension with the bolt’s loosening angle. The experiment focused on a 10.9S M24 large hexagon head steel bridge bolt, commonly used in steel structure bridges. In the experiment, an electric torque wrench was used to tighten three groups of M24 bolts to meet the pretension standard value. Subsequently, an electric inspection wrench was used for successive loosening of M24 bolts by $5^{\circ }$. A polynomial fitting of the experimental data was conducted, and the fitted curves for multiple groups of pretension/initial pretension-loosening angle are shown in Figure 24. The polynomial fitting yields the relationship between experimental measured pretension/initial pretension and bolt loosening angle as:

(15)$y=-0.903x+100.35$

According to the standards [46], when the bolt’s measured pretension loss is less than 10% of the initial pretension, or the measured pretension remains above 90% of the initial pretension, the bolt is considered adequately fastened. Conversely, if the measured pretension drops below 90% of the initial pretension, the bolt requires tightening. Substituting $y=90$ into the polynomial provides a bolt loosening angle $x={11}^{\circ }$. In the image recognition detection system, the maximum bolt loosening threshold is set to ${11}^{\circ }$. Consequently, any bolt loosening angle exceeding ${11}^{\circ }$ will trigger the alarm mechanism within the image recognition detection system.

5. Conclusions

This study focuses on the detection of loosening in millions of high-strength bolt connections in steel structure bridges under complex environmental conditions. Addressing the high costs, low efficiency, and poor safety of several mainstream bolt loosening recognition methods, this research integrates theories from deep learning and digital image processing. It employs advanced technologies such as machine vision to develop a rapid recognition method for mass bolt loosening in steel structure bridges under complex conditions. The main conclusions are as follows:

1. Based on deep learning theory, a high-precision bolt keypoint localization method is proposed. Offline and online augmentation methods were used to expand the acquired raw dataset and construct a bolt image dataset. By adding deconvolutional layers to the ResNet-50 network, a bolt keypoint localization model was built. The keypoint localization experiments using newly acquired bolt images showed that the model achieved good positioning results with minimal error. The robustness of the model was tested by altering lighting conditions, which demonstrated its excellent adaptability to changes in lighting.

2. A bolt loosening angle recognition method based on digital image processing technology was proposed. Laboratory tests demonstrated that the centroid-corner clustering algorithm achieved high accuracy and recall rates. Angle accuracy tests revealed a high correlation between algorithm-identified values and manually measured values. Field tests validated the model’s performance in real-world complex scenarios to demonstrate the generalization capability of the proposed bolt loosening detection method, such as variations in the number of bolts, corrosion effects, and coating conditions. The detection accuracy under different scenarios was compared, and potential improvement methods were analyzed.

3. The relationship between the loosening angle and pretension of commonly used high-strength bolts in laboratory highway steel bridges is established, and the threshold for bolt loosening angles in image recognition is provided. This outcome offers a theoretical foundation for setting effective detection intervals for bolt loosening in future image recognition-based bolt loosening technology systems. It enhances the timely warning capabilities of bolt loosening detection systems, enabling rapid manual inspection and repair.

Author Contributions

Conceptualization, Z.S., X.Q., and Z.Z.; methodology, X.Q., Z.Z., and H.J.; software, X.Q.; validation, Z.S. and X.Q.; formal analysis, X.Q.; investigation, Z.S. and Z.Z.; resources, Z.S.; data curation, X.Q., Z.Z., and H.J.; writing—original draft preparation, Z.S.; writing—review and editing, Z.S.; visualization, X.Q.; supervision, X.Q.; project administration, Z.S.; funding acquisition, Z.S. All authors have read and agreed to the published version of the manuscript.

Institutional Review Board Statement

The original contributions presented in the study are included in the article. Further inquiries can be directed to the corresponding author.

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors on request.

Acknowledgments

The authors wish to express their thanks for the financial assistance.

Conflicts of Interest

Authors Zhiqiang Shang, Zejun Zhang and Hongtao Jiang are employed by the Shandong Hi-Speed Group Co., Ltd. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figures and Tables

Figure 1. Torque Wrench Measurement Method.

Figure 2. Scribed Line and Magnetic Strip Attachment Methods.

Figure 3. Two Categories of Vision-Based Detection Method. (a) Angle Detection; (b) Length Detection.

Figure 4. Framework for Bolt-loosening Detection.

Figure 5. Example of True Position Annotation of Bolt Key Points.

Figure 6. Process Flow of Bolt Keypoint Localization Algorithm.

Figure 7. Illustration of Bolt Keypoint Heatmaps. (a) Corner Heatmap; (b) Corner Heatmap.

Figure 8. Loss Function Curve During Training.

Figure 9. Bolt Keypoint Detection Results (a) Front view; (b) Back view; (c) Slanted view.

Figure 10. Keypoint detection results under varying illumination conditions. (a) Strong lighting condition; (b) normal lighting condition; (c) weak lighting condition; (d) dark condition.

Figure 11. Outlier determination for bolt corner points.

Figure 12. Principle of initial angle calculation for bolts.

Figure 13. Principle of bolt position numbering.

Figure 14. Process of correcting distorted images using planar projection transformation.

Figure 15. Calculation of bolt loosening angle.

Figure 16. Binary image of keypoints.

Figure 17. Clustering effect of bolt keypoints.

Figure 18. Bolt angle identification (a) Front view; (b) Back view.

Figure 19. Comparison between identified and measured values.

Figure 20. Schematic diagram of bolt loosening detection.

Figure 21. Test results for different numbers of bolts.

Figure 22. Test results for different degrees of corrosion.

Figure 23. Test results for different coating conditions.

Figure 24. Polynomial fitted curve of experimental pretension/initial pretension vs. bolt loosening angle.

Table 1

Key Point Positioning Network Structure.

Layer	Structure	Channels	Image Size
1	$7\times 7$ convolution, stride 2 $3\times 3$ max pooling, stride 2	64	$1024\times 768$
2	$\left[\begin{array}{c}1\times 1\mathrm{convolution},64\mathrm{channels}\hfill \\ 3\times 3\mathrm{convolution},64\mathrm{channels}\hfill \\ 1\times 1\mathrm{convolution},256\mathrm{channels}\hfill \end{array}\right]\times 3$	256	$256\times 192$
3	$\left[\begin{array}{c}1\times 1\mathrm{convolution},256\mathrm{channels}\hfill \\ 3\times 3\mathrm{convolution},256\mathrm{channels}\hfill \\ 1\times 1\mathrm{convolution},1024\mathrm{channels}\hfill \end{array}\right]\times 3$	512	$128\times 96$
4	$\left[\begin{array}{c}1\times 1\mathrm{convolution},512\mathrm{channels}\hfill \\ 3\times 3\mathrm{convolution},512\mathrm{channels}\hfill \\ 1\times 1\mathrm{convolution},2048\mathrm{channels}\hfill \end{array}\right]\times 3$	1024	$64\times 48$

Table 2

Coordinate Comparison Results.

Index	Model Coordinates	Manual Coordinates
1	Corner 1 x: 830.8, Corner 1 y: 132.1	Corner 1 x’: 830, Corner 1 y’: 132
2	Corner 2 x: 816.5, Corner 2 y: 119.5	Corner 2 x’: 814, Corner 2 y’: 118
3	Center x: 812.5, Center y: 130.5	Center x’: 810, Center y’: 136
4	Corner 1 x: 677.3, Corner 1 y: 182.9	Corner 1 x’: 676, Corner 1 y’: 184
5	Corner 2 x: 661.8, Corner 2 y: 173.4	Corner 2 x’: 660, Corner 2 y’: 172
6	Center x: 659.5, Center y: 185.4	Center x’: 658, Center y’: 191
7	Corner 1 x: 813.2, Corner 1 y: 312.5	Corner 1 x’: 814, Corner 1 y’: 312
8	Corner 2 x: 793.2, Corner 2 y: 311.4	Corner 2 x’: 792, Corner 2 y’: 310

Table 3

Distance Results between Manual Annotations and Model Detection.

Index	Corner Point 1/Pixel	Corner Point 2/Pixel	Center Point/Pixel
1	0.85	1.89	6.06
2	1.72	2.26	5.81
3	0.90	1.85	4.62
4	0.17	1.00	4.98
5	0.71	0.30	6.28
6	1.51	3.16	4.59
7	0.29	0.82	5.25
8	0.86	1.06	4.71

Table 4

Corner Localization Results.

Lighting Condition	PCK (%)	Accuracy (%)
Strong Light	96.2	93.4
Normal Light	97.5	95.6
Low Light	96.9	95.2
Dark Light	95.3	94.5

Table 5

Errors between identified and measured values.

Maximum Error/∘	Minimum Error/∘	Average Error/∘
2	0	0.87

Table 6

Results of Bolt Loosening Test.

Index	Bolt 1	Bolt 2	Bolt 3	Bolt 4	Bolt 5	Bolt 6
Actual Loosening Angle (∘)	5	16	21	12	25	30
Maximum Error (∘)	1	2	2	2	2	2
RMSE (∘)	0.67	1.33	1.33	1.83	1.33	1.5

Table 7

Errors of testing with different numbers of bolts.

Number of Bolts	24	32	40	56	120	150
Maximum Error (∘)	1	1	2	3	3	5
RMSE (∘)	0.85	0.97	1.22	1.31	1.82	3.12

Table 8

Errors under different corrosion levels.

Condition	Initial Corrosion	Moderate Corrosion	Severe Corrosion
RMSE (∘)	1.22	1.86	2.78
Maximum Error (∘)	2	3	4

Table 9

Errors under different coating conditions.

Condition	White	Black	Peeling	Contamination
RMSE (∘)	1.54	1.58	1.59	1.53
Maximum Error (∘)	2	2	3	2

References

1. Wang, T.; Song, G.; Liu, S.; Li, Y.; Xiao, H. Review of Bolted Connection Monitoring. Int. J. Distrib. Sens. Netw.; 2013; 9, 871213. [DOI: https://dx.doi.org/10.1155/2013/871213]

2. Pan, Q.; Pan, R.; Shao, C.; Chang, M.; Xu, X. Research Review of Principles and Methods for Ultrasonic Measurement of Axial Stress in Bolts. Chin. J. Mech. Eng.; 2020; 33, pp. 1-16. [DOI: https://dx.doi.org/10.1186/s10033-020-0431-x]

3. Miao, R.; Shen, R.; Zhang, S.; Xue, S. A Review of Bolt Tightening Force Measurement and Loosening Detection. Sensors; 2020; 20, 3165. [DOI: https://dx.doi.org/10.3390/s20113165] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/32498473]

4. Feblil, H.; Itsuro, K.; Naoki, H.; Shozo, K. Bolt loosening analysis and diagnosis by non-contact laser excitation vibration tests. Mech. Syst. Signal Process.; 2013; 40, pp. 589-604.

5. Yokoyama, T.; Olsson, M.; Izumi, S.; Sakai, S. Investigation into the self-loosening behavior of bolted joint subjected to rotational loading. Eng. Fail. Anal.; 2012; 23, pp. 35-43. [DOI: https://dx.doi.org/10.1016/j.engfailanal.2012.01.010]

6. Nikravesh, S.M.Y.; Goudarzi, M. A review paper on looseness detection methods in bolted structures. Lat. Am. J. Solids Struct.; 2017; 14, pp. 2153-2176. [DOI: https://dx.doi.org/10.1590/1679-78254231]

7. Land, H.B. Determination of the cause of arcing faults in low-voltage switchboards. IEEE Trans. Ind. Appl.; 2008; 44, pp. 430-436. [DOI: https://dx.doi.org/10.1109/TIA.2008.916595]

8. Park, J.; Kim, T.; Kim, J. Image-based bolt-loosening detection technique of bolt joint in steel bridges. Proceedings of the 6th International Conference on Advances in Experimental Structural Engineering 11th International Workshop on Advanced Smart Materials and Smart Structures Technology; Urbana, IL, USA, 1–2 August 2015; pp. 1-2.

9. Martucci, D.; Civera, M.; Surace, C. Bridge monitoring: Application of the extreme function theory for damage detection on the I-40 case study. Eng. Struct.; 2023; 279, 115573. [DOI: https://dx.doi.org/10.1016/j.engstruct.2022.115573]

10. Huang, J.; Liu, J.; Gong, H.; Deng, X. A comprehensive review of loosening detection methods for threaded fasteners. Mech. Syst. Signal Process.; 2022; 168, 108652. [DOI: https://dx.doi.org/10.1016/j.ymssp.2021.108652]

11. Shao, J. Study on Mechanism and Method of Bolt-Connectionstate Monitoring Based on Frequency Variation of Piezoelectric Impedance. Ph.D. Thesis; Wuhan University of Science and Technology: Wuhan, China, 2018.

12. Graybeal, B.A.; Phares, B.M.; Rolander, D.D.; Moore, M.; Washer, G. Visual inspection of highway bridges. J. Nondestruct. Eval.; 2002; 21, pp. 67-83. [DOI: https://dx.doi.org/10.1023/A:1022508121821]

13. Razi, P.; Esmaeel, R.A.; Taheri, F. Improvement of a vibration-based damage detection approach for health monitoring of bolted flange joints in pipelines. Struct. Health Monit.; 2013; 12, pp. 207-224. [DOI: https://dx.doi.org/10.1177/1475921713479641]

14. Sun, F.; Chaudhry, Z.; Liang, C.; Rogers, C.A. Truss structure integrity identification using PZT sensor-actuator. J. Intell. Mater. Syst. Struct.; 1995; 6, pp. 134-139. [DOI: https://dx.doi.org/10.1177/1045389X9500600117]

15. Xu, B.; Jiang, F. Concrete-steel composite girder bolt loosening monitoring using electromechanical impedance measurements. Earth and Space 2012: Engineering, Science, Construction, and Operations in Challenging Environments; American Society of Civil Engineers: Reston, VA, USA, 2012; pp. 629-634.

16. Yang, J.N.; Xia, Y.; Loh, C.H. Damage identification of bolt connections in a steel frame. J. Struct. Eng.; 2014; 140, 04013064. [DOI: https://dx.doi.org/10.1061/(ASCE)ST.1943-541X.0000831]

17. Sah, S.M.; Thomsen, J.J.; Brøns, M.; Fidlin, A.; Tcherniak, D. Estimating bolt tightness using transverse natural frequencies. J. Sound Vib.; 2018; 431, pp. 137-149. [DOI: https://dx.doi.org/10.1016/j.jsv.2018.05.040]

18. An, Y.K.; Sohn, H. Integrated impedance and guided wave based damage detection. Mech. Syst. Signal Process.; 2012; 28, pp. 50-62. [DOI: https://dx.doi.org/10.1016/j.ymssp.2011.11.016]

19. Zhang, M.; Shen, Y.; Xiao, L.; Qu, W. Application of subharmonic resonance for the detection of bolted joint looseness. Nonlinear Dyn.; 2017; 88, pp. 1643-1653. [DOI: https://dx.doi.org/10.1007/s11071-017-3336-1]

20. Park, J.H.; Huynh, T.C.; Choi, S.H.; Kim, J.T. Vision-based technique for bolt-loosening detection in wind turbine tower. Wind Struct.; 2015; 21, pp. 709-726. [DOI: https://dx.doi.org/10.12989/was.2015.21.6.709]

21. Zhao, X.; Zhang, Y.; Wang, N. Bolt loosening angle detection technology using deep learning. Struct. Control Health Monit.; 2019; 26, e2292. [DOI: https://dx.doi.org/10.1002/stc.2292]

22. Kong, X.; Li, J. Image registration-based bolt loosening detection of steel joints. Sensors; 2018; 18, 1000. [DOI: https://dx.doi.org/10.3390/s18041000]

23. Huynh, T.C.; Park, J.H.; Jung, H.J.; Kim, J.T. Quasi-autonomous bolt-loosening detection method using vision-based deep learning and image processing. Autom. Constr.; 2019; 105, 102844. [DOI: https://dx.doi.org/10.1016/j.autcon.2019.102844]

24. Pham, H.C.; Ta, Q.B.; Kim, J.T.; Ho, D.D.; Tran, X.L.; Huynh, T.C. Bolt-loosening monitoring framework using an image-based deep learning and graphical model. Sensors; 2020; 20, 3382. [DOI: https://dx.doi.org/10.3390/s20123382] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/32549378]

25. Wu, J. A Study on Damage Identification for Bridge Structures. Ph.D. Thesis; South China University of Technology: Guangzhou, China, 2020.

26. Huynh, T.C. Vision-based autonomous bolt-looseness detection method for splice connections: Design, lab-scale evaluation, and field application. Autom. Constr.; 2021; 124, 103591. [DOI: https://dx.doi.org/10.1016/j.autcon.2021.103591]

27. Yang, X.; Gao, Y.; Fang, C.; Zheng, Y.; Wang, W. Deep learning-based bolt loosening detection for wind turbine towers. Struct. Control Health Monit.; 2022; 29, e2943. [DOI: https://dx.doi.org/10.1002/stc.2943]

28. Cha, Y.J.; You, K.; Choi, W. Vision-based detection of loosened bolts using the Hough transform and support vector machines. Autom. Constr.; 2016; 71, pp. 181-188. [DOI: https://dx.doi.org/10.1016/j.autcon.2016.06.008]

29. Ramana, L.; Choi, W.; Cha, Y.J. Fully automated vision-based loosened bolt detection using the Viola-Jones algorithm. Struct. Health Monit.; 2019; 18, pp. 422-434. [DOI: https://dx.doi.org/10.1177/1475921718757459]

30. Wang, C.; Wang, N.; Ho, M.; Chen, X.; Song, G. Design of a New Vision-based Method for the Bolts Looseness Detection in Flange Connections. IEEE Trans. Ind. Electron.; 2019; 67, pp. 1366-1375. [DOI: https://dx.doi.org/10.1109/TIE.2019.2899555]

31. Zhang, Y.; Sun, X.; Loh, K.J.; Su, W.; Xue, Z.; Zhao, X. Autonomous bolt loosening detection using deep learning. Struct. Health Monit.; 2020; 19, pp. 105-122. [DOI: https://dx.doi.org/10.1177/1475921719837509]

32. Gong, H.; Deng, X.; Liu, J.; Huang, J. Quantitative loosening detection of threaded fasteners using vision-based deep learning and geometric imaging theory. Autom. Constr.; 2022; 133, 104009. [DOI: https://dx.doi.org/10.1016/j.autcon.2021.104009]

33. LeCun, Y.; Bengio, Y.; Hinton, G. Deep learning. Nature; 2015; 521, pp. 436-444. [DOI: https://dx.doi.org/10.1038/nature14539]

34. Wu, X.; Song, X.; Gao, S.; Chen, C. Convolution Neural Network Based on Data Enhancement for Fire Identification. Sci. Technol. Eng.; 2020; 20, 5.

35. Sroba, L.; Grman, J.; Ravas, R. Impact of Gaussian noise and image filtering to detected corner points positions stability. Proceedings of the 2017 11th International Conference on Measurement; Smolenice, Slovakia, 29–31 May 2017.

36. Liu, X.; Wang, H.; Zhou, J.; Tao, L. Attention-guided coarse-to-fine network for 2d face alignment in the wild. IEEE Access; 2019; 7, pp. 97196-97207. [DOI: https://dx.doi.org/10.1109/ACCESS.2019.2930304]

37. Zhang, W.; Qian, Y. Streamlined Face Landmark Detection Network with Global Constraint. J. Signal Process.; 2019; 35, pp. 507-515.

38. Xiao, B.; Wu, H.; Wei, Y. Simple baselines for human pose estimation and tracking. Proceedings of the European Conference on Computer Vision (ECCV); Munich, Germany, 8–14 September 2018; pp. 466-481.

39. Chen, H.; Yao, M.; Gu, Q. Pothole detection using location-aware convolutional neural networks. Int. J. Mach. Learn. Cybern.; 2020; 11, pp. 899-911. [DOI: https://dx.doi.org/10.1007/s13042-020-01078-7]

40. Atliha, V.; Šešok, D. Comparison of VGG and ResNet used as Encoders for Image Captioning. Proceedings of the 2020 IEEE Open Conference of Electrical, Electronic and Information Sciences (eStream); Vilnius, Lithuania, 30 April 2020; pp. 1-4.

41. Ioffe, S. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv; 2015; arXiv: 1502.03167

42. Krizhevsky, A.; Sutskever, I.; Hinton, G.E. ImageNet classification with deep convolutional neural networks. Commun. ACM; 2017; 60, pp. 84-90. [DOI: https://dx.doi.org/10.1145/3065386]

43. Cheng, B.; Xiao, B.; Wang, J.; Shi, H.; Huang, T.S.; Zhang, L. Higherhrnet: Scale-aware representation learning for bottom-up human pose estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; Seattle, WA, USA, 13–19 June 2020; pp. 5386-5395.

44. Dabral, R.; Gundavarapu, N.B.; Mitra, R.; Sharma, A.; Ramakrishnan, G.; Jain, A. Multi-person 3d human pose estimation from monocular images. Proceedings of the 2019 International Conference on 3D Vision (3DV); Québec City, QC, Canada, 16–19 September 2019; pp. 405-414.

45. Krishna, K.; Murty, M.N. Genetic K-means algorithm. IEEE Trans. Syst. Man Cybern. Part B (Cybern.); 1999; 29, pp. 433-439. [DOI: https://dx.doi.org/10.1109/3477.764879]

46. GB/T 16823.2-1997 General Rules of Tightening for Threaded Fasteners; Ministry of Machinery of the People’s Republic of China: Beijing, China, 2017.