A Novel Transformer Network Based on Cross–Spatial Learning and Deformable Attention for Composite Fault Diagnosis of Agricultural Machinery Bearings

Abstract

Diagnosing agricultural machinery faults is critical to agricultural automation, and identifying vibration signals from faulty bearings is important for agricultural machinery fault diagnosis and predictive maintenance. In recent years, data–driven methods based on deep learning have received much attention. Considering the roughness of the attention receptive fields in Vision Transformer and Swin Transformer, this paper proposes a Shift–Deformable Transformer (S–DT) network model with multi–attention fusion to achieve accurate diagnosis of composite faults. In this method, the vibration signal is first transformed into a time–frequency graph representation through continuous wavelet transform (CWT); secondly, dilated convolutional residual blocks and efficient attention for cross–spatial learning are used for low–level local feature enhancement. Then, the shift window and deformable attention are fused into S–D Attention, which has a more focused receptive field to learn global features accurately. Finally, the diagnosis result is obtained through the classifier. Experiments were conducted on self–collected datasets and public datasets. The results show that the proposed S–DT network performs excellently in all cases. With a slight decrease in the number of parameters, the validation accuracy improves by more than 2%, and the training network has a fast convergence period. This provides an effective solution for monitoring the efficient and stable operation of agricultural automation machinery and equipment.

1. Introduction

Automation and intelligence in the field of agricultural machinery have received much attention. Rolling bearings, which are important components of rotating machinery, are also important components of agricultural automation equipment, such as automatic tractors, weeding robots, combine harvesters, and drones. Given that the operating environment for rolling bearings in agricultural machinery is typically enclosed, complex, and changeable, rolling bearings are very vulnerable, and their health status affects the performance, stability, and life cycle of the entire rotating machinery [1,2]. Health testing and accurate diagnosis of faults in rolling bearings can minimize maintenance costs and avoid production accidents. Therefore, fault diagnosis of rolling bearings is an important research direction for monitoring the health of agricultural equipment.

The technical routes for rolling bearing fault diagnosis are mainly divided into traditional signal analysis theory and machine learning. As a subset of machine learning, deep learning is highly popular due to its advantages of automatically learning deep features, and there is no need to design complex feature extraction methods, greatly improving diagnostic performance. Convolutional neural networks (CNNs) are good at processing image data and are suitable for the time–frequency image analysis of signals [3,4]. Recurrent neural networks (RNNs) are adept at capturing time series information and can better process complex time series data [5]. Many researchers have utilized these models for diagnosing rolling bearing faults. Reference [6] proposed a general bearing fault diagnosis model converted from the AlexNet model. By replacing the fully connected layer, the input data of different sizes are converted into time–frequency images of uniform size to address the defect that CNN loses universality when the input form is different. This method helps with feature extraction and classification of bearing faults. Reference [7] proposed a fault diagnosis method using the optimal mode component as the input. The optimal component is selected through ABC–VMD and weighted evaluation function, and the improved EfficientNet model is used for diagnosis. It can accurately identify the rolling bearing fault types in corn harvesters in high–noise environments.

With its internal multi–head attention mechanism, Vision Transformer (ViT) [8] overcomes the disadvantage of CNN’s difficulty in effectively capturing long–distance dependencies. It can model dependencies between any position in the input sequence without being limited by distance; this enables ViT to greatly increase the receptive field and obtain global feature information without a large amount of stacking. Although ViT has achieved good results, it relies on a global attention mechanism and struggles to effectively learn the local features of the input; thus, it can perform worse than CNN in image classification tasks [9]. To this end, Liu et al. [10] designed a Transformer based on shift window attention (Swin Transformer), which divides the input into non–overlapping windows of uniform size and limits the self–attention calculation to the local window. This can better capture local features. Chen et al. [11] proposed a multi–channel calibrated Transformer with shift windows (MC Swin–T), which used multiple convolutional layers before the Transformer block to extract low–level signal features, and alternately used two different window partitioning methods to establish connections between windows. Wang et al. [12] proposed a novel transfer residual Swin Transformer (RST), which hierarchically designs 24 residual self–attention layers based on shift windows, achieving end–to–end fault diagnosis based on deep transfer RST. Reference [13] proposed a new method for rolling bearing fault diagnosis of agricultural machinery based on SVD–EDS–GST and ResNet–Vision Transformer (ResViT). ResViT replaces the image segmentation mechanism in ViT with the ResNet34 network, improves the local feature extraction effect, and integrates global and local information for fault classification.

The Swin Transformer model converges slowly when processing data with sparse features due to the smoothness of attention distribution. We use an efficient convolutional residual module to enhance fault features and propose a new model called S–D Transformer for effective fault diagnosis.

The main contributions of this work can be summarized as follows:

1. We collected vibration signals of rolling bearings under normal conditions and nine types of faults using a self–built experimental platform. The collected data and three other publicly available datasets were transformed into time–frequency map datasets using continuous wavelet transform (CWT).

2. An efficient feature extraction block was designed to improve the model’s local perception ability, achieve spatial downsampling, and increase the number of feature maps. Meanwhile, efficient multi–scale attention (EMA) is combined with spatial attention to enhance local features through cross–spatial learning, while simultaneously focusing on important channel and position information.

3. Swin Transformer is introduced with convertible attention, allowing self–attention modules to focus on relevant fields and capture more information. This fault diagnosis method has high accuracy and strong generalization ability in experiments with multiple datasets.

The remainder of this paper is organized as follows. Section 2 introduces the relevant technologies. Section 3 describes the proposed network model in detail. Section 4 introduces the datasets used for the experiment. Section 5 comprehensively demonstrates the performance of the proposed method, including comparisons with other methods, ablation experiments, and generalization testing. Finally, Section 6 presents the conclusions.

2. Related Work

The application of deep learning in rolling bearing fault diagnosis for agricultural machinery is a multidisciplinary issue, encompassing fields such as agricultural machinery, signal processing, and computer vision. The process involves several steps: firstly, collecting vibration signal data from the faulty bearing; secondly, transforming this vibration signal to generate a time–frequency diagram; then, constructing a deep learning network architecture and training the network using the time–frequency maps. Finally, the performance of the trained model is evaluated using a validation set.

2.1. Continuous Wavelet Transform

Continuous wavelet transform is a time–frequency analysis method based on multi– scale analysis. It represents a one–dimensional time series signal as a two–dimensional color wavelet time–frequency map, which can provide detailed information on the vibration signal in time and frequency [14,15]. This mapping can comprehensively reflect the joint time–frequency characteristics of the signal, fully highlight certain features, and focus on any details of the signal. CWT can provide time–frequency localization information of signals, which better captures the instantaneous characteristics of signals in time. CWT can be expressed as follows:

(1)$\begin{array}{c}\hfill Wf(a,b)={\int }_{-\infty }^{+\infty }f\left(t\right){{\psi }^{\ast }}_{a,b}\left(t\right)dt={\displaystyle \frac{1}{\sqrt{\left|a\right|}}}{\int }_{-\infty }^{+\infty }f\left(t\right){\psi }^{\ast }\left({\displaystyle \frac{t-b}{a}}\right)dt\end{array}$

where $Wf(a,b)$ represents the coefficients of the wavelet function; a and b are the scale parameter and translation parameter, respectively, $f\left(t\right)$ represents the original time domain signal; ${\psi }^{\ast }\left(t\right)$ represents the conjugate complex number of the wavelet basis function.

Morlet wavelet, which is used as the wavelet basis function in CWT in our model, is suitable for time–frequency analysis and processing of signals containing frequency changes. The formula is as follows:

(2)$\begin{array}{c}\hfill \psi \left(t\right)={\pi }^{-{\displaystyle \frac{1}{4}}}{e}^{j{w}_{0}t}{e}^{-{\displaystyle \frac{t^2}{2}}}\end{array}$

where $w_0$ is the center frequency; j is the imaginary unit.

By substituting Morlet wavelets into the continuous wavelet transform formula and simplifying it, we can obtain the following:

(3)$\begin{array}{c}\hfill Wf(a,b)={\displaystyle \frac{1}{\sqrt{\left|a\right|}}}{{\int }_{-\infty }^{+\infty }f\left(t\right)\left({\pi }^{-{\displaystyle \frac{1}{4}}}{e}^{-j{w}_0{\displaystyle \frac{t-b}{a}}}{e}^{-{\displaystyle \frac{{(t-b)}^2}{2a^2}}}\right)}^{\ast }dt\end{array}$

The time domain information is divided into 1024 lengths in the form of a sliding window, which is then converted into an RGB image. The image size is 3 × 224 × 244 to match the input format required by the network.

ViT abandons the traditional convolution algorithm and has a novel model structure. On this basis, Swin Transformer cleverly uses the conversion window attention to build the model. Therefore, we introduce its model structure and attention algorithm.

2.2. Attention Mechanism

The self-attention mechanism has powerful global modeling capabilities [16,17]. The multi-head attention mechanism goes a step further, enabling the model to focus on different positions of the input information at the same time and capturing different aspects of the input information. The input includes three vectors: query (Q), key (K), and value (V).

As shown in Figure 1, the multi-head attention mechanism linearly converts the input sequence data into multiple sets of queries, keys, and values, which are then processed in parallel by multiple attention heads. All outputs are concatenated and a linear transformation is performed on the concatenated result to obtain the final attention output [18]. The calculation formula is as follows:

(4)$\begin{array}{c}Multi-Head(Q,K,V)=Concat(hea{d}_1,\dots ,hea{d}_i)W^0\hfill \end{array}$

(5)$\begin{array}{c}hea{d}_i=Attention(Q{W}_i^Q,K{W}_i^K,V{W}_i^V)\hfill \end{array}$

(6)$\begin{array}{c}Attention=\mathrm{softmax}\left({\displaystyle \frac{Q{K}^T}{\sqrt{d_k}}}\right)V\hfill \end{array}$

where $hea{d}_i$ represents the output of the i-th head; $W^0$ is the output transformation matrix; $W_i^Q$, $W_i^K$, and $W_i^V$ are the transformation matrices for the i-th head query, key, and value, respectively; $d_k$ is the dimension of the key vector.

The multi–head mechanism effectively enhances the model’s expressive power, enabling it to learn more diverse and complex features. Swin Transformer performs window partition shifting in continuous self–attention layers, which allows communication between windows and significantly enhances modeling capabilities.

2.3. Swin–T Encoder Block

Similar to ViT, the Swin Transformer model structure is composed of four stages stacked together. Each stage contains several Swin Transformer blocks and a patch merging. The Swin Transformer block is the core encoding layer, which is divided into two sub–layers. The first sub–layer is the multi–head self–attention (MSA) and the second sub–layer is a multi–layer perceptron (MLP). Layer normalization (LN) is performed before these two layers to reduce training time, and the two sub–layers are connected with residuals to cope with the degradation of deep networks. Unlike ViT, the first Swin Transformer block only calculates attention within the window, so the second Swin Transformer block performs multi–head attention on the moving window to model and extract features from the global data.

3. Fault Diagnosis Model

Fault classification is one of the key tasks in bearing vibration signal analysis. We propose a method that combines continuous wavelet transform and deep learning training models. As shown in Figure 2, we have the following:

First, the bearing vibration signal is processed by CWT to obtain the wavelet time– frequency map of the vibration signal. The wavelet time–frequency map is then input into the proposed network, which can automatically learn the complex feature representations of the vibration signal wavelet graph and use them as the basis for fault classification.

3.1. Efficient Feature Extraction Block

CNNs extract features from samples through convolution operations. CNN-based models usually consist of multiple convolutional layers, pooling layers, and fully connected layers [19,20,21]. To increase the receptive field and reduce the number of calculations in CNN, downsampling is always required. Although this can increase the receptive field, the spatial resolution is reduced. Dilated convolution does not lose resolution and still expands the receptive field to obtain more multi-scale contextual information. An efficient feature extraction (EFE) module is designed, and its structure is shown in Figure 3.

As shown in Figure 3, the EFE module mainly consists of three parts. The first part includes a 7 × 7 depthwise separable convolution and a max pooling layer, the second part includes two dilated convolutional blocks, and the third part involves efficient multi–scale spatial–cross attention (EMSA).

Depthwise separable convolution is a fusion of channel–wise convolution and point–wise convolution. It splits feature maps into individual channels, performs single–channel convolution, and then reassembles them. In this process, each convolution kernel is responsible for one channel, and each channel is convolved by its designated kernel. This approach significantly reduces the computational calculation [22].

For tasks such as the fault classification of rolling bearings that consider long–range dependencies, a larger receptive field can help capture more useful feature information. The dilated convolution extends the receptive field of the convolution kernel by introducing dilation, which means that the convolution kernel of the same size can obtain a wider range of input information. The dilated convolution increases the receptive field by inserting element 0, which makes the network more sparse without increasing the number of parameters. Therefore, it can make more efficient use of parameters, reduce the number of parameters that need to be trained, and reduce the risk of overfitting [23].

Due to the characteristics of dilated convolution [24,25], some convolution kernel weights will be in unreachable locations, which will lead to information loss or imbalance. To this end, we designed a new attention module to enhance the extracted local features.

3.2. Efficient Multi–Scale Spatial–Cross Attention

Extensive experiments demonstrate that channel or spatial attention mechanisms significantly produce more identifiable features [26]. However, establishing cross–channel relationships requires channel dimensionality reduction. To preserve the information on each channel and reduce computations, efficient multi–scale spatial–cross attention (EMSA) is designed. Its structure is shown in Figure 4. The module first reshapes the channel into a batch dimension, then builds local cross–channel interactions, and uses cross–space learning methods to establish long–term dependencies between sub–networks. To this end, EMSA is divided into a shared 1 × 1 convolution branch and a 3 × 3 convolution branch.

For a given input, EMSA is divided into g sub–feature groups, for which three parallel paths are used to extract attention weights from the grouped feature maps. Two paths are 1 × 1 convolution branches and the third path is a 3 × 3 convolution kernel. We encode channel information using one–dimensional global average pooling in the 1 × 1 branch, while 3 × 3 convolution captures multi–scale feature representation. The global spatial information of the 1 × 1 branch output is encoded by 2D global average pooling, while the output of the 3 × 3 convolution is directly transformed into the corresponding dimensional shape and fused through matrix multiplication. To highlight the global context, the output feature maps within each group are aggregated using two sigmoid functions with spatial attention weight values. In the channel dimension, the max pooling and average pooling are performed and the pooling features are concatenated to retain the spatial information. Finally, the features extracted by three 3 × 3 convolutional layers are aggregated using the Sigmoid function to obtain the attention weight.

Cross–spatial information aggregation will model long–range dependencies and fuse contextual information of different scales, enabling CNN to pay more attention to high–level feature maps. Taking the above into account, parallel convolution kernels establish short–range and long–range dependencies by cross–spatial learning.

Although EMSA can enhance the features extracted by dilated convolution, its inherent flaw—“limited local receptive fields”—still restricts network performance. For some tasks that require a wider range of information interaction, more layers of network structures are needed to handle them.

3.3. S–D Transformer

3.3.1. Deformable Attention

To avoid excessive attention calculations, many studies have used clever and efficient attention to reduce computational complexity [27,28,29] and the Swin Transformer is representative (see Section 2.3). Swin Transformer attempts to address the issue of excessive attention in ViT, but the insufficient growth of the receptive field of the shift window attention limits its ability to model. Therefore, deformable attention (DA) with a deformable mechanism was born. Deformable attention is a spatially adaptive mechanism that uses a powerful yet simple design to learn a set of global keys shared across visual patches [30].

Implementing deformable attention requires three steps: learning deformable offset, sampling deformed feature maps, and calculating attention with relative position.

Learn deformable offset for deformable attention: The input feature map $x\in {\mathbb{R}}^{H\times W\times C}$ is uniformly sampled and grouped to obtain the point grid $p\in {\mathbb{R}}^{(H/g)\times (W/g)\times 2}$. We perform 1 × 1 convolution on the input to calculate the query; on the one hand, the query is directly used for multi-head attention, and on the other hand, it is used to calculate offset ${\theta }_{\mathrm{offset}}$. This is the core that is called “deformable”. According to Figure 5, the offset network consists of a 5 × 5 DW convolution, a GeLU activation function, and a 1 × 1 convolution. The offset includes horizontal and vertical axes; therefore, the output size of the Offset network is $\raisebox{1ex}{$H$}\!\left/ \!\raisebox{-1ex}{$g$}\right.\times \raisebox{1ex}{$W$}\!\left/ \!\raisebox{-1ex}{$g$}\right.\times 2$. In addition, to prevent the ${\theta }_{\mathrm{offset}}$ excessive size, the parameter S is used to limit it, as follows:

(7)$\begin{array}{c}\hfill \Delta p=stanh\left({\theta }_{\mathrm{offset}}\left(q\right)\right)\end{array}$

where $\Delta p$ is the restricted offset. q is the query value of x, linearly projected.

Sampling deformed feature maps: We add the reference point to the offset and move it to the corresponding position. We select four patches adjacent to the reference point, and perform bilinear interpolation using the sampling function $\varphi (x;(p_x,p_y))$ to extract the sampling features of these patches:

(8)$\begin{array}{c}\varphi (x;(p_x,p_y))={\displaystyle \sum _{(r_x,r_y)}}r(p_x,p_y)r(p_x,g_y)x(g_x,g_y),\hfill \end{array}$

(9)$\begin{array}{c}r(a,b)=max(0,1-|a-b\left|\right)\hfill \end{array}$

where ($p_x,p_y$) denotes the reference point coordinates; ($g_x,g_y$) denotes the positions; and $r(a,b)$ represents the calculation of function r for any point $(a,b)$.

After sampling the features, we obtain the key and value through the projection matrix. The values of the query, key, and value are as follows:

(10)$\begin{array}{c}q=x{W}_q,\tilde{k}=\tilde{x}{W}_k,\tilde{v}=\tilde{x}{W}_v\hfill \end{array}$

(11)$\begin{array}{c}\tilde{x}=\varphi (x;p+\Delta p)\hfill \end{array}$

Calculate attention with relative position: Since the four patches move with the reference point, they may become non–integers after the change. Therefore, these four patches perform bilinear interpolation based on the relative position of the bedding to determine the precise deformation position and cover all reference points:

(12)$\begin{array}{c}\hfill Attention=\mathrm{softmax}({\displaystyle \frac{q{\tilde{k}}^T}{\sqrt{d}}}+\varphi ({\displaystyle \stackrel{⌢}{B}};R\left)\right)\tilde{v}\end{array}$

where $\varphi ({\displaystyle \stackrel{⌢}{B}};R)$ represents the embedding position; ${\displaystyle \stackrel{⌢}{B}}$ is a relative position deviation table, which indexes the table by relative displacement to obtain the relative position deviation, $B$. $R$ is the relative position.

In Equation (12), the relative position offset has the relative position information between the query and key, providing spatial information for attention. Finally, the features of each head are connected and projected through W to obtain the final output (see Equation (4)). According to Figure 5b, the most significant difference between the receptive fields of ViT, Swin Transformer, and DA is that DA only focuses on a small area in the image, and this area is very important, which can reduce the number of calculations and maintain good performance.

3.3.2. Encoder Structure

To fully improve the attention computing ability of the Swin Transformer when analyzing image samples with sparse features such as wavelet time–frequency maps, we integrate deformable multi–head self–attention (D-MSA) into the encoder block. The encoding structure is shown in Figure 6. The entire model adopts a hierarchical design, consisting of four stages. Except for the first stage, each stage will first reduce the resolution of the input feature map through the patch merging layer, perform downsampling operations, and gradually expand the receptive field layer by layer, like CNN, to obtain global information.

At the beginning of the input, patch embedding is first conducted for the image of size H × W × 3. We divide the image into individual patches using a convolutional layer with a patch size of 4 and embed them into embedding, transforming the embedding size to 48; obtain the image size $\raisebox{1ex}{$H$}\!\left/ \!\raisebox{-1ex}{$4$}\right.\times \raisebox{1ex}{$W$}\!\left/ \!\raisebox{-1ex}{$4$}\right.\times 48$. We adjust the number of channels to C (usually 96) through linear embedding. Patch merging is the process of synthesizing adjacent small patches into a large patch, achieving the effect of downsampling feature maps, as shown in Figure 6b. In this way, the input of H × W × C is converted into $\raisebox{1ex}{$H$}\!\left/ \!\raisebox{-1ex}{$2$}\right.\times \raisebox{1ex}{$W$}\!\left/ \!\raisebox{-1ex}{$2$}\right.\times 2C$. After two patch merging operations, the size of the Stage 2 output is $\raisebox{1ex}{$H$}\!\left/ \!\raisebox{-1ex}{$8$}\right.\times \raisebox{1ex}{$W$}\!\left/ \!\raisebox{-1ex}{$8$}\right.\times 2C$. The size changes of Stage 3 and Stage 4 are similar. Subsequently, each stage (except for the first stage), consists of patch merging and multiple encoder blocks. As shown in Figure 6c, the feature $x^{i-1}$ (i is the i-th stage) input to this stage is normalized through LN. And the W-MSA feature is learned and connected to the input using residuals to obtain ${\widehat{x}}^i$, followed by LN and MLP, which are then connected with residuals to obtain the output feature $x^i$ of this layer.

(13)$\begin{array}{c}{\widehat{x}}^i=MS{A}_{\mathrm{front}}\left(LN\left(g^{i-1}\right)\right)+x^{i-1},\hfill \end{array}$

(14)$\begin{array}{c}{x}^i=MLP\left(LN\left({\widehat{g}}^i\right)\right)+{\widehat{x}}^i\hfill \end{array}$

(15)$\begin{array}{c}{\widehat{x}}^{i+1}=MS{A}_{\mathrm{follow}}\left(LN\left(g^i\right)\right)+x^i,\hfill \end{array}$

(16)$\begin{array}{c}{x}^{i+1}=MLP\left(LN\left({\widehat{g}}^{i+1}\right)\right)+{\widehat{x}}^{i+1}\hfill \end{array}$

In the encoder blocks of Stage 3 and Stage 4, the primary attention mechanism is shifting window multi–head self–attention (SW–MSA), followed closely by deformable multi–head attention (D–MSA), together forming Shift–Deformable Attention (SDA). However, in Stages 1 and 2, window multi–head self–attention (W–MSA) and SW–MSA from the Swin Transformer are still employed. These modules are used alternately in each stage; this structure combines the work of the Transformer and CNN series in an extremely clever way.

SW–MSA enhances information exchange between windows by moving them, effectively capturing global information. Meanwhile, D–MSA allows for the dynamic adjustment of attention positions, enabling the network to capture key local features and important details. By combining two attention mechanisms, the model can maintain excellent feature expression ability in complex scenes. In addition, the fusion of SW–MSA and D–MSA reduces the attention area, thereby reducing computation and memory usage.

4. Experiments Settings

4.1. Datasets

4.1.1. Self–Collected Bearing Dataset

The experimental data used for verification come from the self–built bearing test device (self–collected dataset, ScD). As shown in Figure 7, the test device consists of an AC servo motor (model: DB60–01330A6–A, Wenzhou Pufide Electric Co., Ltd., Wenzhou, China), a servo speed controller(model: DB60–01330A6–A, Wenzhou Pufide Electric Co., Ltd., Wenzhou, China), a coupling, two acceleration sensors (model: HN–4A, Shanghai Dongtai Sensing Technology Co., Ltd., Shanghai, China), and a signal collector (model: HN–4A, Shanghai Dongtai Sensing Technology Co., Ltd., Shanghai, China). The two acceleration sensors (one placed horizontally and one vertically) are magnetic suction piezoelectric vibration sensors with a sensitivity of 100 mv/g, and the tested bearing model is HRB–7203AC. We use electrical discharge machining (EDM) to introduce faults located in the inner ring (IR), outer ring (OR), and cage. The types of faults in the inner and outer rings include punching and slotting.

There were a total of nine types of faults (four single fault types and five composite fault types); normal bearings were also tested. The width of the hole and the slot fault of the faulty bearing were 2 mm and the depth was 0.5 mm, except for the single–point slot fault of the outer ring, where the depth was 1 mm. The experimental dataset involved signal data from the motor at different speeds. The motor speeds for Holder, IRP, and IRP_ORG were set at 1600 prm; for IRP_ORP, IRP_ORP_H, and Normal, the speed was 2000 prm, and for ORG, ORG_H, ORP, and ORP_ORG, the speed was 2400 prm. The sampling frequency was 20 kHz, and the load was 0. All data were collected only by the horizontal acceleration sensor. Before collecting data, we accelerated the target bearing to the specified speed, and after running for 5 min, we collected data with the matching collection software.

4.1.2. Public Datasets

To fully test the performance and generalization performance of the method, extensive experiments using three different datasets were used, including the Case Western Reserve University bearing dataset (CWRU) [31], Paderborn University bearing dataset (PU) [32], and Xi’an Jiaotong University bearing dataset (XJTU–SY) [33]. Their experimental setups are shown in Figure 8.

The CWRU dataset uses EDM to introduce single faults into test bearings, with a sampling frequency of 12 kHz. The PU dataset uses a piezoelectric accelerometer to collect vibration signals of the bearing seat, with a sampling frequency of 64 kHz. The XJTU–SY dataset fault points are set in the outer ring, inner ring, and holder, and the sampling frequency of 25.6 kHz.

In this experiment, ten types of bearing faults were selected for each dataset. CWRU selected normal bearings and three different fault diameters, i.e., 0.007 inches, 0.014 inches, and 0.021 inches; PU selected five artificially damaged bearings, three damaged bearings during accelerated life testing, and two normal bearings. XJTU–SY selected three fault types for operating conditions 1 and 2, and four fault types for operating condition 3.

We used the sliding window to slice the original data; the slice length was 1024. Then we used CWT to generate 1000 time–frequency maps, 224 × 224, for each of the 10 signals. The entire dataset was divided into a training set, a validation set, and a test set at a ratio of 600:200:200. The fault data and their labels selected in CWRU, PU, and XJTU–SY bearing datasets are shown in Table 1.

4.2. Evaluation Metrics and Parameter Settings

In deep learning–based classification tasks, selecting appropriate evaluation metrics is crucial for evaluating model performance. In terms of evaluating the accuracy of classification, the following two evaluation indicators are used:

(17)$\begin{array}{cc}\hfill Accuracy& ={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}\hfill \end{array}$

(18)$\begin{array}{cc}\hfill F1& ={\displaystyle \frac{2TP}{2TP+FP+FN}}\hfill \end{array}$

where $TP$, $TN$, $FP$, and $FN$ represent the number of True Positions, True Negatives, False Positions, and False Negatives, respectively.

Accuracy refers to the ratio of the number of samples predicted correctly by the model to the total number of samples. A high accuracy rate means that the model can correctly classify most samples. The F1 score refers to the harmonic mean of accuracy and recall, which comprehensively considers the accuracy and recall of classification models. A high F1 score means that the model has fewer false positives and omissions while correctly classifying. For a model, time and space complexity are crucial to whether the method can be applied in practice. FLOPs are often used to evaluate the time complexity of a model, which refers to the number of floating–point operations required to run the network model once. Parameter quantity refers to the number of parameters that need to be learned in the model, which is an important indicator for measuring the spatial complexity of the model. The more parameters a model has, the more GPU memory is required for training the model, and larger disks are needed to store the model parameters. The model proposed can accurately diagnose faults. Its parameter details are shown in Table 2.

5. Experimental Results and Analysis

5.1. Accuracy Comparison

To fully verify and analyze the performance of the model proposed in this paper for the task of rolling bearing fault classification, we compared it with five other commonly used models on the ScD dataset. The number of epochs for each model was set to 200 to ensure convergence. The comparison results are shown in Table 3. The accuracy curve under the ScD dataset is shown in Figure 9.

As shown in Table 3, the validation accuracy and F1 of the proposed method are the highest for ScD, reaching 94.3% and 94.27%, respectively, while the computational complexity and number of parameters are kept at a low level. Rolling bearing fault signals have complex timing characteristics. CNN–based methods such as AlexNet, Vgg, and InceptionV3 cannot achieve a balance between computational accuracy and parameter quantity due to the difficulty in effectively modeling long–term dependencies. As a Transformer–based model, ViT is slightly inferior to the above CNN model in terms of accuracy. This may be due to its inability to fully extract key local features in the vibration signal. In contrast, Swin Transformer enhances local feature extraction capabilities, so the accuracy is significantly improved. However, its parameter count is higher than that of the method in this paper, and the computational effort is similar, but the accuracy still differs by more than 2%.

On the other hand, Figure 9 shows that the validation accuracy curve of the proposed method converges faster and achieves the highest accuracy, indicating its strong performance in the task of rolling bearing vibration signal fault classification. This is attributed to the EFE module extracting local feature information, and on this basis, S-D attention more flexibly focuses on key feature information, preventing excessive computational costs that could arise from expanding the receptive field of the Transformer, while establishing reliable long-distance dependencies.

The confusion matrix can intuitively display the model’s performance across various fault categories, while clearly displaying the fault categories that each model tends to confuse. Figure 10 shows the confusion matrix of each model on the ScD, which compares the preference of each classification model for each fault category.

The values on the diagonal of the confusion matrix represent the percentages of correct predictions for each category, while the values outside the diagonal represent the percentages of predictions for each category that have been misclassified. As shown in Figure 10, 0 to 9 represent various conditions of rolling bearings (see Table 1), the columns represent true labels, and the rows represent predicted labels. Compared to other classification methods, the proposed method for label 4 (IRP_ORP_H, inner race drilling + outer race drilling + holder) significantly improves the accuracy of fault type recognition, reaching 91%. In sharp contrast, the highest rate of other methods is only 79%. At the same time, except for ViT and the proposed algorithm, the accuracy of identifying fault types for category 5 (normal) is not high.

The TSNE (T–distributed stochastic neighbor embedding) visualization can map high–dimensional feature spaces to two–dimensional spaces for visualization. To visually observe the distribution of different types of fault samples in the feature space and reflect the quality of features extracted by different deep learning models. By comparing the TSNE graph, compare the effectiveness of each model in feature extraction and classification. The TSNE visualization on the ScD is shown in Figure 11.

As shown in Figure 11, the proposed method performs the best among all models, with the highest degree of separation between categories. There is almost no overlap between various clusters, indicating that the model is very effective in feature extraction and classification. At the same time, the category distribution is very compact and the model performs well in extracting and distinguishing features of different categories. This is consistent with the experimental results in Figure 10, indicating that it has significant advantages in rolling bearing fault diagnosis tasks. In comparison, the performance of the InceptionV3 model is relatively poor, with more overlap between categories and the worst feature separation effect. AlexNet, Vgg, ViT, and Swint_t also have their own advantages and disadvantages in feature extraction, but overall they are not as good as the proposed methods.

5.2. Ablation Experiment

Ablation experiments are often used to evaluate the impact of individual modules or components in a model on the overall performance. By gradually removing or adding different components, we can understand the role and contribution of each component. In Table 4, the accuracies of different model configurations on the ScD are compared as functions of training epochs.

As shown in Table 4, the initial accuracy of the basic Transformer model grew slowly and gradually stabilized after about 60 epochs. The final accuracy was the lowest among all configurations, at around 86%. The introduction of the SDT module significantly improved the accuracy of the model. The final accuracy was about 3% higher than that of the Transformer, while significantly reducing the FLOPs and parameter amount. After adding the EFE module, the accuracy of the model fluctuated during the initial training stage. However the EFE module’s overall improvement rate was fast, and the final accuracy reached 92.5, which is significantly higher than the model containing only SDT, indicating that the EFE module further improved the performance of the model. The final accuracy after introducing EMSA was similar to the SDT+EFE module configuration, but the added FLOPs and parameter quantities were only 0.04G and 0.002M, respectively. The complete model (SDT+EFE+EMSA) performed best, with an initial rapid increase in accuracy, stabilizing after about 40 epochs, and ultimately achieving the highest accuracy of 94.3%. This shows that these modules can work together to maximize model performance. It is worth noting that the introduction of both EFE and EMSA modules only increased the number of FLOPs and parameters by 0.17G and 0.296M, respectively, but achieved an accuracy improvement of 4.6%.

5.3. Generalization Performance

Deep learning models may only perform well on a single dataset and perform poorly on unseen data. In practical applications, models need to be able to process data from various sources and types. Generalization performance testing can simulate different scenarios, evaluate the performance of models in the face of diverse data, ensure that the model has broad applicability, and avoid degradation of model performance in practical applications. To this end, the classification accuracy of our model on three different datasets (see Section 4.1.2 for details) is shown in Table 5.

As shown in Table 5, the accuracies and F1 scores of the model are very high in the above datasets, but there are certain differences between the datasets. In CWRU and XJTU–SY, this model performs very well and has strong generalization ability on these two datasets. In PU, although the accuracy and F1 score of the model are slightly lower than the other two, the values are still very high, indicating that the model has good generalization ability and can adapt to various types of noise and faults in different datasets. To gain a deeper understanding of the classification performance and feature extraction of the models on the above three datasets, Figure 12 shows their confusion matrices and TSNE visualizations.

As shown in Figure 12, in the CWRU dataset, the prediction accuracy of most categories is close to 1, and only 0.5% of B014 is misclassified as IRO14. Different categories form clear clusters in this dataset, and the extracted features are very stable. In the PU dataset, the accuracy of some categories reaches 100%, but 23% of K101 is misclassified as KA04. There is an overlap between K101 and KA04, and the model does not perform well in feature separation in this category. In the XJTU–SY dataset, the prediction accuracy of each category is very high, with even the lowest Bearing1_3 and Bearing3_4 having 98% and 99%, respectively. The separation degree of clusters in different categories is high, with almost no overlap, and each cluster is very compact. The feature extraction effect of the model on this dataset is very good.

6. Conclusions and Future Work

The proposed method is a multi–attention fusion Shift–Deformable Transformer (S–DT) network model for rolling bearing fault diagnosis. This method first uses the Morlet function as the wavelet basis function for wavelet decomposition to obtain the time–frequency maps of the fault signals, and then it uses this model for end–to–end fault classification. The characteristics of the model involve the construction and introduction of EMSA and DA attention based on the Swin Transformer. These are, respectively, used to enhance local feature information and optimize the receptive field of transition window attention. The former integrates contextual information of different scales enabling the constructed hole residual block to focus more on high–level features, while the latter adjusts the receptive area by learning adaptive offsets, which helps achieve accurate feature extraction and reduces computational redundancy.

An experimental platform was built to collect data to obtain a real and reliable rolling bearing fault vibration signal. Comparative experiments were conducted on this dataset. The results show that compared with other signal classification network models, the accuracy of the network proposed in this paper reached 94.3%. This shows that it has excellent classification performance and feature extraction capabilities. In addition, the generalization performance was validated on three other publicly available datasets, and the results showed that the model still maintained high classification accuracy, indicating its strong generalization ability. Agricultural machinery is a large complex assembly composed of various mechanical parts, with various types of component failures, including rolling bearings, sliding bearings, gear failures, bolts, etc. The proposed method provides technical references for these types of faults.

Although the network in this paper achieved satisfactory results, there are still some shortcomings that need further research and solutions. The sample size of the time–frequency map data used for training this model is sufficient, and the number of training samples for each class is also consistent. In future work, it is necessary to further verify and explore the network performance of this paper when the sample size is low or inconsistent.

Author Contributions

Conceptualization, M.L. and X.L.; methodology, M.L.; software, M.L.; validation, M.L. and X.L.; resources, S.L.; data curation, C.L.; writing—original draft preparation, B.L. and M.L.; writing—review and editing, B.L., X.L. and M.L.; visualization, B.L.; supervision, S.L. and C.L. All authors have read and agreed to the published version of the manuscript.

Data Availability Statement

The CWRU dataset is available at https://engineering.case.edu/bearingdatacenter/welcome, accessed on 21 November 2023. The PU dataset is available at https://mb.uni-paderborn.de/kat/forschung/kat-datacenter/bearing-datacenter, accessed on 21 November 2023. The XJTU-SY dataset is available at http://biaowang.tech/xjtu-sy-bearing-datasets, accessed on 19 March 2024. The self-collected dataset is available to readers by contacting the corresponding author.

Acknowledgments

All authors thank the Mechanical and Control Engineering 219 Lab and 212 Lab of Baicheng Normal University.

Conflicts of Interest

The authors declare no conflicts of interest.

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figures and Tables

Figure 1. Schematic diagram of the multi-head attention operation. h represents the number of heads.

Figure 2. The structure of the proposed method.

Figure 3. Efficient feature extraction block.

Figure 4. The structure of EMSA.

Figure 5. Deformable attention structure. (a) Details of DA; (b) comparison of attentional receptive fields (ViT, Swin Transformer, and DA).

Figure 6. The structure of the S–D Transformer.

Figure 7. Experimental setup for the self–collected bearing fault dataset.

Figure 8. Experimental setups for the CWRU, PU, and XJTU–SY bearing datasets.

Figure 9. Accuracy curves of each method on the ScD.

Figure 10. Confusion matrix of various methods on the ScD.

Figure 11. Comparison of TSNE visualizations of various methods on the ScD.

Figure 12. TSNE visualization of the model in this paper on various datasets.

Table 1

Rolling bearing fault information and labels for each dataset.

Detail	ScD	CWRU	PU	XJTU–SY	Labels
Pattern	Holder	B007	K001	Bearing1_3	0
	IRP	B014	K005	Bearing1_4	1
	IRP_ORG	B021	KA01	Bearing1_5	2
	IRP_ORP	IRO07	KA03	Bearing2_1	3
	IRP_ORP_H	IRO14	KA04	Bearing2_2	4
	Normal	IRO21	KB23	Bearing2_3	5
	ORG	Normal	KB27	Bearing3_1	6
	ORG_H	OR007	K101	Bearing3_2	7
	ORP	OR014	KI03	Bearing3_3	8
	ORP_ORG	OR021	KI07	Bearing3_4	9
Damage	EDM	EDM	EDM,	Unknown
			Drilling,
			Engraver
Location	OR, IR, Holder, None	OR, IR, Ball	OR, IR, None	OR, IR, Holder
Speed	1600 rpm,	Unknown	1500 rpm,	2100 rpm,
	2000 rpm,		3000 rpm	2250 rpm,
	2400 rpm			2400 rpm
Load	None	2HP	N09_M07_F10	None

Table 2

Model parameter details.

Stage	Layer	Output Size[Batch, Channels, Width, Height]	(Kernel Size, Stride, Padding)or Attention Heads
Input		[4, 3, 224, 224]	/
EFE block	DWConv	[1, 96, 112, 112]	(7, 2, 3)
	MaxPool	[1, 96, 56, 56]	(2, 2)
	DConv block	[1, 96, 56, 56]	(3, 1, 2)
	DConv block	[1, 96, 56, 56]	(3, 1, 2)
EMSA	/	[1, 96, 56, 56]	/
S-D Transformer	Stage 1	[1, 96, 56, 56]	3
	Stage 2	[1, 192, 28, 28]	6
	Stage 3	[1, 384, 14, 14]	12
	Stage 4	[1, 768, 7, 7]	24
Classifier	Fully Connected	[1, 10]	/

Table 3

Performance comparisons of various network models.

Network Model	Accuracy (%)	F1 (%)	FLOPs (G)	Params (M)
AlexNet [34]	90	90	0.7	57
Vgg [35]	90.1	89.84	7.6	128
InceptionV3 [36]	82.95	83.02	2.85	21.8
ViT [8]	86.65	86.38	5.3	28.9
Swin_t [10]	92.05	92.0	4.36	27.5
Ours	94.3	94.27	4.37	26.5

Table 4

Ablation experiments on the ScD.

SDA	EFE	EMSA	Accuracy (%)	FLOPs (G)	Params (M)	[Image omitted. Please see PDF.]
-	-	-	86.65	5.3	28.9
✓	-	-	89.7 (+3.05)	4.2	26.3
✓	✓	-	92.5 (+5.35)	4.35	26.594
✓	-	✓	93.15 (+6.5)	4.24	26.38
✓	✓	✓	94.3 (+7.65)	4.37	26.596

Table 5

Comparison of classification accuracy of this model on different datasets.

Datasets	Accuracy (%)	F1 (%)
CWRU	99.95	99.95
PU	93.55	93.39
XJTU-SY	99.4	99.4

References

1. Xie, F.; Sun, E.; Wang, L.; Wang, G.; Xiao, Q. Rolling Bearing Fault Diagnosis in Agricultural Machinery Based on Multi-Source Locally Adaptive Graph Convolution. Agriculture; 2024; 14, 1333. [DOI: https://dx.doi.org/10.3390/agriculture14081333]

2. Li, J.; Bao, Y.; Liu, W.; Ji, P.; Wang, L.; Wang, Z. Twins transformer: Cross-attention based two-branch transformer network for rotating bearing fault diagnosis. Measurement; 2023; 223, 113687. [DOI: https://dx.doi.org/10.1016/j.measurement.2023.113687]

3. Jia, L.; Chow, T.W.S.; Yuan, Y. GTFE-Net: A Gramian Time Frequency Enhancement CNN for bearing fault diagnosis. Eng. Appl. Artif. Intell.; 2023; 119, 105794. [DOI: https://dx.doi.org/10.1016/j.engappai.2022.105794]

4. Fu, W.; Jiang, X.; Li, B.; Tan, C.; Chen, B.; Chen, X. Rolling bearing fault diagnosis based on 2D time-frequency images and data augmentation technique. Meas. Sci. Technol.; 2023; 34, 045005. [DOI: https://dx.doi.org/10.1088/1361-6501/acabdb]

5. Ji, C.; Zhang, C.; Suo, L.; Liu, Q.; Peng, T. Swarm intelligence based deep learning model via improved whale optimization algorithm and Bi-directional long short-term memory for fault diagnosis of chemical processes. ISA Trans.; 2024; 147, pp. 227-238. [DOI: https://dx.doi.org/10.1016/j.isatra.2024.02.014] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/38443273]

6. Wang, J.; Mo, Z.; Zhang, H.; Miao, Q. A Deep Learning Method for Bearing Fault Diagnosis Based on Time-Frequency Image. IEEE Access; 2019; 7, pp. 42373-42383. [DOI: https://dx.doi.org/10.1109/ACCESS.2019.2907131]

7. Liu, Z.; Sun, W.; Chang, S.; Zhang, K.; Ba, Y.; Jiang, R. Corn Harvester Bearing Fault Diagnosis Based on ABC-VMD and Optimized EfficientNet. Entropy; 2023; 25, 1273. [DOI: https://dx.doi.org/10.3390/e25091273] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/37761571]

8. Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S. et al. An Image is Worth 16 × 16 Words: Transformers for Image Recognition at Scale. arXiv; 2021; arXiv: 2010.11929

9. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, L.; Polosukhin, I. Attention Is All You Need. arXiv; 2023; arXiv: 1706.03762

10. Liu, Z.; Lin, Y.; Cao, Y.; Hu, H.; Wei, Y.; Zhang, Z.; Lin, S.; Guo, B. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV); Montreal, BC, Canada, 11–17 October 2021; pp. 9992-10002. [DOI: https://dx.doi.org/10.1109/ICCV48922.2021.00986]

11. Chen, Z.; Chen, J.; Liu, S.; Feng, Y.; He, S.; Xu, E. Multi-channel Calibrated Transformer with Shifted Windows for few-shot fault diagnosis under sharp speed variation. ISA Trans.; 2022; 131, pp. 501-515. [DOI: https://dx.doi.org/10.1016/j.isatra.2022.04.043] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/35551816]

12. Wang, H.; Wang, J.; Sui, Q.; Zhang, F.; Li, Y.; Jiang, M.; Paitekul, P. Intelligent Fault Diagnosis Method of Rolling Bearings Based on Transfer Residual Swin Transformer with Shifted Windows. Struct. Durab. Health Monit.; 2024; 18, pp. 91-110. [DOI: https://dx.doi.org/10.32604/sdhm.2023.041522]

13. Xie, F.; Wang, Y.; Wang, G.; Sun, E.; Fan, Q.; Song, M. Fault Diagnosis of Rolling Bearings in Agricultural Machines Using SVD-EDS-GST and ResViT. Agriculture; 2024; 14, 1286. [DOI: https://dx.doi.org/10.3390/agriculture14081286]

14. Guo, M.; Zeng, X.; Chen, D.Y.; Yang, N. Deep-Learning-Based Earth Fault Detection Using Continuous Wavelet Transform and Convolutional Neural Network in Resonant Grounding Distribution Systems. IEEE Sens. J.; 2018; 18, pp. 1291-1300. [DOI: https://dx.doi.org/10.1109/JSEN.2017.2776238]

15. Gou, L.; Li, H.; Zheng, H.; Li, H.; Pei, X. Aeroengine Control System Sensor Fault Diagnosis Based on CWT and CNN. Math. Probl. Eng.; 2020; 2020, 5357146. [DOI: https://dx.doi.org/10.1155/2020/5357146]

16. Pan, X.; Ye, T.; Xia, Z.; Song, S.; Huang, G. Slide-Transformer: Hierarchical Vision Transformer with Local Self-Attention. Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); Vancouver, BC, Canada, 17–24 June 2023; pp. 2082-2091. [DOI: https://dx.doi.org/10.1109/CVPR52729.2023.00207]

17. Zhang, J.; Li, X.; Tian, J.; Luo, H.; Yin, S. An integrated multi-head dual sparse self-attention network for remaining useful life prediction. Reliab. Eng. Syst. Saf.; 2023; 233, 109096. [DOI: https://dx.doi.org/10.1016/j.ress.2023.109096]

18. Cordonnier, J.B.; Loukas, A.; Jaggi, M. Multi-Head Attention: Collaborate Instead of Concatenate. arXiv; 2020; arXiv: 2006.16362

19. Han, Y.; Li, C.; Li, T.; Zhang, F. A Multi-task Bearing Fault Diagnosis Method Based on Multi-scale Residual CNN with Dual Attention. Proceedings of the 2023 18th International Conference on Intelligent Systems and Knowledge Engineering (ISKE); Fuzhou, China, 17–19 November 2023; pp. 374-381. [DOI: https://dx.doi.org/10.1109/ISKE60036.2023.10481273]

20. Chen, H.; Meng, W.; Li, Y.; Xiong, Q. An anti-noise fault diagnosis approach for rolling bearings based on multiscale CNN-LSTM and a deep residual learning model. Meas. Sci. Technol.; 2023; 34, 045013. [DOI: https://dx.doi.org/10.1088/1361-6501/acb074]

21. Guo, Y.; Mao, J.; Zhao, M. Rolling Bearing Fault Diagnosis Method Based on Attention CNN and BiLSTM Network. Neural Process. Lett.; 2022; 55, pp. 3377-3410. [DOI: https://dx.doi.org/10.1007/s11063-022-11013-2]

22. Chollet, F. Xception: Deep Learning with Depthwise Separable Convolutions. Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); Honolulu, HI, USA, 21–26 July 2017; pp. 1800-1807. [DOI: https://dx.doi.org/10.1109/CVPR.2017.195]

23. Yu, F.; Koltun, V. Multi-Scale Context Aggregation by Dilated Convolutions. arXiv; 2015; arXiv: 1511.07122

24. Gao, R. Rethinking Dilated Convolution for Real-time Semantic Segmentation. Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW); Vancouver, BC, Canada, 17–24 June 2023; pp. 4675-4684. [DOI: https://dx.doi.org/10.1109/CVPRW59228.2023.00493]

25. Li, Y.; Lu, J.; Chen, H.; Wu, X.; Chen, X. Dilated Convolutional Transformer for High-Quality Image Deraining. Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW); Vancouver, BC, Canada, 17–24 June 2023; pp. 4199-4207. [DOI: https://dx.doi.org/10.1109/CVPRW59228.2023.00442]

26. Ouyang, D.; He, S.; Zhan, J.; Guo, H.; Huang, Z.; Luo, M.; Zhang, G.L. Efficient Multi-Scale Attention Module with Cross-Spatial Learning. Proceedings of the ICASSP—IEEE International Conference on Acoustics, Speech and Processing; Signal, Rhodes Island, Greece, 4–10 June 2023; pp. 1-5. [DOI: https://dx.doi.org/10.1109/ICASSP49357.2023.10096516]

27. Alabdulmohsin, I.; Zhai, X.; Kolesnikov, A.; Beyer, L. Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Design. arXiv; 2024; arXiv: 2305.13035

28. Bao, F.; Nie, S.; Xue, K.; Cao, Y.; Li, C.; Su, H.; Zhu, J. All are Worth Words: A ViT Backbone for Diffusion Models. Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); Vancouver, BC, Canada, 17–24 June 2023; pp. 22669-22679. [DOI: https://dx.doi.org/10.1109/CVPR52729.2023.02171]

29. Zhao, W.; Tang, J.; Han, Y.; Song, Y.; Wang, K.; Huang, G.; Wang, F.; You, Y. Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptation. arXiv; 2024; arXiv: 2403.11808

30. Xia, Z.; Pan, X.; Song, S.; Li, L.E.; Huang, G. Vision Transformer with Deformable Attention. Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); New Orleans, LA, USA, 18–24 June 2022; pp. 4784-4793. [DOI: https://dx.doi.org/10.1109/CVPR52688.2022.00475]

31. Smith, W.A.; Randall, R.B. Rolling element bearing diagnostics using the Case Western Reserve University data: A benchmark study. Mech. Syst. Signal Process.; 2015; 64–65, pp. 100-131. [DOI: https://dx.doi.org/10.1016/j.ymssp.2015.04.021]

32. Lessmeier, C.; Kimotho, J.K.; Zimmer, D.; Sextro, W. Condition Monitoring of Bearing Damage in Electromechanical Drive Systems by Using Motor Current Signals of Electric Motors: A Benchmark Data Set for Data-Driven Classification. PHM Soc. Eur. Conf.; 2016; 3, [DOI: https://dx.doi.org/10.36001/phme.2016.v3i1.1577]

33. Yaguo, L.; Han, T.; Biao, W.; Li, N.; Tao, Y.; Jun, Y. XJTU-SY Rolling Element Bearing Accelerated Life Test Datasets: A Tutorial. J. Mech. Eng.; 2019; 55, pp. 1-6. [DOI: https://dx.doi.org/10.3901/JME.2019.16.001]

34. Krizhevsky, A.; Sutskever, I.; Hinton, G.E. ImageNet classification with deep convolutional neural networks. Commun. ACM; 2017; 60, pp. 84-90. [DOI: https://dx.doi.org/10.1145/3065386]

35. Simonyan, K.; Zisserman, A. Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv; 2014; arXiv: 1409.1556

36. Szegedy, C.; Vanhoucke, V.; Ioffe, S.; Shlens, J.; Wojna, Z. Rethinking the Inception Architecture for Computer Vision. Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR); Las Vegas, NV, USA, 27–30 June 2016; pp. 2818-2826. [DOI: https://dx.doi.org/10.1109/CVPR.2016.308]