Multiclass Evaluation of Vision Transformers for Industrial Welding Defect Detection

Abstract

Automating industrial processes, particularly quality inspection, is a key objective in manufacturing. While welding tasks are frequently automated, inspection processes remain largely manual. Advances in computer vision and AI, especially ViTs, now enable more effective defect detection and classification, offering opportunities to automate these workflows. This study evaluates ViTs for identifying defects in aluminum welding using the Aluminum 5083 TIG dataset. The analysis spans binary classification (detecting defects) and multiclass categorization (Good Weld, Burn Through, Contamination, Lack of Fusion, Misalignment, and Lack of Penetration). ViTs achieved 98% to 99% accuracy across both tasks, significantly outperforming prior models such as dense and CNNs, which struggled to surpass 80% accuracy in binary and 70% in multiclass tasks. These results, achieved with datasets of 2400 to 8000 images, highlight ViTs’ efficiency even with limited data. The findings underline the potential of ViTs to enhance manufacturing inspection processes by enabling faster, more reliable, and cost-effective automated solutions, reducing reliance on manual inspection methods.

1. Introduction

The manufacturing industry has experienced a significant transformation in recent years, particularly since 2017, as numerous authors have underscored [1,2,3,4] the growing impact of Artificial Intelligence (AI) in the sector. This shift is largely due to the release and public availability of several powerful AI development frameworks, such as TensorFlow, Keras, and PyTorch. These frameworks have accelerated advancements not only in manufacturing, but also across various scientific fields by simplifying AI development and enabling broader experimentation and application. Previous studies [1] have emphasized the transformative potential of AI in manufacturing, particularly in areas such as predictive maintenance, production optimization, and defect detection within industrial applications. Recent discussions in [4] highlighted predictive maintenance, which enables companies to schedule maintenance before problems arise, reducing unexpected downtime and improving machine reliability. Another area, degradation estimation, helps predict when equipment may fail, allowing companies to plan replacements during scheduled downtime or periods of lower production demand. Furthermore, AI-driven product inspection, the primary focus of this paper, aims to improve quality control by automating the detection of defects and inconsistencies in products, addressing the inefficiencies and limitations of manual inspection processes. A common goal among many companies is to achieve fully automated production pipelines. Although production processes are increasingly automated, inspection workflows remain largely manual, with studies indicating that manual inspection accounts for more than 50% of quality control efforts in some industries. For example, the PHMSA Office of Pipeline Safety (OPS) has reported that OPS workers spend approximately 55% of their time ensuring the safety and consistency of the production pipelines [5]. This inspection process is time-intensive and could benefit significantly from AI-driven automation, potentially reducing manual inspection processes to close to 0% in an ideal scenario. Such automation would allow workers to focus on more complex and high-impact tasks, enhancing both productivity and operational efficiency. Manufacturing companies have long sought to automate both production and inspection processes [6,7,8]. For example, automation can enhance productivity, quality control, and safety measures in smart factories, such as fine-grained activity classification in assembly based on multi-visual modalities, as presented by Chen in [9]. Although automation of product manufacturing is outside the scope of this study, inspection processes [10,11]—particularly for welding defect detection—are highly relevant. Early examples of automation in this area include neurofuzzy architectures introduced in 1997 [12] and more physics-based models in 2012, such as those used for specular weld pool surface characterization [13]. However, most of these approaches provided only partial solutions and struggled to generalize. With advancements in technology, such as increasingly powerful CPUs and GPUs, and the availability of frameworks like TensorFlow, Keras, and PyTorch, a paradigm shift toward deep learning became inevitable. For example, in 2019, a group led by Daniel Bacioiu explored various neural network architectures, ranging from simple dense networks to more complex combinations of dense and convolutional layers, achieving promising results. Although most approaches could detect the presence of a defect, the advantage of deep learning models was their ability to accurately classify specific defect types. This approach resulted in the detection of weld defects with an accuracy of approximately 70% [14]. This research opened new possibilities for testing diverse architectures, including the use of pre-trained models and transformers. Introduced in 2017, the transformer architecture revolutionized deep learning by enabling the modeling of complex relationships in data. Initially applied to Natural Language Processing (NLP), transformers have since been adapted for computer vision tasks, offering a promising avenue for defect detection. Its standout feature, the attention mechanism, enables the transformer to understand complex relationships within data effectively. Initially designed for sequential data processing—specifically text—the attention mechanism facilitates the creation of contextual embeddings. These embeddings encode the relationships between words in a phrase, helping the model differentiate meanings based on the surrounding context. For instance, the word “apple” may refer to a fruit in some contexts, while in others, it could denote the technology company. This ability to capture nuanced meanings is one of the main innovations of transformers. As highlighted in the original paper [15], this architecture builds upon the encoder-decoder framework of recurrent neural networks. Although recurrent networks could handle basic NLP tasks, such as machine translation, they often struggled with more complex challenges. The emergence of the transformer generated significant interest among researchers and companies alike, leading to its application in a variety of sophisticated NLP tasks, including text classification, sentiment analysis, Natural Language Understanding (NLU), text summarization, text generation, and question answering [16]. This discovery also led to the formation of communities, with Hugging Face [17] emerging as one of the most notable examples. Initially focused on NLP, Hugging Face has expanded its reach into other AI subfields, such as computer vision. However, since transformers were originally designed for sequential data, it was recognized that images, which are inherently non-sequential, posed a challenge. In 2020, researchers at Google tackled this issue by introducing a method called image patching, which converts image data into embeddings in a manner similar to text processing [18]. This approach involves dividing images into smaller segments, or “patches”, which are processed independently by a neural network composed of dense layers, convolutional layers, or a combination of both. Each patch is transformed into an embedding, and the embeddings are then concatenated in sequence, creating a structure that allows the use of the attention mechanism to generate contextual embeddings. Unlike text contextual embeddings, image contextual embeddings help the model understand spatial relationships within the image, identifying which areas are most relevant for accurate predictions. Vision Transformer (ViT) architectures have shown significant potential across various fields, often surpassing traditional state-of-the-art models. For instance, in real-time object detection, where models like YOLO previously dominated [19], new models based on transformer architectures, such as Detection Transformer (DETR) [20], have demonstrated remarkable performance. More recently, Real-Time Detection Transformer (RT-DETR) [21] was introduced, claiming to outperform YOLO and other object detection models. These examples illustrate only a fraction of the applications and variations that have evolved from the ViT architecture, which excels at identifying the most relevant regions within an image for accurate predictions. Building on these advancements, our study seeks to surpass the current state-of-the-art in welding defect detection by leveraging ViTs for multiclass classification and exploring their robustness with reduced dataset sizes. We will focus on enhancing accuracy and reliability by leveraging the pre-trained nature of these models to achieve better results. In addition, we plan to assess how many images are necessary to obtain optimal outcomes, highlighting that many existing models can benefit from this approach. This paper presents the following contributions:

Adapting and implementing ViTs for welding defect detection, including both binary classification to determine defect presence and multiclass classification to identify specific defect types, addressing challenges unique to industrial applications.

The proposed model represents a substantial improvement over previous implementations, achieving over 90% accuracy in a multiclass classification setting.

Investigating the impact of dataset size reduction and balancing techniques, identifying minimal data requirements for reliable performance.

The structure of this paper is organized as follows: Section 1 outlines the fundamental concepts and recent advancements in ViTs and their relevance in the detection of industrial defects. It also provides a review of the literature on defect detection techniques, with a focus on machine learning, deep learning, and transformer-based approaches, particularly ViTs. Section 2 describes the methodology, including data collection, multiclass labeling, preprocessing steps, and model training processes. Section 3 presents the experiments conducted, detailing the evaluation metrics and performance results of the models in multiclass defect classification. A discussion compares these results with established benchmarks and previous studies in the field. Finally, Section 4 summarizes the findings, highlights the contributions, and suggests directions for future work in industrial defect detection.

2. Materials and Methods

This section provides a comprehensive overview of the dataset, methodology, models, and evaluation metrics used in this study.

2.1. Dataset

This study uses the public Al5083 Tungsten Inert Gas (TIG) weld defects dataset, created by Bacioiu et al. in 2019 [14]. This dataset was selected for its relevance to industrial defect detection, particularly for Al5083, an alloy widely used in the aerospace and automotive sectors due to its strength and corrosion resistance. It includes 33,254 weld images spanning six defect categories, making it well-suited for evaluating classification models. Although various studies have used different datasets for the detection of welding defects [22,23,24,25,26,27,28,29], the Al5083 dataset has emerged as a prominent choice due to its relevance in the advancement of detection techniques. Recent research [30,31,32,33], including the study by Wang et al. (2024) [34], highlights its significance in improving the accuracy of defect detection methods. The Al5083 dataset includes TIG weld images on 5083 aluminum, captured with an HDR camera focused on the weld pool area during the welding process. The dataset has an imbalanced class distribution, which is evident in the default distribution shown in Table 1.

Figure 1 presents image samples from the dataset, along with their respective classes: Good Weld, Burn Through, Contamination, Lack of Fusion, Misalignment, and Lack of Penetration.

2.2. Hugging Face

The Hugging Face platform was selected to store the dataset and enable collaborative access due to its robust tools to manage datasets and pre-trained models. Hugging Face’s Transformers library simplified the fine-tuning process for ViTs, streamlining model development and experimentation [35]. Hugging Face is a data science platform that aims to promote collaboration in machine learning research and development [17]. It provides a space for sharing datasets, AI models, and tools, allowing users to upload, share, and access a vast collection of pre-trained models. These models can be fine-tuned for specific tasks or serve as a basis for training new ones. We utilize the Transformers and Datasets libraries from Hugging Face, which simplify the process of deploying models and managing datasets in multiple domains.

2.3. ViT

In this study, we use ViTs for defect detection and classification due to their ability to process images as patch sequences, capturing long-range dependencies more effectively than traditional Convolutional Neural Networks (CNNs) [36,37,38]. This global context-awareness makes ViTs particularly suited for identifying subtle patterns in welding defects.

Previous studies have used CNNs, achieving favorable results in the TIG Aluminum 5083 dataset [14,34]. However, we proposed using ViTs to explore whether this approach could yield improvements in metrics. For our work, we have fine-tuned a pre-trained ViT model, named BEiT, for our classification tasks. We will provide a detailed explanation of the BEiT model in Section 2.4. ViT is an architecture that is used for image recognition tasks such as image classification. It employs a Transformer-like architecture that processes images by dividing them into smaller patches. These patches are treated as sequences, allowing the model to learn spatial relationships across the image, similar to how Transformers process sequences of words in NLP tasks [18].

An example of the ViT architecture is shown in Figure 2.

2.4. BEiT

In this work, we propose using a ViT model as an alternative to CNNs. We selected the BEiT model, a popular pre-trained transformer-based model commonly used for image classification. BEiT is a self-supervised ViT model pre-trained on ImageNet-21k at 224 × 224 resolution and can be fine-tuned on specific image classification tasks [39]. We chose BEiT for this study because of its outstanding performance in image classification tasks [38]. Unlike traditional CNNs, BEiT captures global dependencies and contextual information more effectively, which makes it particularly well suited for image defect detection [36,37]. Additionally, its pre-trained architecture significantly enhances its ability to generalize across various image-related tasks.

2.5. Classification Tasks

In this work, we approach the problem of detecting welding defects by defining two classification tasks, defining both binary and multiclass classification tasks for detecting welding defects. The binary classification task distinguishes between good and bad welding, while the multiclass classification task categorizes different types of welding defects. Previous studies have used both approaches, highlighting their relevance in real-world quality control applications. The classification tasks are defined as follows:

Binary Classification Task: This task simplifies the original six classes into two categories: good or defective welding.

Multiclass Classification Task: This task involves classifying the weld into its original six categories, preserving the specific types of defects.

Figure 3 illustrates the workflow for each classification task, with different classes assigned to each. In the binary task, the classes are Good Weld and Defective, while in the multiclass task, there are six classes: Good Weld, Burn Through, Contamination, Lack of Fusion, Misalignment, and Lack of Penetration. The figure shows that for each classification task, multiple images may be involved. Each image is processed through its corresponding model, ultimately producing a predicted class.

2.6. Design of Experiments

We defined five experiments to evaluate the performance of ViTs across varying dataset sizes and task complexities, from binary to multiclass classification. This setup aims to assess the model’s robustness, scalability, and ability to generalize across diverse scenarios. For each experiment, modified versions of the original dataset (shown in Table 1) were created to evaluate the performance of the model with varying data volumes. The goal of the experiments was to evaluate the impact of reducing the number of images to 3000 or less on model performance, comparing binary classification and multiclass classification to assess how effectively the model could be trained with this reduced dataset. Each experiment maintained balanced class proportions to prevent bias in performance metrics due to data imbalance. The purpose of the first two experiments was to tackle the simpler, binary version of the problem first, followed by the more complex, multiclass version, allowing us to compare our results with the state-of-the-art models for each version of the problem.

Table 2 presents the data distribution in all experiments. Experiments 1 and 2 focused on binary classification, while Experiments 3 to 5 addressed multiclass classification. Experiment 2 used 25% of Experiment 1’s data, Experiment 4 used approximately 57% of Experiment 3’s data, and Experiment 5 used around 28% of Experiment 3’s data.

2.7. Experimental Setup and Hardware Configuration

The hardware configuration for training, evaluation, and testing comprised an Intel(R) Xeon(R) CPU running at 2.00 GHz, identified under the GenuineIntel vendor. The processor belongs to CPU family 6, model 85, and features 2 cores per socket, with 2 threads per core, totaling 4 logical cores. The system was equipped with 30 GB of RAM and operated within a KVM hypervisor environment. The computer architecture was x86_64 with a 64-bit CPU mode, running Ubuntu 22.04.4 LTS Linux operating system. The experiments were run using Kaggle notebooks, Python 3.10, PyTorch 2.4.0, and the transformers library version 4.45.1.

2.8. Training Pipeline

Once the list of experiments was defined, the process of running each experiment involved training five classifiers per experiment. Each classifier was trained using a stratified cross-validation approach, where datasets were randomly partitioned into training and validation subsets to ensure balanced representation across classes. This method reduced the variance in performance metrics and improved reliability. At the end of the training, all five classifiers in the experiment were evaluated and their metrics were averaged. Each classifier was trained for five epochs, based on preliminary experiments showing that model performance stabilized after this number of iterations. This choice balanced computational efficiency and performance consistency. This approach, implemented to perform cross-validation, involved training five classifiers per experiment to reduce variance and improve the reliability of the model. The steps remained consistent for all experiments and proceeded as follows:

Select experiment: Define the parameters and settings for each experiment, such as the task, the number of training samples and the number of folds F, which corresponds to the number of runs per experiment.

Create a dataset for each fold: For each fold, create a dataset by randomly selecting N images for training, with the remaining images allocated for validation.

Model training: Train a model for each fold and validate it using the validation dataset.

Model testing: Evaluate each model in the test dataset, ensuring that the same evaluation process is applied in all experiments and folds.

As shown in Figure 4, the training pipeline consists of defining experiments, creating balanced class datasets, training classifiers, and evaluating their performance. The pipeline ensures consistency between experiments and facilitates comparison between models.

2.9. Model Evaluation Metrics

These metrics were selected because they comprehensively capture different aspects of classifier performance, including general precision, sensitivity to false positives (precision), ability to detect true positives (recall), and balanced evaluation (F1 score), as they provide a comprehensive assessment of their effectiveness [14,29]. Metrics such as Receiver Operating Characteristic Area Under the Curve (ROC-UAC) were not used due to their limited applicability in imbalanced multiclass scenarios.

In the context of defect detection, particularly for welding, each evaluation metric provides unique insights into the model’s performance:

Accuracy: Measures overall correctness but can be misleading with imbalanced data, such as rare welding defects.

Precision: Important to minimize false positives, avoiding unnecessary repairs or actions.

Recall: Ensures most defects are identified, reducing the risk of missing critical issues.

F1-score: Balances precision and recall, making it crucial for detecting welding defects, where both false positives and false negatives are costly.

Before defining the metrics, we should define the common terms. The key terms are defined as follows:

True Positives ($TP$): These represent the number of instances where the model correctly predicts the positive class.

True Negatives ($TN$): These refer to instances where the model accurately predicts the negative class.

False Positives ($FP$): These indicate instances where the model incorrectly predicts the positive class for an actual negative instance.

False Negatives ($FN$): These occur when the model predicts the negative class for an instance that is actually positive.

2.9.1. Accuracy

Accuracy measures the proportion of correct predictions made by a model of all predictions [14]. It is calculated as:

(1)$\mathrm{Accuracy}={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$

2.9.2. Precision

Precision indicates the model’s ability to predict positive outcomes [29,40]. It is calculated as the ratio of true positives to the sum of true and false positives:

(2)$\mathrm{Precision}={\displaystyle \frac{TP}{TP+FP}}$

2.9.3. Recall

Recall indicates the model’s ability to correctly identify all relevant instances [29,40]. It is the ratio of true positives to the total of true positives and false negatives:

(3)$\mathrm{Recall}={\displaystyle \frac{TP}{TP+FN}}$

2.9.4. F1-Score

The F1-score balances precision and recall as their harmonic mean, ranging from 0 to 1, where 1 indicates perfect performance [29,40]. It is especially useful when both precision and recall are important:

(4)$\mathrm{F}1\text{-}\mathrm{Score}=2\times {\displaystyle \frac{\mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}}$

3. Results & Discussion

After defining the selected model, detailing the training process, outlining the experimental configurations, and specifying the evaluation metrics, we will conduct a thorough analysis of the results, organized by experiment and fold. This section aims to explore how the model performs after applying proper balancing and data undersampling techniques, both for binary and multiclass tasks. In addition, it will highlight the minimal amount of data required to achieve strong performance, demonstrating the efficiency of the ViTs.

3.1. Training Results

Before analyzing the individual evaluation results, it is essential to review the model’s performance during training. Table 3 presents the accuracy values obtained on the validation set for each experiment. As previously mentioned, each experiment was repeated five times to ensure consistency, totaling 25 runs across all experiments. Overall, the models showed strong performance, with an average accuracy of 97% across all runs. For the binary classification experiments (Exp. 1 and Exp. 2), the models achieved an average accuracy of 99% and 98%, respectively. Notably, despite a significant reduction in dataset size (from 12,000 to 3000 samples), accuracy dropped by only 1%, highlighting the model’s robustness. Similarly, in the multiclass classification experiments (Exp. 3 to Exp. 5), the models achieved average accuracies of 98%, 97%, and 96%, respectively. Even with only 2400 images (400 per class), Experiment 5 achieved state-of-the-art results, maintaining an accuracy above 95%. Additionally, across the 25 runs, the epoch loss in binary classification experiments ranged from 0.015 to 0.030, while in multiclass experiments, it ranged from 0.05 to 0.09. These results present the model’s stability across different dataset sizes, showing that high performance can still be achieved with reduced data availability.

3.2. Binary Models

The first two experiments focused on a binary classification task aimed at determining whether an image depicted a well-executed weld or a defective one. The primary difference between these experiments was the number of images used for training. As mentioned before in the design of experiments section, Experiment 1 used 12,000 images for training. In Experiment 2, however, we aimed to assess how much we could reduce the dataset size while maintaining model performance, so in this case, only 3000 images were used for training.

3.2.1. Experiment 1

In this first experiment focused on binary classification, we used a balanced dataset of 12,000 images, 6000 showing good welds and 6000 showing defective ones. This setup was chosen to establish a baseline for performance comparison, reflecting the importance of balanced datasets in binary classification tasks, as remarked in previous studies. As described in the design of experiments section, each of the five runs generated a new dataset by automatically selecting 6000 good weld images and 6000 defective weld images for training, along with 2000 images for validation, while maintaining a constant set of 6000 images for evaluation. As seen in Figure 5, we observed minimal variability in metrics across runs, consistently ranging between 98% and 99%. This highlights the robustness of ViTs when applied to a well-curated, high-quality dataset. The balanced distribution of classes ensures impartial learning, allowing the model to detect both good and defective welds equally well. Furthermore, effective model tuning enhances feature generalization across runs, further contributing to the stability and reliability of the results.

In Figure 6, which presents the confusion matrix for each run of the experiment on the evaluation dataset, we observe that most models generalize well, exhibiting a similar number of errors, at approximately 60 across runs. This consistency indicates that, with appropriate balancing techniques, favorable results can be achieved even when the dataset varies, as the error count remains relatively constant.

3.2.2. Experiment 2

In this second experiment, we explore the feasibility of reducing the size of a dataset for practical applications, such as early-stage development or resource-constrained environments. Focusing on binary classification, we reduced the training set to 25% of its original size, using only 3000 images for training in each run. As in the previous experiment, each dataset was unique, while the remaining 9000 images were reserved for validation and evaluation. Despite the significant reduction in training data, performance decreased by only 1 to 2% between metrics, as shown in the Figure 7. This result underscores the efficiency of pre-trained transformers, which require less data than traditional models to achieve high performance. These findings suggest that industries could reduce the time and effort spent on dataset creation, enabling earlier deployment of models for production purposes. This experiment highlights the feasibility of using smaller datasets for practical applications, such as early-stage development or resource-constrained environments, while maintaining reliable model performance. The ability of pre-trained transformers to generalize effectively with limited data opens pathways for faster, cost-effective integration into manufacturing workflows.

As can be seen Figure 8, compared to the first experiment, we observed approximately twice the number of errors per run, with around 120 misclassifications in all classes. This increase in misclassifications highlights the trade-off between dataset size and accuracy. Although the models in Experiment 1 outperformed those in Experiment 2, the fact that the error rate only doubled on the test dataset despite using only a quarter of the original data suggests that the models still performed relatively well. This result demonstrates that, while larger datasets can improve accuracy, smaller datasets can still yield good performance, offering a practical advantage in scenarios where reducing data volume or speeding up the training process is necessary, such as in early-stage production or resource-constrained environments.

3.3. Multiclass Models

In the second section, we address the multiclass classification challenge, evaluating whether the ViT can accurately distinguish among six different classes. Our experiments aimed to minimize the amount of data used for training, reducing the dataset from 8400 images (1400 images per class) to just 400 images per class in the final experiment.

3.3.1. Experiment 3

Experiment 3 serves as the baseline for the multiclass problem, using a total of 8400 images (1400 per class). This dataset size was selected to provide a robust baseline for multiclass classification, enabling a balanced evaluation across all six classes and serving as a reference for subsequent experiments with reduced datasets. As shown in the following Figure 9, all metrics consistently fall within the 97 to 98% range, significantly surpassing the 70% state-of-the-art results reported by other authors. Notably, this level of performance was achieved while using only around a third of the original dataset size (33,000 images). These results highlight the effectiveness of ViTs in leveraging reduced data volumes when combined with proper data balancing and subsampling. This outstanding accuracy demonstrates that transformers may offer more reliable and efficient defect detection than traditional models, potentially setting a new standard for industrial applications.

As with the previous experiments, Figure 10 indicates that approximately 60 images were misclassified in each run, similar to Experiment 1. Upon reviewing the dataset, including sample images from earlier sections (e.g., Figure 1), we identified several images that appeared to be incorrectly labeled. This suggests that the misclassifications may result from labeling errors or from challenging features that complicate classification, particularly for visually similar defect types. Correcting these labeling errors in future studies could further improve the model’s performance and reliability.

3.3.2. Experiment 4

In the fourth experiment, we reduced the dataset to 4800 images, with each class balanced, representing about 60% of the baseline dataset presented in Experiment 3. As shown in the following Figure 11, even with just 60% of the data, each metric decreased by only 1%, still achieving strong performance at around 97%. Similar to Experiment 2 in the binary case, this result suggests that even with reduced training data, high accuracy can still be achieved, indicating the potential to develop and deploy models faster and reduced the time consuming task of data labeling.

Figure 12 reveals a notable observation: despite reducing the dataset size by nearly half, the number of misclassifications remained nearly constant at around 70 errors, similar to Experiment 3. This indicates that the reduction in data volume had minimal impact on the model’s performance or generalization ability. In comparison to traditional CNN-based models, which often experience significant performance drops with smaller datasets, ViTs exhibit superior generalization, highlighting their effectiveness in data-limited scenarios. Additionally, some of these errors could be attributed to incorrect labels, where the model’s predictions were accurate, but the labels were flawed. Further investigation and retraining with a correctly labeled dataset could help validate these findings and enhance performance.

3.3.3. Experiment 5

In this final multiclass experiment, we reduced the dataset to just 2400 images, representing about 30% of the original data. As a result, as seen in Figure 13, we observed a performance drop of around 98% across most metrics in Experiment 3 (using the full dataset) to 95–96% in this experiment. This minimal performance decrease, despite a 70% reduction in data, demonstrates the model’s efficiency and robustness with limited data. The model’s ability to maintain high accuracy even with substantially less data suggests it can be effective in scenarios where data collection is costly or time-consuming, further supporting its viability for practical applications.

In Figure 14, it can be observed that, even with the dataset reduced to about one-third of its original size, the model maintained solid performance. Although the number of misclassifications rose to around 90, this increase was minimal—only about 1.5 higher than when using the full dataset. This modest rise in errors, despite a significant reduction in training data, highlights the model’s robustness in multiclass classification tasks, demonstrating its ability to generalize effectively even with limited data.

3.4. Inference Time

We evaluated the model’s feasibility in an industrial setting by testing its prediction speed. In a simple experiment, we sequentially processed 2400 images from different classes (see Figure 15), simulating a camera sending data for evaluation. Using an NVIDIA Tesla T4, the model achieved a preprocessing speed of 25 images per second. While this performance is decent, it could likely be improved with a distilled model.

3.5. Summary of All Experiments (Testing)

After evaluating each model’s performance, we compiled the final results, summarized in Table 4. These results show that reducing the amount of training data can still yield strong performance when proper balancing techniques are applied. For instance, in Experiment 2, reducing the dataset size led to only a 1% decrease across all metrics. Similarly, in Experiments 3 to 5, where the multiclass dataset was reduced to just 2400 images, performance dropped by only 2%. This demonstrates the efficiency of ViTs and highlights their potential for real-world applications, particularly in scenarios where data collection and labeling are time-consuming or costly, such as in niche industrial use cases. In contrast, an additional experiment (not included in the summary) using the full 33,000 images without balancing the classes resulted in significantly lower metrics, ranging from only 65 to 70%. This underscores the critical importance of data balancing for model accuracy, especially in transformer-based models. These experiments highlight the ability of ViTs to generalize effectively with smaller, balanced datasets, paving the way for future research into their deployment in resource-constrained industrial environments. Furthermore, the findings emphasize the role of balanced datasets and preprocessing techniques, such as data augmentation and weighted loss functions, in enabling the model to learn effectively from all classes and maintain high performance even with reduced data. Given the good metrics, these models can be used for transfer learning in weld defect classification on other types of welding datasets [41].

3.6. Comparison with SOTA

After evaluating the ViTs across various experiments, we compared the results with the state-of-the-art outcomes referenced in the introduction. Some teams addressing the same dataset developed 12 models named 1 to 12, categorized into two groups: the first six utilized CNNs, while the second group employed fully connected neural networks. In these experiments, both their models and ours were trained five times. However, a key difference lies in their approach, as described in [14], where each of the 12 models was trained five times using a different learning rate for each iteration, and the entire dataset of 33,000 images was utilized. In contrast, our experiments utilized a smaller subset of the dataset, with 12,000 images for Experiment 1 and 3000 images for Experiment 2. Despite the reduced data, the ViTs demonstrated superior performance, achieving accuracies approaching 99%. Table 5 below presents the best-performing fully connected and convolutional architectures from their experiments, emphasizing the remarkable effectiveness of the ViTs. Object detection models have been used in manufacturing for defect classification [25,26], but for weld defect classification using the AL5083 TIG dataset, only CNNs and dense neural networks have been applied. Thus, the performance of the proposed models is only compared to that of these models [14,34].

In Table 6, we present a summary of the multiclass experiments with a focus on accuracy. The table compares experiments conducted by Daniel Bac et al. [14] and a more recent study by Rongdi Wang et al. [34], which focused on smaller pretrained models. The experimental conditions for Daniel Bac’s team remained consistent: five experiments with different learning rates using the full dataset of 33,000 images. In contrast, Rongdi Wang’s team employed various models and implemented data augmentation techniques to balance the classes, resulting in a dataset exceeding 33,000 images. In comparison, our experiments prioritized reduced datasets. Notably, Experiment 5 utilized the smallest dataset of all, with just 2400 images, yet achieved an impressive accuracy of approximately 96%, even better than the WeldNet + FE method that uses over 33,000 images. Furthermore, we observed that using around 8000 images brought our accuracy close to 99%. These results demonstrate that our approach outperformed state-of-the-art (SOTA) models, and also highlighted the efficiency of our methodology in achieving high accuracy with limited data.

Experiment #5 stands out as the most challenging test, yet our model still delivered remarkable results, outperforming all existing models. It achieved an impressive accuracy of 96.20%, which is a substantial improvement over the best convolutional model [14] (69.9%) and fully connected architectures [14] (46.8%), with relative performance gains of 37.63% and 105.56%, respectively. Even compared to WeldNet + FE [34], the previous top-performing model with 89.1% accuracy, our model showed a 7.97% increase. These results underscore the robustness of our approach, proving its effectiveness even when faced with the hardest experimental conditions. Despite the dataset’s limited size (2400 images), our model indicated superior defect classification capabilities, reinforcing its efficiency in learning from minimal data. Furthermore, similar patterns of improvement are evident across all other experiments, solidifying the reliability and generalizability of our methodology.

Relative improvement was calculated as follows:

(5)$\mathrm{Relative}\mathrm{Improvement}={\displaystyle \frac{\mathrm{New}\mathrm{Score}-\mathrm{Baseline}\mathrm{Score}}{\mathrm{Baseline}\mathrm{Score}}}\times 100$

4. Conclusions

In this study, we implemented and evaluated ViT classifiers for industrial welding defect detection, demonstrating their ability to outperform state-of-the-art performance in binary and multiclass scenarios using the AL5083 TIG dataset. This research focused on the development of five experiments: two aimed at binary classification and the other three focused on multiclass classification. Each experiment was designed to robustly validate and train a model using cross-validation. All models achieved an accuracy exceeding 95%. Our best binary classification model reached an accuracy of 99%, surpassing the state-of-the-art by 9%. Similarly, our top-performing multiclass model achieved an accuracy of 98%, outperforming the state-of-the-art by 8%. These metrics not only exceed the state-of-the-art but also demonstrate a key advantage: our models were trained using significantly fewer images. For instance, in Experiment 2, we used only 3000 images to achieve 98% accuracy, and in Experiment 5, we used 2400 images to reach 96.2%. In contrast, state-of-the-art methods often rely on the full dataset or extensive data augmentation, while we employed downsampling techniques. The advantage of our work is that it achieves high accuracy with a small amount of data, allowing us to slightly reduce accuracy while significantly reducing the dataset through downsampling. We performed a comprehensive performance analysis of various ViT configurations, comparing their effectiveness against existing models in the defect detection domain, with a particular focus on the challenges posed by multiclass classification. Multiclass classification, which is inherently more complex than binary classification due to visual similarities between defect types and class imbalances, was effectively addressed in this study, achieving high accuracy even with reduced datasets. Additionally, we found that reducing the dataset size from 20,000 to 3000 images resulted in a small performance drop, demonstrating the potential for deploying ViTs in data-limited scenarios, such as early-stage defect detection workflows or industries with restricted data availability. The experimental results demonstrated strong performance, with ViTs achieving up to 8% higher accuracy compared to existing models, particularly excelling in multiclass defect classification. Although our study has achieved high accuracy, there are still certain limitations that need to be addressed. One such limitation is that the ViT architecture does not offer fast inference times.

Future research could establish a benchmark by evaluating different ViT models, such as DeiT (Data-efficient Image Transformer) or Swin Transformer (Shifted Window Transformer), while also focusing on improving model performance by identifying the minimum dataset size required for high accuracy, enhancing inference speed for real-time applications, and adapting the model to handle various types of welding defects and datasets. When it comes to exploring the deployment of the model in diverse manufacturing environments, we are considering the use of specialized hardware, such as embedded systems. Additionally, to gain deeper insight into the model’s decision-making in defect detection, we intend to explore interpretability techniques like attention maps, Gradient-weighted Class Activation Mapping (Grad-CAM), and Shapley Additive Explanations (SHAP) to visualize the regions influencing the model’s decisions. We will extend our training process to other welding defect classification problems and datasets, with the aim of generalizing the approach to broader industrial applications, such as pipeline inspection or automotive assembly. Another aspect we intend to explore is the incorporation of transfer learning strategies, such as fine-tuning pre-trained models, and exploring domain adaptation techniques. Additionally, we will investigate data augmentation methods, including the generation of synthetic data, to minimize misclassification between highly similar classes. Finally, we will optimize prediction inference time by prioritizing efficient model selection, such as faster ViT models, to ensure rapid inference and CPU-only feasibility. This is crucial for real-time quality control and decision-making in industrial workflows, especially high-throughput manufacturing environments. In industrial applications, rapid inference is essential for real-time decision-making and operational efficiency, making AI solutions more practical and impactful. By demonstrating the effectiveness of ViTs for welding defect detection, this work lays the foundation for faster, more reliable, and fully automated inspection workflows, significantly reducing operational costs while enhancing quality assurance in manufacturing.

Author Contributions

Conceptualization, A.C.O. and R.R.S.; Data curation, A.C.O. and R.R.S.; Formal analysis, A.C.O. and R.R.S.; Investigation, A.C.O. and R.R.S.; Methodology, A.C.O., R.R.S. and M.L.-M.; Project administration, M.L.-M.; Resources, A.C.O. and R.R.S.; Software, A.C.O. and R.R.S.; Supervision, D.E.H. and M.L.-M.; Validation, A.C.O. and R.R.S.; Visualization, A.C.O. and R.R.S.; Writing—original draft, A.C.O. and R.R.S.; Writing—review&editing, A.C.O., R.R.S., D.E.H. and M.L.-M. All authors have read and agreed to the published version of the manuscript.

Data Availability Statement

The data presented in this study are available in the Kaggle repository at https://www.kaggle.com/datasets/danielbacioiu/tig-aluminium-5083/data (accessed on 22 January 2025), reference number [14].

Acknowledgments

This research was made possible by ITJUANA, S. DE R.L. DE C.V. through its ITJ Labs department. Furthermore, we extend our gratitude to the Tecnológico Nacional de México for the invitation to the Numerical and Evolutionary Optimization workshop. This work was partially supported by TecNM through the project “Modelos predictivos para clasificación de series de tiempo usando aprendizaje de máquina”. We are truly grateful for the contributions and support of the ITJ Labs team, which has played a crucial role in successfully completing this endeavor.

Conflicts of Interest

The authors declare no conflicts of interest. This research was conducted as a Proof of Concept (PoC) within the R&D activities of ITJUANA, S. DE R.L. DE C.V. and is not currently associated with any commercial product or financial interest. While there is a possibility that this research could contribute to a commercial product in the future, no commercialization efforts or financial agreements related to this work exist at the time of publication. The findings presented in this study are independent and solely for academic and scientific dissemination.

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figures and Tables

Figure 1 Sample images of six classes.

Figure 2 ViT Architecture. The image is divided into fixed-size patches, linearly embedded with position embeddings, and passed to a Transformer encoder. An extra learnable “classification token” is added for classification.

Figure 3 Classification tasks.

Figure 4 Training pipeline. (a) The pipeline consists of five experiments. (b) For each experiment, we create five datasets based on the specific experiment setup. (c) Then, we train a classifier on each generated dataset. (d) Finally, each model is evaluated and the metrics of all models are averaged to summarize the results.

Figure 5 Box plot of different performance metrics from Experiment 1, showing the distribution of results across multiple runs. The model demonstrates consistent stability, with metrics consistently around 99%.

Figure 6 Confusion matrices for the 5 different runs of the dataset used in Experiment 1.

Figure 7 Box plot of different performance metrics from Experiment 2, showing the distribution of results across multiple runs. Minimal reduction in metrics in comparison with Experiment 1.

Figure 8 Confusion matrices for the 5 different runs of the dataset used in Experiment 2.

Figure 9 Box plot of different performance metrics from Experiment 3, showing the distribution of results across multiple runs. Baseline for multiclass welding detection.

Figure 10 Confusion matrices for the 5 different runs of the dataset used in Experiment 3.

Figure 11 Box plot of different performance metrics from Experiment 4, showing the distribution of results across multiple runs. Performance reduction of less than 1%.

Figure 12 Confusion matrices for the 5 different runs of the dataset used in Experiment 4.

Figure 13 Box plot of different performance metrics from Experiment 5, showing the distribution of results across multiple runs. Performance metrics for multiclass classification with limited data.

Figure 14 Confusion matrices for the 5 different runs of the dataset used in Experiment 5.

Figure 15 Each class with its associated prediction score.

Table 1

Number of categories in the training and test sets.

Label	Train	Test
Good weld	8758	2189
Burn through	1783	351
Contamination	6325	2078
Lack of fusion	4028	1007
Misalignment	2953	729
Lack of penetration	2819	234
Total	26,666	6588

Distribution of TIG 5083 aluminum dataset. The table shows the distribution of training and test samples across different welding defect categories.

Table 2

Dataset distribution across different experiments.

Experiment	Training Samples	Validation Samples	Testing Samples	Classification Task
1	12,000	2000	6000	Binary
2	3000	11,000	6000	Binary
3	8400	1200	2400	Multiclass
4	4800	4800	2400	Multiclass
5	2400	7200	2400	Multiclass

The table presents the number of samples in the training, validation, and testing sets for each experiment, along with the classification task type.

Table 3

Final result of each run.

Experiment	Run 1	Run 2	Run 3	Run 4	Run 5	Avg.Accuracy
Exp. #1	0.9915	0.9910	0.9905	0.9910	0.9910	0.9910
Exp. #2	0.9827	0.9827	0.9824	0.9824	0.9824	0.9825
Exp. #3	0.9725	0.9833	0.9833	0.9783	0.9833	0.9801
Exp. #4	0.9742	0.9721	0.9708	0.9708	0.9725	0.9721
Exp. #5	0.9553	0.9550	0.9640	0.9640	0.9640	0.9605

The table summarizes the accuracy results for each experiment taking the last result of the last epoch.

Table 4

Final results for all experiments.

Experiment	Accuracy	Precision	Recall	F1-Score	Training Size
Exp. #1—binary	0.990	0.990	0.990	0.990	12,000
Exp. #2—binary	0.980	0.980	0.980	0.980	3000
Exp. #3—multiclass	0.980	0.980	0.980	0.980	8400
Exp. #4—multiclass	0.974	0.976	0.974	0.976	4800
Exp. #5—multiclass	0.962	0.962	0.962	0.962	2400

Table shows the final results for all experiments.

Table 5

Binary experiments comparison with the SOTA (average accuracy).

Model Name	Accuracy
BeiT-ViT (Experiment 1)	0.990
Best Convolutional Architecture (#5) [14]	0.899
Best Fully Conected Architecture (#9) [14]	0.727

The best results are highlighted in bold.

Table 6

Multiclass experiments comparison with the SOTA (average accuracy).

Model Name	Accuracy
BeiT-ViT (Experiment 3)	0.980
Best Convolutional Architecture (#1) [14]	0.699
Best Fully Conected Architecture (#12) [14]	0.468
WeldNet + FE [34]	0.891

The best results are highlighted in bold.

References

1. Li, B.; Hou, B.; Yu, W.; Lu, X.; Yang, C. Applications of artificial intelligence in intelligent manufacturing: A review. Front. Inf. Technol. Electron. Eng.; 2017; 18, pp. 86-96. [DOI: https://dx.doi.org/10.1631/FITEE.1601885]

2. Lee, J.; Davari, H.; Singh, J.; Pandhare, V. Industrial Artificial Intelligence for industry 4.0-based manufacturing systems. Manuf. Lett.; 2018; 18, pp. 20-23. [DOI: https://dx.doi.org/10.1016/j.mfglet.2018.09.002]

3. Nti, I.K.; Adekoya, A.F.; Weyori, B.A.; Nyarko-Boateng, O. Applications of artificial intelligence in engineering and manufacturing: A systematic review. J. Intell. Manuf.; 2022; 33, pp. 1581-1601. [DOI: https://dx.doi.org/10.1007/s10845-021-01771-6]

4. Arinez, J.F.; Chang, Q.; Gao, R.X.; Xu, C.; Zhang, J. Artificial Intelligence in Advanced Manufacturing: Current Status and Future Outlook. J. Manuf. Sci. Eng. Trans. ASME; 2020; 142, 110804. [DOI: https://dx.doi.org/10.1115/1.4047855]

5. United States Department of Transportation. Federal Effort Allocation Overview; United States Department of Transportation: Washington, DC, USA, 2024.

6. Wang, Y.J. A Robust FOPD Controller That Allows Faster Detection of Defects for Touch Panels. Math. Comput. Appl.; 2024; 29, 29. [DOI: https://dx.doi.org/10.3390/mca29020029]

7. Jäntschi, L.; Louzazni, M. Accelerating Convergence for the Parameters of PV Cell Models. Math. Comput. Appl.; 2024; 29, 4. [DOI: https://dx.doi.org/10.3390/mca29010004]

8. Chung, S.B.; Venter, M.P. Analysis and Design for a Wearable Single-Finger-Assistive Soft Robotic Device Allowing Flexion and Extension for Different Finger Sizes. Math. Comput. Appl.; 2024; 29, 79. [DOI: https://dx.doi.org/10.3390/mca29050079]

9. Chen, H.; Zendehdel, N.; Leu, M.C.; Yin, Z. Fine-grained activity classification in assembly based on multi-visual modalities. J. Intell. Manuf.; 2024; 35, pp. 2215-2233. [DOI: https://dx.doi.org/10.1007/s10845-023-02152-x]

10. Ramatlo, D.A.; Wilke, D.N.; Loveday, P.W. Digital Twin Hybrid Modeling for Enhancing Guided Wave Ultrasound Inspection Signals in Welded Rails. Math. Comput. Appl.; 2023; 28, 58. [DOI: https://dx.doi.org/10.3390/mca28020058]

11. Frixione, M.G.; Roffet, F.; Adami, M.A.; Bertellotti, M.; D’Amico, V.L.; Delrieux, C.; Pollicelli, D. Integrating Deep Learning into Genotoxicity Biomarker Detection for Avian Erythrocytes: A Case Study in a Hemispheric Seabird. Math. Comput. Appl.; 2024; 29, 41. [DOI: https://dx.doi.org/10.3390/mca29030041]

12. Zhang, Y.M.; Li, L.; Kovacevic, R. Neurofuzzy model based control of weld fusion zone geometry. Proceedings of the 1997 American Control Conference (Cat. No.97CH36041); Albuquerquem, NM, USA, 6 June 1997; Volume 4, pp. 2483-2487. [DOI: https://dx.doi.org/10.1109/ACC.1997.609225]

13. Zhang, W.; Liu, Y.; Wang, X.; Zhang, Y. Characterization of three-dimensional weld pool surface in GTAW. Weld. J.; 2012; 91, pp. 195s-203s.

14. Bacioiu, D.; Melton, G.; Papaelias, M.; Shaw, R. Automated defect classification of Aluminium 5083 TIG welding using HDR camera and neural networks. J. Manuf. Process.; 2019; 45, pp. 603-613. [DOI: https://dx.doi.org/10.1016/j.jmapro.2019.07.020]

15. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Łukasz Kaiser,; Polosukhin, I. Attention is All you Need. In Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS 2017); Long Beach, CA, USA, 4–9 December 2017; Volume 30.

16. Tunstall, L.; Werra, L.V.; Wolf, T. Natural Language Processing with Transformers, Revised ed.; O’Reilly Media: Sebastopol, CA, USA, 2022; Volume 19, pp. 11-15.

17. Shen, Y.; Song, K.; Tan, X.; Li, D.; Lu, W.; Zhuang, Y. HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face. Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS 2023); New Orleans, LA, USA, 10–16 December 2023; Volume 36.

18. Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S. . An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. Proceedings of the ICLR 2021—9th International Conference on Learning Representations; Virtual, 3–7 May 2020.

19. Redmon, J.; Divvala, S.; Girshick, R.; Farhadi, A. You only look once: Unified, real-time object detection. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; Las Vegas, NV, USA, 27–30 June 2016; pp. 779-788.

20. Carion, N.; Massa, F.; Synnaeve, G.; Usunier, N.; Kirillov, A.; Zagoruyko, S. End-to-End Object Detection with Transformers. Proceedings of the 16th European Conference on Computer Vision—ECCV 2020; Glasgow, UK, 23–28 August 2020; Volume 12346, pp. 213-229. [DOI: https://dx.doi.org/10.1007/978-3-030-58452-8_13]

21. Zhao, Y.; Lv, W.; Xu, S.; Wei, J.; Wang, G.; Dang, Q.; Liu, Y.; Chen, J. DETRs Beat YOLOs on Real-time Object Detection. Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); Seattle, WA, USA, 16–22 June 2023.

22. Bacioiu, D.; Melton, G.; Papaelias, M.; Shaw, R. Automated defect classification of SS304 TIG welding process using visible spectrum camera and machine learning. NDT E Int.; 2019; 107, 102139. [DOI: https://dx.doi.org/10.1016/j.ndteint.2019.102139]

23. Xia, C.; Pan, Z.; Fei, Z.; Zhang, S.; Li, H. Vision based defects detection for Keyhole TIG welding using deep learning with visual explanation. J. Manuf. Process.; 2020; 56, pp. 845-855. [DOI: https://dx.doi.org/10.1016/j.jmapro.2020.05.033]

24. Hou, W.; Wei, Y.; Guo, J.; Jin, Y.; Zhu, C. Automatic Detection of Welding Defects using Deep Neural Network. J. Phys. Conf. Ser.; 2018; 933, 012006. [DOI: https://dx.doi.org/10.1088/1742-6596/933/1/012006]

25. Zhang, K.; Shen, H. Solder joint defect detection in the connectors using improved faster-rcnn algorithm. Appl. Sci.; 2021; 11, 576. [DOI: https://dx.doi.org/10.3390/app11020576]

26. Oh, S.J.; Jung, M.J.; Lim, C.; Shin, S.C. Automatic detection of welding defects using faster R-CNN. Appl. Sci.; 2020; 10, 8629. [DOI: https://dx.doi.org/10.3390/app10238629]

27. Yang, Y.; Pan, L.; Ma, J.; Yang, R.; Zhu, Y.; Yang, Y.; Zhang, L. A high-performance deep learning algorithm for the automated optical inspection of laser welding. Appl. Sci.; 2020; 10, 933. [DOI: https://dx.doi.org/10.3390/app10030933]

28. Guo, W.; Huang, L.; Liang, L. A weld seam dataset and automatic detection of welding defects using convolutional neural network. Proceedings of the Advances in Intelligent Systems and Computing; Shanghai, China, 17–19 August 2018; Springer: Cham, Switzerland, 2020; Volume 905, pp. 434-443. [DOI: https://dx.doi.org/10.1007/978-3-030-14680-1_48]

29. Ghimire, R.; Selvam, R. Machine Learning-Based Weld Classification for Quality Monitoring. Eng. Proc.; 2023; 59, 241. [DOI: https://dx.doi.org/10.3390/engproc2023059241]

30. Varshney, D.; Kumar, K. Application and use of different aluminium alloys with respect to workability, strength and welding parameter optimization. Ain Shams Eng. J.; 2021; 12, pp. 1143-1152. [DOI: https://dx.doi.org/10.1016/j.asej.2020.05.013]

31. Bendikiene, R.; Sertvytis, R.; Ciuplys, A. Comparative evaluation of AC and DC TIG-welded 5083 aluminium plates of different thickness. Int. J. Adv. Manuf. Technol.; 2023; 127, pp. 3789-3800. [DOI: https://dx.doi.org/10.1007/s00170-023-11779-2]

32. Ramana, M.V.; Krishna, M.; Kumar, B.V.R.; Kumar, E.P.; Dilip, Y.; Teja, V.N.S. Investigation on Joint Properties of AA5083 Aluminium Alloy Welded using A-TIG Process. AIP Conf. Proc.; 2023; 2754, 030004. [DOI: https://dx.doi.org/10.1063/5.0161200]

33. Kumar, J.D.; Thangaraj, K.; Kaliyaperumal, G.; Gogulan, C.; Kalidas, A.; Yokesvaran, K.; Liyakat, N.A.; Raj, L.S. Mechanical properties and fracture behaviour of varying filler rod composition in TIG welding of 5083 alloys. Mater. Today Proc.; 2024; in press [DOI: https://dx.doi.org/10.1016/j.matpr.2024.03.001]

34. Wang, R.; Wang, H.; He, Z.; Zhu, J.; Zuo, H. WeldNet: A lightweight deep learning model for welding defect recognition. Weld. World; 2024; 68, pp. 2963-2974. [DOI: https://dx.doi.org/10.1007/s40194-024-01759-9]

35. Wolf, T.; Debut, L.; Sanh, V.; Chaumond, J.; Delangue, C.; Moi, A.; Cistac, P.; Rault, T.; Louf, R.; Funtowicz, M. . HuggingFace’s Transformers: State-of-the-art Natural Language Processing. arXiv; 2019; arXiv: 1910.03771

36. Maurício, J.; Domingues, I.; Bernardino, J. Comparing Vision Transformers and Convolutional Neural Networks for Image Classification: A Literature Review. Appl. Sci.; 2023; 13, 5521. [DOI: https://dx.doi.org/10.3390/app13095521]

37. Cuenat, S.; Couturier, R. Convolutional Neural Network (CNN) vs Vision Transformer (ViT) for Digital Holography. Proceedings of the 2022 2nd International Conference on Computer, Control and Robotics, ICCCR 2022; Shanghai, China, 18–20 March 2022; [DOI: https://dx.doi.org/10.1109/ICCCR54399.2022.9790134]

38. Himel, G.M.S.; Islam, M.M.; Al-Aff, K.A.; Karim, S.I.; Sikder, M.K.U. Skin Cancer Segmentation and Classification Using Vision Transformer for Automatic Analysis in Dermatoscopy-Based Noninvasive Digital System. Int. J. Biomed. Imaging; 2024; 2024, 3022192. [DOI: https://dx.doi.org/10.1155/2024/3022192]

39. Bao, H.; Dong, L.; Piao, S.; Wei, F. BEiT: BERT Pre-Training of Image Transformers. Proceedings of the ICLR 2022—10th International Conference on Learning Representations; Virtual, 25–29 April 2022.

40. Géron, A. Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems; O’Reilly Media: Sebastopol, CA, USA, 2019.

41. Kumaresan, S.; Aultrin, K.S.; Kumar, S.S.; Anand, M.D. Transfer Learning with CNN for Classification of Weld Defect. IEEE Access; 2021; 9, pp. 95097-95108. [DOI: https://dx.doi.org/10.1109/ACCESS.2021.3093487]