3D reconstruction and defect pattern recognition of bonding wire based on stereo vision

Abstract

Non‐destructive detection of wire bonding defects in integrated circuits (IC) is critical for ensuring product quality after packaging. Image‐processing‐based methods do not provide a detailed evaluation of the three‐dimensional defects of the bonding wire. Therefore, a method of 3D reconstruction and pattern recognition of wire defects based on stereo vision, which can achieve non‐destructive detection of bonding wire defects is proposed. The contour features of bonding wires and other electronic components in the depth image is analysed to complete the 3D reconstruction of the bonding wires. Especially to filter the noisy point cloud and obtain an accurate point cloud of the bonding wire surface, a point cloud segmentation method based on spatial surface feature detection (SFD) was proposed. SFD can extract more distinct features from the bonding wire surface during the point cloud segmentation process. Furthermore, in the defect detection process, a directional discretisation descriptor with multiple local normal vectors is designed for defect pattern recognition of bonding wires. The descriptor combines local and global features of wire and can describe the spatial variation trends and structural features of wires. The experimental results show that the method can complete the 3D reconstruction and defect pattern recognition of bonding wires, and the average accuracy of defect recognition is 96.47%, which meets the production requirements of bonding wire defect detection.

INTRODUCTION

As an important part of modern electronic products, the bonding wire is an important bridge to connect various electronic components. Wire bonding is a process that makes the connection between an IC chip and the base material. Currently, lead wire bonding technology has been widely used in the manufacturing of PCBs, LEDs, and lithium batteries. Due to welding temperature, bonding wire quality, and other reasons, the bonding wire in the welding process will produce a variety of defects, including Lost wire, Curved wire, Collapsed wire, Borken wire-Middle, Borken wire-Single end etc., common types of bonding wire defects shown in Figure 1. The quality inspection of the bonding wire is significant because it is the basis and key to improving the quality of the product in question. Efficient and accurate defect detection of bonding wires is now a popular research topic in the semiconductor industry [1].

[IMAGE OMITTED. SEE PDF]

Currently, there are two main types of wire defect detection methods: manual visual inspection and image processing-based methods [2–4]. Manual visual inspection is inefficient, time-consuming, and labour-intensive. Although the detection method based on image processing can analyse the defects of bonding wires based on the images, it still has some shortcomings.

Limited by the two-dimensional data of the image, the camera needs to take pictures from multiple angles and analyse the data to detect the three-dimensional defect details of the bonding wire. This will make the detection process more complicated, and the detection time will take longer. It is difficult to complete the positioning and detection of the wire using image processing methods when the bonding wire and image background colours are similar. In other words, there is no simpler and more accurate method to detect defects in bonding wire.

With the rapid development of 3D data acquisition technology, the cost of 3D sensors is gradually decreasing and they are becoming more widely used. 3D data can provide rich geometric information, such as shape, size, and 3D spatial location, compared with images [5]. Point clouds, one type of common 3D data, can accurately depict the physical world and represent 3D shapes with a more concise data structure than image data. Point cloud-based defect detection techniques have been widely used in various industries, such as aircraft manufacturing [6], rail manufacturing [7], PCB [8], heritage restoration [9], and agriculture and forestry industries [10]. Object defect detection based on 3D point clouds usually includes methods based on standard model comparison [11–14] and methods based on point cloud geometric features [6, 15–17]. Standard model comparison-based methods must have standard model point clouds, but the length and curvature of bonding wires are variable, as shown in Figure 2, so this method is not flexible enough. The inspection objects are primarily flat or simple surfaces in the point cloud feature-based defect detection methods. The bending of bonding wires is complex, and existing methods cannot identify multiple defect patterns.

[IMAGE OMITTED. SEE PDF]

To address the inflexibility of model-based comparison methods, this paper focuses on the geometric features and spatial structure of bonding wires, and proposes a stereo vision-based method for defect pattern recognition of bonding wires. The method provides a framework for bonding wire localisation, 3D reconstruction, and feature analysis. The main innovative work of this paper is as follows.

• (1)

  To address the problem of noisy points in 3D reconstruction, we propose a point cloud segmentation method based on spatial surface feature detection (SFD). The method separates valid points from noisy points by analysing the density of point clouds in space and the feature relationship between points.

• (2)

  A directional discretisation descriptor with multi-local normal vectors is proposed for the problem of defect detection and pattern recognition in bonding wires. This descriptor combines local and global features of the bonding wire to describe the spatial trends and structural characteristics of the bonding wire.

• (3)

  Our method can complete the 3D reconstruction of bonding wires and multiple defect pattern recognition, and the average pattern recognition accuracy is over 96.47%. Compared with traditional image processing methods, this method can provide richer defect information by analysing the 3D point cloud of the bonding wire.

The rest of the paper is organised as follows. The second part reviews the existing related methods. The third part presents the details of the methods for the detection of bonding wire defects. We provide a rich comparative experiments of the proposed methods in the fourth part. Finally, the conclusion of the paper is assigned.

RELATED WORKS

Bonding wire defect detection

Non-destructive detection techniques are gradually being applied to the defect detection of semiconductors as machine vision technology advances. Perng et al. [2] were the first to use machine vision to detect defects in bonding wires and propose detection methods for various defect types. Perng et al. [3] later proposed a bonding wire defect detection method that combined image processing and simulation. To detect defects of bonding wires, Chan et al. [4] used Otsu threshold selection and connected component algorithms. Ye et al. [18] proposed a qualified solder joint image dictionary for defect detection in solder joints. The method mentioned above necessitates using a template image during the detection. In this case, it is inflexible to use the template image to determine the location and defects of the object to be detected. Deep learning has brought new dynamics to semiconductor defect detection [19]. Chen [20] created a data-driven framework for detecting defects of bonding wires that consists of three processes: data preprocessing, feature engineering, and chip classification. Chen [21] proposed a ResNet-based chip image defect detection method with a chip defect detection accuracy of 98.57% for simple circuit structures. Hu [8] created an attention network to determine the location of solder joints and to evaluate solder joint quality. Ding [22] proposed a TDD-Net network to improve the performance of PCB defect detection. Although deep learning and image processing are widely used in the semiconductor industry, they have not been adequately discussed for bonding wire defect detection. Currently, no relevant methods exist to model the bonding wires and investigate the specific details of the bonding wires.

Three-dimensional defect detection

The defect detection methods based on point clouds have been applied in several fields. We can classify detection methods into two categories: model comparison and feature extraction. Model comparison-based methods accomplish defect detection by comparing defect models with standard models [11]. Zhang et al. [12] converted point clouds into voxel models and extracted defects by comparing the differences between the models. Furthermore, point cloud alignment methods based on ICP [13] and Octree [14] have been applied to defect detection by model comparison. When a large amount of training data is available, defect detection models can be trained using SVM [23], 3D convolutional neural networks [8, 24, 25], and other methods. However, the model comparison-based method suffers from the low efficiency of 3D modelling and the time-consuming detection process [26]. Also, there is not a large amount of experimental data available for training defect detection models in bonded wire defect detection. Therefore, the above methods cannot be directly applied to the bonding wire defect detection process. The feature extraction-based method detects defects by extracting point cloud feature descriptors. Jovancevic et al. [6] detected the surface flatness of an aircraft by comparing the difference of the normal vectors of adjacent points. Zheng et al. [15] used curve fitting to solve the defect detection problem in high-voltage cable joints. Miao et al. [16] used the FPFH (Fast Point Feature Histograms) to extract characteristics and detect defects on turbine blades. Madrigal et al. [17] were inspired by the PFH (Point Feature Histograms) and proposed a new 3D local descriptor model point feature histogram (MPFH) for defect detection.

The raw point cloud data is a collection of points representing spatial information on the surface of an object with high redundancy. Therefore, it does not contain the same topological relationship information as 2D images or traditional tangible grid data. Point clouds and depth maps have a mapping relationship, and defect detection can be improved by combining the benefits of images and point clouds. Therefore, defect detection methods based on data fusion of 2D images and 3D point clouds are also a mainstream trend nowadays. Cao et al. [7] completed the detection of rail defects by projecting the point cloud and fitting the model to the projected data. Zong et al. [9] use images to locate the defect, then maps the defect information to 3D space and marks it. Chu et al. [27] first used the images for defect detection, and the defective ROI(Region Of Interest) regions were reconstructed in 3D to evaluate the defects in detail. Zhang et al. [28] classify “true” and “false” holes using complementary information from 3D point clouds and 2D images.

The bonding wire bends in space in a variety of ways, and some faults, such as broken defects, cause the wire to be discontinuous. As a result, the straight-line detection method in 3D point clouds [29] does not apply to the extraction of bonding wires. Because of the aforementioned reasons, it is extremely difficult to locate and extract bonding wires directly from raw point clouds. Inspired by the data fusion of image and point cloud, it is possible to locate the bonding wires in the depth map first and then complete the 3D reconstruction of the bonding wires.

METHODOLOGY

This section details the 3D point cloud reconstruction and defect detection method for bonding wires. The method implementation process is simplified, as shown in Figure 3.

[IMAGE OMITTED. SEE PDF]

3D reconstruction of bonding wires

To accurately detect bonding wire defects, we must extract the point cloud of bonding wires from the original point clouds. However, it is challenging to distinguish bonding wires from other electronic components in the original point cloud. Because it requires designing targeted features for the bonding wires and extensive data computation. There is a mapping relationship between the depth image and the original point cloud. Each pixel value in the depth image represents the distance from the camera of a point in the scene. In the PCB depth image, the image background is far from the camera, and the electronic components are close to the camera. We can use threshold segmentation to segment the electronic components and the image background. Furthermore, to distinguish different electronic components in the depth map, this section focuses on the geometric characteristics of the bonding wires and other electronic components on the PCB board. The geometric characteristics of bonding wires are slender lines with a large ratio of length to width of the external rectangle. Other electronic components, while of various types, have a smaller ratio of length to width of their external rectangle. After distinguishing the different electronic components, the point cloud of the bonding wires was extracted using the coordinate conversion method.

The oriented bounding box (OBB) [29] is a method for calculating the external rectangular of various objects. By performing PCA analysis on the data, OBB can draw more compact rectangular than axis-aligned bounding box (AABB). The PCB image must be pre-processed before drawing rectangular for various electronic components in the PCB image. We can use image enhancement and threshold segmentation to improve the geometric features of electronic components in PCB images, as shown in Figure 4.

[IMAGE OMITTED. SEE PDF]

Median Absolute Deviation (MAD) is a popular one-dimensional data analysis method for screening outliers. Compared with other electronic components, the length-to-width ratio of the external rectangle of the bonding wire is unusually large, which is an outlier. Therefore, we can use the MAD algorithm to differentiate bonding wire from other electronic components. Each electronic component has an external rectangle, as shown in Figure 4f. The length of each external rectangle is W, the width is H, and the ratio of W to H is D_i. The set of the length-to-width ratios of all rectangular is D. D_meadian represents the median of the set D. Set D is obtained by subtracting D_meadian from the new set D′. The median of D′ is calculated and denoted as $${D}_{\mathit{meadian}}^{\prime }$$. D_i of the electronic component that satisfies Equation (1) is the bonding wire. 1 $${D}_{\mathrm{i}}-{D}_{\text{median}\,} > 8\ast {D}_{\text{median}\,}^{\prime }$$

The bonding wire data is converted into a point cloud by the coordinate transformation of Equation (2). Figure 5 shows the depth image of bonding wires. Figure 6 shows the point cloud data of bonding wires. 2 $$\left[\begin{array}{@{}c@{}}\hfill X\hfill \\ \hfill Y\hfill \\ \hfill Z\hfill \end{array}\right]=\left[\begin{array}{@{}c@{}}\hfill {X}_{\text{scale}\,},0,0\hfill \\ \hfill 0,{Y}_{\text{scale}\,},0\hfill \\ \hfill 0,0,{Z}_{\text{scale}\,}\hfill \end{array}\right]\left[\begin{array}{@{}c@{}}\hfill x\hfill \\ \hfill y\hfill \\ \hfill z\hfill \end{array}\right]+\left[\begin{array}{@{}c@{}}\hfill {X}_{\text{offset}\,}\hfill \\ \hfill {Y}_{\text{offset}\,}\hfill \\ \hfill {Z}_{\text{offset}\,}\hfill \end{array}\right]$$ where X, Y, and Z are the point cloud coordinates in space, x and y are the pixel points in the depth map, z is the height information in the depth map, and X_scale, Y_scale, Z_scale, X_offset, Y_offset and Z_offset are the internal parameters of the camera.

[IMAGE OMITTED. SEE PDF]

[IMAGE OMITTED. SEE PDF]

Point cloud segmentation method based on spatial surface feature detection

Due to the characteristics of the camera itself, noise points will inevitably be generated when collecting point cloud data. The noisy point cloud is intertwined with the bonding wire point cloud, which will affect the correct representation of the spatial information of the bonding wire. In other words, the presence of noisy point clouds can affect the accuracy of defect detection results. Therefore, we propose a point cloud segmentation method to filter the noise points based on the above characteristics. The frame diagram of the proposed method is shown in Figure 7. To obtain a more streamlined point cloud of the bonding wire, we analysed the spatial structure of the bonding wire. We concluded that the reconstructed point cloud has the following characteristics. (1) Low density of noisy point clouds and high density of point clouds on the surface of the bonding wire. (2) The point cloud of the bonding wire is distributed outside of the original point cloud.

[IMAGE OMITTED. SEE PDF]

Affine transformation of bonding wires

The different poses and heights of bonding wires in space are not conducive to the design of point cloud segmentation methods. Therefore, we performed an affine transformation on the point cloud. First, the eigenvalues and eigenvectors of the point cloud are determined using principal component analysis (PCA). After calculation, the eigenvector corresponding to the largest eigenvalue is determined as the new X-axis. The affine transformation of the point cloud is performed using Equation (3). Meanwhile, in order to normalise the length of the bonding wire, we normalise the point cloud so that the distribution of the point cloud in the X-axis is scaled to between [0, 1]. Figure 8 shows the pose of the point cloud before and after the affine transformation. 3 $$\left[\begin{array}{@{}c@{}}\hfill {X}^{\prime }\hfill \\ \hfill {Y}^{\prime }\hfill \\ \hfill {Z}^{\prime }\hfill \end{array}\right]=\left[\begin{array}{@{}ccc@{}}\hfill \mathrm{C}\mathrm{o}\mathrm{s}\theta \hfill & \hfill -\mathrm{S}\mathrm{i}\mathrm{n}\theta \hfill & \hfill 0\hfill \\ \hfill \mathrm{S}\mathrm{i}\mathrm{n}\theta \hfill & \hfill \mathrm{C}\mathrm{o}\mathrm{s}\theta \hfill & \hfill 0\hfill \\ \hfill 0\hfill & \hfill 0\hfill & \hfill 1\hfill \end{array}\right]\left[\begin{array}{@{}c@{}}\hfill X\hfill \\ \hfill Y\hfill \\ \hfill Z\hfill \end{array}\right]$$ where θ is the angle between new X-axis and the original X-axis. X, Y, and Z are the original point cloud, X′, Y′, and Z′ are the point cloud after affine transformation.

[IMAGE OMITTED. SEE PDF]

Calculate the normal vector of the point cloud

The point clouds of bonding wire, denoted as $$S\left({p}_{i}\right)$$, p_i is each point could in the space. Use the k-d tree [30] to query the n points near point p_i, denoted as $$N\left({p}_{i}\right)$$. Calculate the normal vector n_i of p_i. The sum of the distances from all points to the plane satisfies Equation (4). 4 $$L\left({n}_{i}\right)=\underset{{n}_{i}}{\mathrm{arg min}\,}\sum\limits _{{p}_{j}\in N\left({p}_{i}\right)}{\left({\left({p}_{j}-{p}_{o}\right)}^{T}{n}_{i}\right)}^{2}$$ where p_o is the centre point of $$N\left({p}_{i}\right)$$, and the calculation Equation is (5). 5 $${p}_{o}=\frac{1}{n}\sum\limits _{{p}_{j}\in N\left({p}_{i}\right)}{p}_{j}$$

According to PCA, it is known to find a direction such that the sum of projections of all neighbouring points in $$N\left({p}_{i}\right)$$ is the smallest. This means that the distribution of projection points in that direction is the most concentrated, and the variance is the smallest. We can consider that direction as the normal vector n_i. n_i is the eigenvector corresponding to the smallest eigenvalue found by PCA. The solution of 4 can be transformed into the eigenvalue decomposition of the covariance matrix S, see Equation (6). 6 $$S=\sum\limits _{{p}_{j}\in N\left({p}_{i}\right)}\left({p}_{j}-{p}_{o}\right){\left({p}_{j}-{p}_{o}\right)}^{T}$$ where S is the covariance matrix of $$N\left({p}_{i}\right)$$.

The corresponding normal vector n_i is calculated for each p_i in $$S\left({p}_{i}\right)$$. The direction of n₁ is taken as the reference normal vector direction, and the adjacent normal vectors are required to have the same sign, that is, n_i ⋅ n_i−1 ≥ 0, as shown in Figure 9.

[IMAGE OMITTED. SEE PDF]

Filtering of noisy point clouds

In the original point cloud, noisy points and bonding wire point clouds are clustered together, which is challenging for point cloud segmentation. Traditional point cloud segmentation methods such as RANSAC [31], region growing [32], and DBSCAN [33] accomplish the segmentation of point cloud regions directly based on the difference of point cloud features. The above methods do not try to reduce the density of noise points to improve the segmentation accuracy. In this section, we research the noise point density reduction method and complete the initial filtering of the point cloud according to the difference in point cloud density in space. At the same time, we also consider the features between point clouds to improve the anti-interference and accuracy of the point cloud segmentation method. The framework diagram of our designed point cloud segmentation method is shown in Figure 10.

[IMAGE OMITTED. SEE PDF]

First, we reduce the density of the noise points by slicing the point cloud. The point cloud data are sliced along the X-axis, and the slice thickness is the minimum interval between adjacent points in the X-axis. The sliced point cloud is shown in Figure 10b. Extract all the point clouds of the same X scale, denoted as $$I\left({p}_{i}\right)$$. These point clouds satisfy Equation (7) and are distributed in the same Y-Z axis. By slicing the point clouds, the aggregation of noise points is avoided and the density of noise points is reduced. 7 $$\forall {p}_{j}\in I\left({p}_{i}\right),{p}_{j}^{x}={p}_{j-1}^{x}$$ where $${p}_{j}^{x}$$ is the X coordinate of point p_j.

As shown in Figure 10b, in $$I\left({p}_{i}\right)$$, the noise points are more discrete, and the bonding wire point cloud is more densely distributed. We plotted the Z-axis histogram of $$I\left({p}_{i}\right)$$, as shown in Figure 10c. The point with the highest frequency in the histogram is the point cloud of the bonding wire, noted as the central point p_c. p_c is calculated by Equation (8). We used the histogram to analyse the density of the point cloud and completed the initial filtering. 8 $${p}_{c}={p}_{j}=\underset{{p}_{j}\in I\left({p}_{i}\right)}{\mathrm{arg}\,\max \,}\left({\Phi }\left({p}_{j}^{z}\right)\right)$$ where $${\Phi }\left({p}_{j}^{z}\right)$$ is the number of points with the same Z-axis coordinates and $${p}_{j}^{z}$$ is the Z-coordinate of point p_j.

In $$I\left({p}_{i}\right)$$, the points distributed within the proximity of p_c can also be considered as the point cloud of the bonding wire. Inspired by FPFH [34], we further complete the filtering of noisy points based on the similarity of point cloud features. Iteratively calculate the distance l_j, Y-axis angle α_j, and normal vector angle θ_j between the centre point p_c and each of the remaining points $${p}_{j},{p}_{j}\in I\left({p}_{i}\right)$$ to generate the set of descriptors $$\left({l}_{j},{\alpha }_{j},{\theta }_{j}\right)$$. Let v = (0, 1, 0), descriptors are calculated as (9)(10)(11). The proposed descriptors are shown in Figure 10d. 9 $${l}_{j}=L\left({p}_{j}\right)=\left\Vert {\mathbf{p}}_{\mathbf{c}}-{\mathbf{p}}_{\mathbf{j}}\right\Vert $$ 10 $${\alpha }_{j}=\mathrm{arccos}\left(\frac{\left({\mathbf{p}}_{\mathbf{c}}-{\mathbf{p}}_{\mathbf{j}}\right)\cdot \mathbf{v}}{\left\Vert {\mathbf{p}}_{\mathbf{c}}-{\mathbf{p}}_{\mathbf{j}}\right\Vert }\right)$$ 11 $${\theta }_{j}=\mathrm{arccos}\left(\frac{{\mathbf{n}}_{\mathbf{c}}\cdot {\mathbf{n}}_{\mathbf{j}}}{\left\Vert {\mathbf{n}}_{\mathbf{c}}\right\Vert \times \left\Vert {\mathbf{n}}_{\mathbf{j}}\right\Vert }\right)$$ where $$L\left({p}_{j}\right)$$ is the distance between point p_j and point p_c, n_c is the normal vector of point p_c, and n_j is the normal vector of point p_j.

If any $$\left({l}_{j},{\alpha }_{j},{\theta }_{j}\right)$$ of $${p}_{j},{p}_{j}\in I\left({p}_{i}\right)$$ satisfies two conditions in Equation (12), Equation (13) and Equation (14), then the point will be considered as the point cloud of the bonding wire. 12 $$\left\{\begin{array}{@{}l@{}}{l}_{\mu }={\sum }_{{p}_{j}\in I\left({p}_{i}\right)}L\left({p}_{j}\right)\quad \hfill \\ {l}_{\sigma }=\sqrt{\frac{1}{n}{\sum }_{{p}_{j}\in I\left({p}_{i}\right)}{\left(L\left({p}_{j}\right)-{l}_{\mu }\right)}^{2}}\quad \hfill \\ {l}_{j}\le \left\vert {l}_{\mu }-{l}_{\sigma }\right\vert \quad \hfill \end{array}\right.$$ where l_μ and l_σ are the mean and variance respectively. 13 $${\alpha }_{j}< {\alpha }_{\max },{\alpha }_{\max ,}\in \left(5{}^{\circ},10{}^{\circ}\right)$$ 14 $${\theta }_{j}< {\theta }_{\max },{\theta }_{\max }\in \left(5{}^{\circ},10{}^{\circ}\right)$$ where α_max and θ_max are the set threshold values. In this paper, we set α_max and θ_max to 5°.

Bonding wire defect detection method based on point cloud projection and feature descriptors

The types of defects in the bonding wire can be divided into two categories based on the continuity of the bonding wire. The intermediate break divides the bonding wire into two parts, as shown in Figure 1e, which makes the bonding wire no longer have continuity. Other types of defective bonding wires remain as a continuous whole. For each of the two types of bonding wires, we proposed corresponding defect detection methods. The order of detection is as follows. (1) Broken wires. (2) Other types of defects.

Broken wire detection method based on point cloud projection

Image processing-based methods use connected domain detection, threshold segmentation, and other methods to detect broken wires. However, the dilation and erosion operations in image processing may change the continuity of the bonding wire and affect the authenticity of the detection results. When the bonding wire and the image background are similar, the image processing method is challenging to complete in detecting broken wires. The continuous bonding wire is composed of discrete points in the point cloud data. The distance between adjacent point clouds in the disconnected section becomes larger. We can calculate the interval between adjacent points to detect a break defect to determine if the bonding wire is defective.

The number of original point clouds is too large, which slows down the computation of the algorithm. The overall structure of the bonding wire can be preserved while reducing computation by downsampling the point cloud. This section sets the point cloud to 200 points after downsampling. To further reduce the computational effort, we only calculate the X-axis interval of adjacent points. We project the point cloud to the X-Z axis and note that the projected point is p_i = (x_i, z_i). After projection, the interval between adjacent points in the X-axis is calculated and noted as E_i. The calculation equation is (15). 15 $${E}_{i}={x}_{i}-{x}_{i-1}$$

We can set a threshold value for E_i to detect if the bonding wire is broken. If E_i is greater than the set threshold, the wire is broken. In order to reduce the human factor in the threshold setting and improve the anti-interference of the detection algorithm, we use the mean and variance in statistics to complete the threshold setting. We denote the mean of E_i as μ and the variance as σ. The 3-sigma algorithm assumes that in a normal distribution, when the data lies in (μ − 3σ, μ + 3σ), then it is considered normal data. When the data value is greater than μ + 3σ, the confidence level of the data decreases, and it is regarded as a discrete value. The 3-sigma algorithm can be used not only for data analysis of normal distribution but also for judgement of the reasonableness of daily data. In this paper, the detection threshold setting is completed according to the 3-sigma algorithm. Equations (16) and (17) complete the calculation of the mean and variance of E_i. If there exists E_i satisfying Equation (18), then the bonding wire is broken. 16 $$\mu =\frac{1}{n}\sum\limits _{i=1}^{n}{E}_{i}$$ 17 $$\sigma =\sqrt{\frac{1}{n}\sum\limits _{i=1}^{n}{\left({E}_{i}-\mu \right)}^{2}}$$ 18 $${E}_{i} > u+3\sigma $$

Bonding wire defect detection based on feature descriptors

Different types of defects result from different places on the bonding wire bending. The various defect patterns, however, can all be summed up as various trends in the local or general changes of the bonding wire. To accurately analyse the three-dimensional defects of the bonding wire, we summarise the key to the bonding wire defect detection method. (1) The correct expression of the local trend of the bonding wire. (2) The description of the overall structure of the bonding wire in space. We propose a directional discretisation descriptor for multi-local plane normal vectors to solve the above problems.

Inspired by the region growth, we use seed points to divide the point cloud of the bonding wire into multiple parts. The local trend of the bonding wire is expressed by computing feature descriptors for each part of the point cloud. The point cloud of the wire is downsampled, and there are M points of wire data after downsampling, denoted as $$M\left({p}_{i}\right)$$, and p_i is each point in space. A k-d tree structure is established for $$M\left({p}_{i}\right)$$, and a seed point is randomly selected among the first 10 points of $$M\left({p}_{i}\right)$$ along the X-axis direction and recorded as p_s1. A seed point is set at an interval of K points and a total of N seed points, and N is a positive integer. The calculation formula is shown in Equations (19) and (20). 19 $$N=\lfloor (M-10)/K\rfloor +1$$

where “⌊⌋” is rounded down. 20 $$\begin{array}{rl}\hfill & \mathrm{S}\mathrm{E}\mathrm{E}\mathrm{D}\left({p}_{s}\right)=\left\{{p}_{s1},{p}_{s2},{\cdots}{p}_{sn}\right\}=\left\{{p}_{k+(n-1)K}^{{x}_{s+(n-1)}}\right\},k\in [1,10],n\in [1,2,{\cdots}N]\hfill \end{array}$$ where $$SEED\left({p}_{i}\right)$$ is the set of seed points and $$SEED\left({p}_{i}\right)\subseteq M\left({p}_{i}\right)$$, $${p}_{k+(n-1)K}^{{x}_{s+(n-1)}}$$ is the k + nKth point in the x-direction. In this paper, k is 5, M is 200 and N is 10.

Normal vector is a common descriptor in point clouds. We can first fit the normal vector of the plane adjacent to the seed point. The angle between the normal vector and the X-axis is calculated to express the local characteristics of the bonding wire, as shown in Figure 11. Use the k-d tree to query the K neighbouring points near the j th seed point p_sj, denoted as $${N}_{j}\left({p}_{i}\right),{N}_{j}\left({p}_{i}\right)\subseteq M\left({p}_{i}\right)$$. Calculate the normal vector n_i for each point within $${N}_{j}\left({p}_{i}\right)$$. The normal vector is calculated in Section 3.2. Calculate the normal vector mean of $${N}_{j}\left({p}_{i}\right)$$, denoted as n_μj. The calculation equation is given in (21). 21 $${\mathbf{n}}_{\mu \boldsymbol{j}}=\frac{1}{K}\sum\limits _{i=1}^{K}{\mathbf{n}}_{\mathbf{i}}$$

[IMAGE OMITTED. SEE PDF]

Use Equation (22) to calculate the angle between n_μj and the X-axis, noted as γ_j. 22 $${\gamma }_{j}=\mathrm{arccos}\left(\frac{{\mathbf{n}}_{\text{pj}}\cdot \mathbf{u}}{\left\Vert {\mathbf{n}}_{\text{uj}}\right\Vert }\right)$$ where u = (1, 0, 0).

We use multiple local features to illustrate the overall structure of the bonding wire. However, the bending of the bonding wire is different, and the calculated angle γ_j is diverse, which poses difficulties for defect pattern recognition. Therefore, to make the defect pattern recognition more standardised, we further simplified the expression of the descriptors. First, using Equation (23) to calculate the local plane descriptor δ_j. 23 $${\delta }_{j}=\left\{\begin{array}{@{}l@{}}+1,{\gamma }_{j}\ge \frac{\pi }{2}\quad \hfill \\ -1,{\gamma }_{j}< \frac{\pi }{2}\quad \hfill \end{array}\right.$$

The generation of the collection of descriptors Q follows the computation of N local plane descriptors. 24 $$Q=\left\{{\delta }_{1},{\delta }_{2},{\cdots}{\delta }_{n}\right\}$$

The descriptor set Q can also be written as Equation (25). 25 $$Q=\left\{\overset{N}{\overbrace{\underset{n}{{\underbrace{i,\text{\ldots },i}}},\underset{m}{{\underbrace{j,\text{\ldots },j}}}}}\right\}$$

Since there is no literature on defect detection for bonding wires using 3D point clouds. With the help of experienced engineers, we performed extensive experiments on defect pattern recognition. Table 1 was constructed to perform defect pattern recognition. Figure 11 shows the descriptors for different types of bonding wires.

TABLE 1 Descriptors and defect patterns reference table.

		Descriptor
Defect pattern		i	j	n	m
Broken wire -single end	Right end	1	−1	⌊(0.8 ∼ 1)*N⌋	N-n
	Left end	1	−1	⌊(0 ∼ 0.2)*N⌋	N-n
Normal wire		1	−1	⌊(0.3 ∼ 0.7)*N⌋	N-n
Collapsed wire		−1	1	⌊(0.1 ∼ 0.9)*N⌋	N-n
Curved wire		Others

EXPERIMENTAL RESULTS AND ANALYSIS

Experiment preparation

The point cloud acquisition system consists of laser profiler (MV-DP2307-01H), motion stage, camera fixed stage, and PCB boards, as shown in Figure 12. The parameters of the laser profiler are shown in Table 2. The length and width of the PCB board used for experimental acquisition are 50 mm × 20 mm, and the thickness of the bonding wire is 0.5 mm. From Table 2, the camera field of view and resolution accuracy can meet the experimental requirements. The method in this paper is written in Python and uses the open3d library. All experiments were run on AMD Ryzen 7 5800H 3.2 GHz CPU and 16 GB RAM.

[IMAGE OMITTED. SEE PDF]

TABLE 2 SPECIFIC parameters for MV-DP2307-01H.

Parameters	Value
Reference range (mm)	96
Resolution in the Z direction (μm)	3–4.5
Resolution in the X direction (μm)	27–47
Wavelength (nm)	405
Number of points on per outline	2048
Linearity in the Z direction (% of MR)	0.01
Scan frequency for maximum measurement range (HZ)	700
Repeatability in the Z direction (μm)	0.5

Comparison of AABB and OBB in the application of 3D reconstruction of bonding wires

In this section, AABB and OBB are used to calculate the external rectangles of the electronic components in Figure 4f. As shown in Figure 13, for the same electronic component, the ratio of the length and width of the external rectangle calculated by AABB is 1.16 and that of OBB is 6.44. By comparison, the OBB can correctly reflect the actual situation of the bonding wire contour. The results of the two different methods for the external rectangles of each contour in the PCB image are shown in Figure 14.

[IMAGE OMITTED. SEE PDF]

[IMAGE OMITTED. SEE PDF]

The external rectangles numbered 25, 26, and 27 in Figure 14b are the external rectangles of the bonding wire. Calculate the ratio of the length and width of each rectangle in Figure 14b. Using the method proposed in section 3.1 to complete the distinction between bonding wires and other electronic components, the results are shown in Figure 15. Figure 15 shows that only the numbers 25, 26, and 27 are higher than zero. This indicates that the proposed method can distinguish between bonding wires and other electronic components. Furthermore, $${D}_{\mathit{median}}+8{D}_{\mathit{median}}^{\prime }$$ is the threshold value defined in this paper, which can be altered in practice to make the detection algorithm more generalisable.

[IMAGE OMITTED. SEE PDF]

Point cloud segmentation results of bonding wires and comparative analysis of different methods

In this section, we collected 9 sets of bonding wire point cloud data for point cloud segmentation experiments. The experimental data are shown in Figure 6. In Figure 6a,b,d,e, and (e) are Normal wires, (c) is a Broken wire-Single end, (f) and (g) are Broken wires-Middle, (h) is a Curved wire, and (i) is a Collapsed wire. We use the point cloud segmentation method proposed in Section 3.2 to segment nine groups of point clouds, and the results are shown in Figure 16.

[IMAGE OMITTED. SEE PDF]

In order to verify the effectiveness of our method, this paper also uses a variety of commonly used segmentation algorithms for experimental comparison of point cloud segmentation. In this paper, we used the model-based segmentation method: RANSAC [31], region-based segmentation method: Region Growing [32], clustering-based segmentation method: DBSCAN [33], Euclidean clustering [35] and deep learning-based method: PointNet++ [36]. Where the model parameters of PointNet++ were pre-trained in shapenet40. The above method was used to segment the nine groups of point clouds and the front view of the point clouds was taken. The segmentation results are shown in Figure 17. In order to quantify the results of the point cloud segmentation, we downsampled the segmented point cloud to 200 points. The number of point clouds of bonding wires within 200 points is noted as P_seg. We use the Equation (26) to quantify the accuracy of the point cloud segmentation. The results of the point cloud segmentation quantification are shown in Table 3. 26 $$A{C}_{\mathit{seg}}=\frac{{P}_{\mathit{seg}}}{AL{L}_{\mathit{seg}}}$$ where AC_seg is the accuracy of point cloud segmentation results, P_seg is the number of point clouds of bonding wires, and ALL_seg is the number of all point clouds. ALL_seg is 200.

[IMAGE OMITTED. SEE PDF]

TABLE 3 Point cloud segmentation results of different methods.

Method	ACSeg
OD1	OD2	OD3	OD4	OD5	OD6	OD7	OD8	OD9
RANSAC	0.73	0.92	0.88	0.96	0.63	0.78	0.32	0.47	0.61
DBSCAN	0.94	0.73	0.55	0.98	0.97	0.67	0.84	0.78	0.85
Euclidean clustering	0.29	0.55	0.77	0.33	0.47	0.29	0.67	0.23	0.19
Region growth	0.95	0.89	0.91	0.82	0.87	0.94	0.80	0.99	0.88
PointNet++	0.71	0.69	0.68	0.56	0.62	0.71	0.65	0.65	0.51
Our method	1.00	1.00	1.00	1.00	0.98	1.00	0.995	1.00	0.95

As shown in Figure 17 and Table 3, our segmentation method can filter the noise points in the point cloud of the bonding wire with the best results. Other point cloud segmentation methods cannot accurately remove the noise points from the point cloud of each bonding wire. Among the other point cloud segmentation methods, the better-performing one is the region-based method, which gives the overall contour of the bonding wire in the segmentation result. However, the segmented point cloud still contains some noisy points. The noisy point clouds near the surface of the bonding wire express the same features as the bonding wire, and the region-based method cannot accurately determine the difference in their features. As a result, such methods for accurate point cloud segmentation require further development. Model-based RANSAC and hierarchical clustering-based DBSCAN can complete the segmentation of point clouds of partial bonding wires and effectively remove most of the redundant point. However, for noisy point clouds that form strong features in space, such as straight lines, RANSAC and DBSCAN will not segment the point clouds and will consider them to be valid data. Euclidean clustering performs the worst in point cloud segmentation because it finishes segmenting point clouds by judging the Euclidean distance between similar point clouds. When the noise data is dense in space, the method of Euclidean clustering cannot complete the segmentation of the noise data. PointNet++ shows strong point cloud segmentation ability among the deep learning methods in the shapenet40 dataset, but it does not perform well when segmenting the data in this paper. This also reflects one of the current problems of deep learning: without pre-training deep learning models on targeted data, such models cannot perform well. On the whole, the method in this paper can better segment the point cloud of bonding wires compared with many other methods.

Analysis and application of the defect detection methods

Broken wire detection method based on point cloud projection

This section uses a point cloud of defective bonding wires, as shown in Figure 16f, and normal bonding wires, as shown in Figure 16a, to illustrate the defect detection results. The method proposed in Section 3.3 is used to detect defects in both sets of bonding wires. Figure 18a shows the projection of the point cloud of the bonding wires. Figure 18b shows the X-axis interval of the adjacent point clouds. From Figure 18b, we can see that the defective bonding wire will show a large peak at the breakout, and the peak value is larger than the set threshold. Therefore, the method proposed in this paper can complete the detection of broken wire defects. The threshold value set in this paper is μ + 3σ. We can achieve the accuracy of detection by resetting the threshold value.

[IMAGE OMITTED. SEE PDF]

Results of bonding wire defect detection based on feature descriptors

Figure 16a shows a Normal wire, Figure 16h shows a Curved wire, Figure 16c shows a Broken wire-Single end, and Figure 16i shows a Collapsed wire. We use the method proposed in Section 3.3 to calculate the feature descriptors for the above four different defect modes of the wire, with M set to 200, K set to 20, and N set to 10. The calculation of these descriptors is shown in Figure 19b.

[IMAGE OMITTED. SEE PDF]

As shown in Figure 19b, the Normal wire descriptors are positive and then negative, and the number of ‘+1’ and ‘−1’ is the same. In the case of Curved wire, the bonding wire will be bent several times, so the descriptor symbol is not continuous. The defect descriptor for Collapsed wire is the opposite of Normal wire, where there is a negative and then a positive condition. The Broken wire-Single end has a partial loss of point cloud at one end. Therefore, the number of ‘+1’ or ‘−1’ is relatively large in the descriptor. As illustrated in Figure 19b, the method described in this paper generates descriptors for various types of defects in bonding wires with a high degree of variability. We can roughly determine the spatial structure of the bonding wire by the descriptors.

Comparison of interpretability of bonding wire defect descriptors

To further illustrate the suitability of our method for defect detection of bonding wires, we investigated the interpretability of the proposed descriptors. We compared our method with the local descriptor FPFH [34] and the global descriptor VFH [37]. Because FPFH calculates descriptors for each point, each point has a 33-dimensional descriptor. After downsampling the bonding wire data to 200 points, the wire features extracted with FPFH are 200 × 33-dimensional arrays, and there is no way to visualise them. Inspired by PointNet++, we pool descriptors for FPFH. After pooling, the data changed from 200 × 33 to 1 × 33. Average pooling, Max pooling, and Sum pooling are commonly used pooling methods. VFH is a viewpoint feature histogram, a global feature descriptor that retains the ability of FPFH to extract local features. The functions of VFH and FPFH have been integrated with the PCL [38] (Point Cloud Library) function library. The calculated descriptors are shown in Figure 19c–f.

Average Pooling: Calculate the average value of each dimension in 33 dimensions.

Max Pooling: Calculate the maximum value of each dimension in 33 dimensions.

Sum Pooling: Calculate the sum of each dimension in 33 dimensions.

The results of the feature descriptors calculated by VFH for the four types of bonding wires are shown in Figure 20. As shown in Figures 19 and 20, FPFH and VFH can extract different feature descriptors for different defective bonding wires. However, these descriptors are not directly used for defect pattern identification. Because of the high dimensionality of the features extracted by the FPFH and VFH, we cannot roughly determine the defect type of the bonding wire based on the feature histogram. Our method is designed for the defect pattern type of the bonding wire and is more suitable for the detection of bonding wire defects and the identification of defect patterns.

[IMAGE OMITTED. SEE PDF]

Accuracy of bonding wire defect detection

Experimental data preparation

We collected 170 bonding wire data, including 40 Normal wires, 30 Broken wires-Middle, 30 Broken wires-Single end, 30 Collapsed wires, and 40 Curved wires. The defect detection method proposed in section 3.3 is used to defect the experimental data. The relevant parameters are as follows M is 200, K is 20, and N is 10.

Comparison with the state-of-the-art methods

The existing detection methods include image processing-based methods (A vision detection system, VDS) [2] and deep learning-based methods (A data-driven framework, DF) [20]. We have chosen state-of-the-art methods to compare with our method. The evaluation metrics include the type of defects detected, whether samples are required, the detection accuracy, and the detection time.

First, as shown in Table 4, we analysed the types of defects that different methods can detect. Because the image-processing method cannot extract depth information from the image, it cannot detect Curved wires. Second, we also give quantitative results for the different methods, as shown in Table 5. The image-processing-based methods must compare the standard model image with the defect image to detect the defects. Deep learning-based methods also require a large number of sample images to complete the training of the model. Because these methods rely on sample images, their practical application is limited. In contrast, our method uses feature descriptors to accomplish defect detection, which does not require the use of samples. Finally, the detection accuracy was quantified. As shown in Table 5, the proposed method in this paper outperforms the current state-of-the-art methods in terms of accuracy. This is because the 3D model of bonding wires provides us with more defect information.

TABLE 4 Comparison of the types of defects detected by different methods.

Methods	Type of defect
Normal	Broken wire-Single end	Broken-wire-Middle	Curved	Collapsed
VDS	✓	✓	✓	×	✓
DFa	✓	✓	✓	×	✓
Our method	✓	✓	✓	✓	✓

TABLE 5 Quantification of bonding wire defect detection results.

Methods	Whether samples are required	Accuracy	Time-consuming (/s)
VDS	Yes	90.14%	0.02
DFa	Yes	93.64%	0.0782
Our method	No	96.47%	0.0532

We also analyse the time consumption of the different methods. The image-based processing method and our method require segmenting the bonding wires from the original image. Image segmentation reduces the detection time by removing unnecessary pixel data. The difference is that the image-processing-based method performs defect detection directly after completing image segmentation. Our method also requires 3D reconstruction and feature extraction. Therefore, our method takes a longer time. The deep learning-based method performs feature extraction and defect detection directly on the original image and takes the longest time. On balance, our method has more advantages, and its time consumption and detection accuracy can meet industrial applications.

Comparison with data-driven methods

To verify the effectiveness of the methods in this paper, we also compared the data-driven-based methods with our methods. The data-driven-based methods include SVM(Support Vector Machines), KNN(K-NearestNeighbor), MLP (Multi-Layer Perceptron), and RF(Random Forest). Three metrics in in Equation (27) were used to evaluate the performance of the detection methods. 27 $$\left\{\begin{array}{@{}rl@{}}\hfill & \text{Recall}=\frac{\text{TP}}{\text{TP}+\text{FN}}\hfill \\ \hfill & \text{Precision}=\frac{\text{TP}}{\text{TP}+\text{FP}}\hfill \\ \hfill & \text{Accuracy}=\frac{\text{TP}+\text{TN}}{\text{TP}+\text{FN}+\text{TN}+\text{FP}}\hfill \end{array}\right.$$ where TP and FN represent the number of defect-free samples that are predicted as defect-free and defective respectively. TN and FP represent the number of defective samples that are predicted as defective and defect-free respectively.

As shown in Table 6, the accuracy of all methods is greater than 90%. SVM has higher detection accuracy than the state-of-the-art methods. We think that building 3D point cloud models for bonding wires can help improve accuracy because 3D point clouds contain more defect information than 2D images. The data-driven model can learn more features from the 3D model to make more accurate decisions. In addition, the accuracy of our method is higher than that of the data-driven method. Because our method was developed after analysing the spatial structure of bonding wires, it is better suited for wires defect detection.

TABLE 6 Quantitative results with data-driven methods for defect detection.

Methods	TP	FN	TN	FP	Recall	Precision	Accuracy
SVM	38	2	122	8	0.950	0.826	0.9412
KNN	38	2	121	9	0.950	0.809	0.9353
MLP	32	8	125	5	0.800	0.865	0.9235
RF	28	12	125	5	0.700	0.848	0.9000
Ours	39	1	125	5	0.975	0.886	0.9647

Shortcomings of our method

Although our method can be used for the identification of defect patterns in bonding wires, it still has some shortcomings. The point cloud of the wire was divided into 10 parts, and we generated 10 descriptors for each wire. If the number of ‘+1’ or ‘−1’ in the descriptors of a Normal wire is less than 2, which is misclassified as a Broken wire-Single end, as shown in Figure 21a. If the number of ‘+1’ or ‘−1’ in the descriptor of a Broken wire-Single end is greater than 3, it will also be misclassified as a Normal wire, as shown in Figure 21b. In addition, when detecting Broken wires-Middle, our method does not detect defects when the interval between the two broken ends is less than μ + 3σ. In this case, we can readjust the threshold value so that the adjusted threshold value is less than μ + 3σ to obtain better detection results. The accuracy of the descriptor also affects the detection results. When the bending change of the bonding wire is not obvious, the algorithm may have missed detection. We can increase the number of descriptors to improve the accuracy of detection. However, this will add more computation time.

[IMAGE OMITTED. SEE PDF]

CONCLUSION

For the problems of bonding wire defect detection, this paper proposes a three-dimensional reconstruction and defect pattern recognition method based on stereo vision. First, to complete the 3D reconstruction of bonding wires, the OBB and MAD methods are used to locate the contours of bonding wires in the depth image. Second, a point cloud segmentation method based on spatial surface feature detection is proposed to complete the segmentation of the point cloud of the bonding wire. The proposed method is more suitable for segmenting bonding wires than other methods. Finally, this paper designs a detection process for bonding wires defect detection and proposes a descriptor that can be used for pattern recognition of multiple defects in bonding wires. The proposed method is also more interpretable in bonding wire detection than FPFH and VFH.

We provide a wealth of experiments on the proposed method. The results show that the method in this paper can accomplish the defect recognition of Normal wire, Curved wire, Broken wire-single end, Collapsed wire, and Broken wire-Middle, and the average recognition accuracy is about 96.47%. In the future, we expect to build a mathematical model of the bonding wire to solve the problem of misclassification during defect detection.

ACKNOWLEDGEMENTS

This research was supported by the Beijing Municipal Education Commission and Beijing Natural Science Foundation (No. KZ202010005004), the Natural Science Foundation of China (62076014) and the Intelligent Manufacturing and Robot Technology Innovation Project of Beijing Municipal Commission of Science and Technology and Zhongguancun Science and Technology Park Management Committee (NO. Z221100000222016).

CONFLICT OF INTEREST STATEMENT

The authors declare no conflict of interest.

DATA AVAILABILITY STATEMENT

The data that support the findings of this study are available on request from the corresponding author. The data are not publicly available due to privacy or ethical restrictions.

References

Yun, M., et al.: Bond wire damage detection method on discrete mosfets based on two‐port network measurement. Micromachines 13(7), [eLocator: 1075] (2022). [DOI: https://dx.doi.org/10.3390/mi13071075]

Qi, C.R., et al.: Pointnet++: deep hierarchical feature learning on point sets in a metric space. Adv. Neural Inf. Process. Syst. 30 (2017)

RaduRusu, B., et al.: Fast 3d recognition and pose using the viewpoint feature histogram. In: 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2155–2162. IEEE (2010)

RaduRusu, B., Cousins, S.: 3d is here: point cloud library (pcl). In: 2011 IEEE International Conference on Robotics and Automation, pp. 1–4. IEEE (2011)