Heterogeneous Signal Fusion Method in Driving Fatigue Detection Signals

Abstract

Driving fatigue is a physiological phenomenon that often occurs during driving. When the driver enters a fatigue state, they will become distracted and unresponsive, which can easily lead to traffic accidents. The driving fatigue detection method based on a single information source has poor stability in a specific driving environment and has great limitations. This work helps with being able to judge the fatigue state of the driver more comprehensively and achieving a higher accuracy rate of driving fatigue detection. This work mainly introduces research into different signal fusion methods to detect fatigue drive. This work will take the normal driver’s breathing signal, eye signals, and steering wheel signal as research objects and collect and isolate the characteristics of the fatigue detection signal. Research was then conducted on different signal fusion methods for the detected depth of breath. Change of steering angle, eyelid closure, and blinking marks and the fatigue driving experiment was designed to evaluate the results of different data fusion methods. Experimental results show that the detection accuracy of the heterogeneous signal fusion method in fatigue detection is as high as 80%.

This work is licensed under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.

1. Introduction

1.1. Research Background and Significance

With the continuous improvement of domestic infrastructure construction and the continuous improvement of people’s economic income level, the number of private cars has expanded significantly in recent years. Vehicles, such as cars and trains, still rely on the manual driving of the driver, and driving fatigue is a physiological phenomenon that often occurs during driving and is unavoidable [1]. In highway traffic, fatigue driving has become one of the main causes of traffic accidents. According to the statistics, more than 60% of drivers have had a long continuous driving experience. Among the major traffic accidents in the entire country, 27% of the drivers are fatigued and cause the accident [2]. Toyota has conducted investigations into the causes of traffic accidents. Among the three key factors: people, cars, and roads [3], the traffic accidents directly or indirectly caused by the “people” factor accounted for 92.9%, and the vast majority of traffic accidents are directly or indirectly related to the status of the driver [4]. The driver’s driving state directly affects the operating error rate and the ability to deal with emergencies. The fatigue that occurs during driving can cause the driver to become distracted and slow to react. Driving fatigue has become an invisible killer in road traffic and rail traffic. When the driver is tired and lethargic, the vehicle will not be able to control as expected. And the consequences are unpredictable and unexpected as well. This not only causes loss to one’s own life and property, but also brings great harm to pedestrians. Therefore, it is urgent to solve the problem of fatigue driving. To prevent accidents such as fatigue driving, it is necessary to develop effective fatigue detection methods; at this stage, fatigue detection is carried out by a single signal, and the detection of a single signal is one-sided. Therefore, in fatigue detection, research on how to detect fusion signals is imperative, which can detect driver fatigue in multiple directions and reduce driver fatigue.

1.2. Related Content

Fatigue can affect normal work and even cause accidents. To reduce the impact of fatigue on people, Wang proposed a method to provide real-time fatigue detection [5]. First, he uses an active shape model to detect human faces and extracts the histogram features of directional gradients of the eyes and mouth. Secondly, the support vector machine (SVM) is used to classify the state and posture of the orthography and scaling with iterations algorithm to estimate the head posture. Third, according to the face state, the fatigue decision index is obtained, and the weight of the fatigue decision index is calculated by the entropy method. Finally, based on the calculated fatigue decision index, he applies the Bayesian method to evaluate the driver’s fatigue level. The final average accuracy of his method is 83.3%. However, the operation of this method is more complicated. Li et al. explored a feature weight-driven signal fusion method [6] and proposed interactive mutual information modeling to improve the accuracy of mental workload classification. They used EEG and ECG signals to verify the proposed heterogeneous organisms the effectiveness of signal fusion methods. they invited ten subjects to participate in simple, medium, and difficult tasks to collect brain and ECG signals of different mental load levels. Then, they can be used for classification according to the heterogeneous physiological signals of different mental workload states. Their experiments show that the ECG can be used as a supplement to the EEG, optimize the fusion model, and improve the estimation of mental workload. The classification results show that the proposed biosignal fusion method IMIM can improve the classification accuracy of feature-level and classifier-level fusion. Their research shows that multimodal signal fusion is expected to identify the level of mental workload, and the fusion strategy has potential psychological workload estimation applications in cognitive activities in daily life. However, there are limitations of location in terms of use. The disadvantage is that only ten experimental subjects were selected, which is too small to have the validity of the experiment. Zheng et al. proposed a dynamic fatigue detection model based on hidden Markov model [7]. The model can use various physiological and detection information to estimate driver fatigue in a probabilistic manner. In the actual driving process, they simultaneously recorded electroencephalogram (EEG), electromyography (EMG), and breathing signals through wearable sensors and sent them to the computer via Bluetooth. Then, according to the physiological information, the distribution estimates of different time periods are used to obtain the fatigue probability. Their HMM-based fatigue identification method can dynamically obtain fatigue testing. However, the calculation amount of this detection method is too large, and the detection has greater difficulty. Pilataxi et al. introduced a driving assistance system that detects when the driver is drowsy [8]. The system is tested by a car-like robot that is wirelessly controlled by a computing interface developed in Visual Studio 2010, which simulates a car panel. An artificial vision system monitors the driver’s head direction to determine whether the driver is drowsy. However, this study did not use heterogeneous signal fusion methods.

1.3. Main Content and Innovation

The main content of this article is to study the respiratory physiological signals, driver operation signals, and eye signal detection in the fatigue driving detection, collect and extract the characteristics of the physiological signals and eye detection signals, and then compare the signal fusion methods. The best decision-level fusion method is selected for the fusion of heterogeneous signals, and finally the high accuracy of the heterogeneous information fusion method in the detection of fatigue driving is confirmed by the detection of fatigue driving signals. The innovation of this paper is to combine the fatigue driving detection signal with heterogeneous signal fusion and achieve a high accuracy rate of fatigue driving detection by fusing the collected heterogeneous signals.

2. Fatigue Driving Concepts and Methods

2.1. Definition of Fatigue Driving

To allow readers to better understand the meaning of driving fatigue, I will talk about some theoretical knowledge as the theoretical basis for the article. Fatigue driving refers to the phenomenon that the driver has physical and psychological disorders and decreased driving skills after driving for a long time. Fatigue is a very complex physiological phenomenon, which can generally be divided into mental fatigue and physical fatigue. Mental fatigue is manifested as restlessness, loss of motivation, difficulty in concentration, slow thinking, low mood, decreased work efficiency, prolonged reaction time, and decreased work accuracy. Its continued development will cause headaches, dizziness, insomnia, and dysfunction of the cardiovascular system, respiratory system, and digestive system. Mental fatigue and physical fatigue often occur at the same time [9, 10]. The driving fatigue discussed in this article refers to the phenomenon of slow response and misjudgment when the driver is driving. Driving fatigue is often a mixture of physical fatigue and psychological fatigue [11]. Continuous driving time is the most important and common cause of driving fatigue. According to the statistics of the literature, after 3 hours of continuous driving, most drivers will start to feel fatigue and start to experience operational errors. After continuous driving, the incidence of car accidents will rise to 1.5 times that of normal conditions. Therefore, the laws of various countries have strictly controlled the driving time of drivers, as shown in Table 1.

Table 1

The control of driving time by laws of various countries.

Country	Continuous driving time	Cumulative driving time (one day)
China	4 hours	8 hours
United States	None	10 hours
Japan	4 hours	9 hours
Australia	5 hours	12 hours
Europe	4.5 hours	10 hours

The generation of driving fatigue is also closely related to the driver’s physical and mental state when starting to drive. Factors such as the driver’s circadian rhythm, sleep time, age, and gender will all induce driving fatigue to varying degrees. The driving environment is also a factor in driving fatigue. Conditions such as temperature, humidity, oxygen content in the air, noise, and vibration of car seats are much worse than indoor environments. Unfavorable conditions driving fatigue usually occurs in driving environments [12, 13].

2.2. Introduction to Over-Limit Learning Machine

Over-limit learning machine is an algorithm in neural network research. It is a generalized single hidden layer feedforward neural network used for classification, regression, clustering, sparse approximation, compression, and single or multilayer hidden nodes. Feature learning [14, 15] is where the parameters of the hidden nodes (not just the weights connected to the input to the hidden nodes) do not need to be adjusted [16]. These hidden nodes can be randomly assigned and never updated (i.e., they are random projections, but have nonlinear transformations), or they can be inherited from their parents without being changed [17, 18]. In most cases, the output weights of hidden nodes are usually learned in a single step, which is equivalent to learning a linear model [19]. There will be no common problems such as falling into local minimum and overfitting in traditional neural networks, and it is suitable for modeling systems with complex nonlinear input and output relationships. Its advantage is that the learning speed is fast. For the algorithm applied to the product, it can greatly reduce the calculation amount, increase the calculation speed, and greatly improve the calculation speed [20, 21]. Its characteristic is that the weights of hidden layer nodes are randomly or artificially given and do not need to be updated. The learning process only calculates the output weights. Figure 1 shows the simple results and applications of over-limit learning.

[figure omitted; refer to PDF]

The ELM structure is as follows: suppose there are K pairs of input and output combinations, x is the feature vector used for training, and y is the corresponding output result, where i = 0 [22, 23]. Assuming it is an N-dimensional feature vector, at the same time, the number of hidden nodes in the overlimit learning machine is also set to N [24]. Let U be a unitary matrix of size N ∗ N in the first layer of ELM. f is the nonlinear activation function of ELM, and W is the weight vector of the second layer of ELM. At this time, the output of ELM can be expressed as$$\begin{array}{}\text{(1)}& fu\Lambda f_{n-1}{u}_{i}W=y.\end{array}$$

2.3. Introduction to Fractional-Order System

Fractional-order system is a generalization of the integer order system. Many physical processes can be modeled by fractal sequence systems such as diffusion and thermal conductivity, diffusion circuits, electrochemical processes, polarized dielectric viscous materials such as polymers and rubber, the release phenomenon of organic dielectric materials, flexible structure data network and biological traffic, etc. [25, 26]. This article will be used in data modeling. The general fractional order system can be expressed as$$\begin{array}{}\text{(2)}& yt+{\displaystyle \sum _{i=1}^{n}ayt}={\displaystyle \sum _{i=1}^{m}bDt}.\end{array}$$

According to the definition of Miller et al., the generalized form of the fractional difference is defined as$$\begin{array}{}\text{(3)}& D_k^{a}fk=\lim \frac{1}{h}{\displaystyle \sum _{m=0}^a{-1}^m\begin{array}{c}a\\ m\end{array}},\end{array}$$where h is the sampling period, where it is defined as$$\begin{array}{}\text{(4)}& \begin{array}{c}a\\ m\end{array}=\frac{Ka+1}{Ka-m+1Km+1},\end{array}$$where K(a) is defined as$$\begin{array}{}\text{(5)}& Ka={\displaystyle {\int }_0^{+\infty }{\varsigma }^{z-1}{e}^{-\zeta }d}.\end{array}$$

In actual calculations, to simplify the calculations, matrix operators can be used to approximate and implement fractional differential operations; namely.$$\begin{array}{c}\text{(6)}& \begin{array}{c}Dfh\\ M\\ Dfkh\end{array}=\frac{1}{h}\begin{array}{ccc}{w}_0& \Lambda & 0\\ M& w_0& 0\\ w_k& \Lambda & w_0\end{array},w_k={-1}^k\begin{array}{c}a\\ k\end{array}.\end{array}$$

Due to the edge effect and the truncation effect [27, 28], equation (6) cannot always start from k = 1. However, each row of the fractional differential operator matrix can be regarded as the weight of the input x. When k > 1 and a is not an integer, it will approach 0 [29, 30]. Therefore, a suitable upper limit K needs to be selected. At this time, formula (6) can be rewritten as$$\begin{array}{}\text{(7)}& \begin{array}{c}Dfkh\\ M\\ DfTh\end{array}=\frac{1}{h^a}\begin{array}{ccc}{w}_{k-1}& \Lambda & 0\\ M& w_0& 0\\ 0& \Lambda & w_0\end{array}.\end{array}$$

3. Feature Extraction and Heterogeneous Signal Fusion in Fatigue Detection

3.1. Physiological Signal Feature Collection

3.1.1. Working Principle of Detecting Physiological Signals

In this paper, continuous wave radar is selected as the detection sensor. The typical block diagram of continuous wave radar is shown in Figure 2, including power divider, transmitting antenna, receiving antenna, oscillator, mixer, detector, and other circuits. The transmitting antenna transmits the microwave signal in a directional direction, and it is reflected back after encountering obstacles and received by the receiving antenna. After being mixed with the oscillator, the low-frequency signal is detected by the mixer. When using continuous wave radar to detect the human thoracic cavity, the movement of the thoracic cavity will produce frequency modulation. According to the Doppler effect, the reflected radar signal will also have a corresponding frequency shift, so the obtained low-frequency signal contains the movement information of the thoracic cavity.

[figure omitted; refer to PDF]

Human body thoracic cavity that is used as a target for detecting the radar emission signal returned by the thoracic cavity movement will generate frequency modulation; the phase information related to the thoracic cavity movement can retrieve the modulation data. Can be late, the phase data can reflect the tester’s breath and heartbeat. Doppler radar has a good ability to penetrate clothing or bedding and can achieve noncontact detection. However, for the radio frequency, whether the signal is in the air or on the surface of the skin, there is reflection loss, which results in a strong correlation between the performance of the radar and the frequency. Therefore, the choice of radar sensor is a key indicator, which directly affects the quality of the experimental results. From the theory of continuous wave radar, it can be known that the smaller the wavelength of the radar in the output baseband signal B(t), the more obvious the displacement change of the detected chest cavity, which means that the higher the frequency of the selected radar, the better. According to the electromagnetic field theory, the higher the radar frequency, the smaller the signal reflection after encountering the human body surface, the weaker the ability to penetrate clothing, and the greater the energy consumption of transmitting signals of the same power. It can be seen that the frequency selection of radar is a contradictory process. According to the reasons such as penetration ability, volume, and transmission power, in this article, the 10.525 GHz radar module HB100 is selected as the front-end detection sensor. This module is a microwave moving object detector designed by using the Doppler effect. The actual sample is shown in Figure 3. The radar module is mainly used for automatic door switches, safety precaution systems, automatic train signals, and other occasions. The internal integrated transmission and receiving antennas, oscillators, mixer, detectors, and other modules have the advantages of strong antiinterference ability, low output power, and long detection distance.

[figure omitted; refer to PDF]

The maximum resistance value of the rotary potentiometer is 5 KΩ. The actual change range of the potentiometer is 500 Ω-5 KΩ when the steering wheel rotates 0–1080 degrees. Considering the power consumption of the circuit and the signal change range, the voltage divider resistance is 5 KΩ, and the analog signal output change range is 0.45 V–2.5 V. It can be seen that the resolution of the rotary encoder is very low, but the experiment only needs to record the changing trend and position of the steering wheel angle, and there is no need to have high requirements for accuracy.

3.2.2. Reverse Disk Angle Signal Correlation Quantity

In this paper, the fatigue driving detection uses a rotary potentiometer to record the rotation track of the entire steering wheel. When the steering wheel rotates clockwise, the sampled voltage signal value will continue to increase, and when the steering wheel rotates counterclockwise, the sampled voltage signal value will continue to decrease. Therefore, it can be judged whether the current road condition is a continuous curve, a large curve, a small curve, or a straight road according to the change trend of the sampled voltage signal. In this study, the voltage value can be used directly instead of the angle value to calculate. Because the analysis is only concerned with the change trend of the steering wheel angle of the angle sensor, and the change of the small angle, the Hall angle sensor outputs the voltage value corresponding to the angle.

(1) The steering wheel angular velocity V can be expressed as$$\begin{array}{}\text{(10)}& Vt=\frac{{\theta }_i-{\theta }_{i-1}}{t_i-t_{i-1}}.\end{array}$$

  Among them, θ is the digital number of the output conversion of the voltage value corresponding to the steering angle of the steering wheel at t.

(2) The standard deviation of the steering wheel angular velocity can be expressed as$$\begin{array}{}\text{(11)}& w_t=\sqrt{\frac{1}{n-1}{\displaystyle \sum _{i=1}^n{v}_i-v_{\text{ave}}}}.\end{array}$$

  Among them,$$\begin{array}{}\text{(12)}& v_{\text{ave}}=\frac{1}{N}{\displaystyle \sum _{i=1}^n{v}_i}\text{,}\hspace{1em}N\text{is\hspace{0.17em}the\hspace{0.17em}number\hspace{0.17em}of\hspace{0.17em}samples}\text{.}\end{array}$$

(3) The standard deviation of the steering angle of the steering wheel can be expressed as$$\begin{array}{}\text{(13)}& w=\sqrt{\frac{1}{N-1}{\displaystyle \sum _{i-1}^n{\theta }_i-{\theta }_{\text{ave}}}}.\end{array}$$

Calculate the steering angular velocity, angular velocity standard deviation, and angle standard deviation per unit time during the driving process of the driver, and compare it with expert scoring standards to analyze the correlation between the degree of driving fatigue and the parameter information of the driving vehicle.

3.3. The Driver’s Eye Feature Point Collection Based on the Over-Limit Learning Machine

Eye feature point positioning belongs to the category of face alignment, which refers to finding various facial feature points (landmarks) from the detected face, such as key positions such as eyes, eyebrows, nose, mouth, and facial contours. Figure 5 shows the facial feature point model.

[figure omitted; refer to PDF]

As shown in Figure 6, the feature points are distributed in the lines in the figure, where the eye area contains the main information of the eyes, and the schematic diagram of the eye feature points is shown in Figure 6, including 2 points on the upper eyelid, 2 points on the lower eyelid, and 2 points on the left and right corners of the eye.

[figure omitted; refer to PDF]

Over-limit learning machine is an algorithm in neural network research, used for classification, regression, clustering, sparse approximation, compression, and feature learning of single or multilayer hidden nodes. The algorithm builds a cascaded residual regression tree, thus making the face gradually return to the true position from the initial value of the feature point. Use the super learning machine to align the detected face area to find the position of the face. The detection result is shown in Figure 7. The eyelid aspect ratio can characterize the degree of eyelid opening. We select 3 typical eyelid closure types for analysis.

[figure omitted; refer to PDF]

For various eyelid states, the eyelid height-to-width ratio can be used to represent it. The eyelid height-to-width ratio can reflect the various transformation characteristics of the driver’s eyes in a time series. The blink frequency refers to the number of blinks of the driver in a unit time. Under normal circumstances, the driver’s blinking frequency remains within a relatively stable range during normal driving. When the driver is in a fatigued driving state, the blink frequency will be significantly reduced or significantly increased, which is very unstable. Eye-closing speed refers to the time it takes for the eyelids to go from a normal open state to a fully closed state. The shorter the time, the faster the eyelid closure speed, and the shorter the time, the slower the eyelid closure speed. The speed of closing the eyes of the driver during normal driving is much faster than during fatigue driving. When the driver is in a state of fatigue, the closing speed of the eyelids is very slow.

3.4. Heterogeneous Signal Fusion Method for Fatigue Driving Detection

Driving fatigue is a complex physiological and psychological phenomenon. The single-modal driving fatigue detection method cannot fully characterize the driver’s fatigue. And in actual driving scenarios, due to various environmental interferences (such as unstable lighting conditions in the driving environment, vehicle vibration, spatial electromagnetic interference, etc.), the detection method based on a single information source is not reliable. This section discusses the combination of normal driver respiration data, eye data, and steering angle parameters and judge the overall fatigue condition of the driver, so as to achieve higher driving fatigue detection accuracy, stability, and environmental adaptability. Differential fusion means a complete preprocessing process. Data entry, forecasting, and arbitration decisions are based on information from many different sources to achieve more accurate, stable, and reliable target data from a single source.

3.4.1. Type of Signal Fusion

The heterogeneous signal fusion method actually uses the data of multiple sensors to obtain complete information about the object or the environment. Its core part is the fusion algorithm, and different fusion algorithms have different advantages. There are many information fusion algorithms that have been proposed. According to the different levels of data abstraction, they can be divided into three levels: data layer fusion, feature layer fusion, and decision layer fusion. The integration of these three levels has its own advantages. As shown in Figure 8, the data layer fusion is to directly perform the fusion without any processing and processing of the raw data from the sensors of the same category, and then perform feature extraction with the data obtained after the fusion, and finally obtain the discrimination result. This method belongs to the lowest level of fusion, and there is no data loss. However, this fusion method has considerable limitations. It has certain restrictions on multiple sensors. The sensors to be fused must be of the same type, and the consistency of time and space must be ensured before fusion. This means that the increase in the amount of calculation will result in slow processing speed and poor real-time performance.

[figure omitted; refer to PDF]

Compared with the other two methods, the feature-level fusion belongs to the middle-level fusion. Different from the data-level fusion, it first extracts the corresponding features from the original data collected by different sensors, performs certain processing on the extracted features, and then fuses them. Finally, the model is modeled to obtain the discrimination result. The fusion process is to fuse the features extracted from the original data, reducing a lot of calculations, improving the transmission speed and processing speed, and has a certain real-time performance, but compared with data-level fusion, a certain amount of information will still be lost. This leads to a decrease in accuracy. Feature layer fusion is shown in Figure 9.

[figure omitted; refer to PDF]

Decision-level fusion is the highest level method among the three fusion methods. Unlike the previous two, it uses multiple sensors to monitor an object at the same time. However, unlike the data level, different types of sensors can be used, and each sensor is processed independently. That is, the model is established after the features are extracted, and the decision result is obtained, and then all decision results are merged and judged to obtain the final decision result. The decision fusion method is aimed at the decision results of different sensors, which makes it more computationally intensive and more real-time. However, it has undergone multiple layers of information extraction, and the information loss is more serious, which greatly reduces the accuracy of the fusion result. Therefore, the heterogeneous signal fusion in this paper adopts the decision-making layer fusion calculation, and the decision-making layer fusion is shown in Figure 10.

[figure omitted; refer to PDF]

Statistical recognition rates of fatigue driving samples were 81%, 75%, 79%, 76%, and 76%, respectively. Judging by the predictions of the Extreme Learning Machine, the detection rate of the alarm state was as high as 81% of the steering wheel signal under varying degrees of fatigue in the steering wheel change trend. The signal currently predicts the level of fatigue while driving. In drowsy conditions, the range of steering angle changes is gradually reduced and the number of settings is significantly reduced. The depth of breathing in the normal signals of the driver is accompanied by a deeper sleep. The depth and width of the breath and the frequency gradually decrease.

Figure 13 shows the blink frequency per unit time calculated from the electrooculogram signal. This article chooses to use 1 minute as the unit. It can be clearly seen that as the experiment progresses, the blink frequency of the five subjects has increased significantly.

[figure omitted; refer to PDF]

Based on the above statistical results, it can be seen that there is a certain correlation between blinking frequency and fatigue. The higher the blinking frequency, the more fatigued. Participants blinked more frequently when they entered a state of fatigue. In addition, it can be seen from Figure 13 that as fatigue deepens, the blink frequency of many subjects has a certain downward trend relative to the maximum value.

Figure 14 shows the eyelid distance per unit time calculated from the eye movement signal. The unit of 1 min is chosen. It can be clearly seen that as the experiment progresses, the eyelid distance gradually decreases in all five subjects.

[figure omitted; refer to PDF]

From the above figure, it appears that the degree of eyelid closure can be used as a relative indicator to determine fatigue. In addition, Figure 14 shows that the fourth person sleeps in 15 minutes, when their eyelids are closed, and woke up after 16 minutes. Although different from the data of other volunteers, the general trend of change for each person, of course, is a downward one.

In summary, the experimental results show that the fatigue detection method of heterogeneous signal fusion can improve the accuracy of fatigue detection while avoiding the impact on driver behavior.

5. Discussion

This paper firstly collects and extracts features from physiological signals, driver operation signals, and eye signals in fatigue detection, and then performs signal fusion on the extracted signals to achieve a higher accuracy rate of fatigue driving detection, and finally performs signal fusion. In the detection experiment of the method, it is concluded that the accuracy of the fatigue driving detection method of the heterogeneous signal fusion method is higher than that of the driving fatigue detection method of a single signal source through the statistics of the super learning machine algorithm. The research of this article has not enough understanding of the signal fusion method, but the research of this article has certain reference value for the detection method of fatigue driving. It provides a certain research route for future fatigue driving detection methods. This article still has some shortcomings. For example, the number of samples selected in our research process is not very large, and the data obtained is often not comprehensive enough. Driving fatigue is a problem that drivers often face. Combining with heterogeneous signal fusion methods can certainly make the research on driving fatigue more in-depth.

Acknowledgments

This work was supported by the Project of Department of Education, Jiangxi Province (no. GJJ191000).

References

[1] A. Němcová, Y. Tian, H. Jia, "Driving fatigue detection based on feature fusion of information entropy," Journal of Computational Methods in Science and Engineering, vol. 18 no. 8,DOI: 10.3233/JCM-180839, 2018.

[2] Z. Liu, W. Zhang, S. Lin, T. Q. S. Quek, "Heterogeneous sensor data fusion by deep multimodal encoding," IEEE Journal of Selected Topics in Signal Processing, vol. 11 no. 3, pp. 479-491, DOI: 10.1109/jstsp.2017.2679538, 2017.

[3] R. Fu, H. Wang, W. Zhao, "Dynamic driver fatigue detection using hidden Markov model in real driving condition," Expert Systems with Applications, vol. 63 no. 5, pp. 397-411, DOI: 10.1016/j.eswa.2016.06.042, 2016.

[4] L. Wang, C. Zhang, X. Yin, R. Fu, H. Wang, "A non-contact driving fatigue detection technique based on driver’s physiological signals," Qiche Gongcheng/Automotive Engineering, vol. 40 no. 3, pp. 333-341, 2018.

[5] A. Němcová, O. Janoušek, M. Vítek, I. Provazník, "Testing of features for fatigue detection inEOG," Bio-Medical Materials and Engineering, vol. 28 no. 4, pp. 379-392, DOI: 10.3233/BME-171683, 2017.

[6] G. Li, B. Li, G. Wang, J. Zhang, J. Wang, "A new method for human mental fatigue detection with several EEG channels," Journal of Medical and Biological Engineering, vol. 37 no. 2, pp. 240-247, DOI: 10.1007/s40846-017-0224-6, 2017.

[7] C. Zheng, B. Xiaojuan, W. Yu, "Fatigue driving detection based on Haar feature and extreme learning machine," The Journal of China Universities of Posts and Telecommunications, vol. 23 no. 4, pp. 91-100, DOI: 10.1016/s1005-8885(16)60050-x, 2016.

[8] J. Pilataxi, W. Vinan, D. Chavez, "Design and implementation of a driving assistance system in a car-like robot when fatigue in the user is detected," IEEE Latin America Transactions, vol. 14 no. 2, pp. 457-462, DOI: 10.1109/tla.2016.7437179, 2016.

[9] M. Kolodziej, P. Tarnowski, D. J. Sawicki, A. Majkowski, R. J. Rak, A. Bala, A. Pluta, "Fatigue detection caused by office work with the use of EOG signal," IEEE Sensors Journal, vol. 20 no. 24,DOI: 10.1109/jsen.2020.3012404, 2020.

[10] J. Yin, J. Hu, Z. Mu, "Developing and evaluating a mobile driver fatigue detection network based on electroencephalograph signals," Healthcare Technology Letters, vol. 4 no. 1, pp. 34-38, DOI: 10.1049/htl.2016.0053, 2017.

[11] F. You, Y. Gong, Y. Gong, X. Li, H. Wang, "R2DS: a novel hierarchical framework for driver fatigue detection in mountain freeway," Mathematical Biosciences and Engineering, vol. 17 no. 4, pp. 3356-3381, DOI: 10.3934/mbe.2020190, 2020.

[12] Z. K. Gao, Y. L. Li, Y. X. Yang, C. Ma, "A recurrence network-based convolutional neural network for fatigue driving detection from EEG," Chaos, vol. 29 no. 11,DOI: 10.1063/1.5120538, 2019.

[13] L. G. Hernandez-Rojas, E. Martinez, J. M. Antelis, "Detection of emergency braking intention using driver’s electroencephalographic signals," IEEE Latin America Transactions, vol. 17 no. 01, pp. 111-118, DOI: 10.1109/tla.2019.8826702, 2019.

[14] M. Hanamura, T. Sawada, T. Serizawa, "In-paper self-assembly of cellulose oligomers for the preparation of all-cellulose functional paper," ACS Sustainable Chemistry & Engineering, vol. 9 no. 16, pp. 5684-5692, DOI: 10.1021/acssuschemeng.1c00815, 2021.

[15] Y. Zhu, J. Li, X. Lin, X. Huang, M. R. Hoffmann, "Single-cell phenotypic analysis and digital molecular detection linkable by a hydrogel bead-based platform," ACS Applied Bio Materials, vol. 4 no. 3, pp. 2664-2674, DOI: 10.1021/acsabm.0c01615, 2021.

[16] H. Sun, S. P. Chen, L. P. Xu, "Research on cloud computing modeling based on fusion difference method and self-adaptive threshold segmentation," International Journal of Pattern Recognition and Artificial Intelligence, vol. 32 no. 6, pp. 1859010.1-1859010.15, DOI: 10.1142/s0218001418590103, 2018.

[17] W. Yan, J. Tan, H. Zhan, H. Wang, "Research on the method of fault diagnosis based on multiple classifiers fusion," International Journal of Hospitality Information Technology, vol. 9 no. 2, pp. 195-202, DOI: 10.14257/ijhit.2016.9.8.32, 2016.

[18] X. J. Qin, Z. J. Duan, H. B. Zheng, Y. Tang, "Efficient smoothness-preserving fusion modelling method for mesh models," International Journal of Simulation Modelling, vol. 16 no. 3, pp. 527-540, DOI: 10.2507/ijsimm16(3)co14, 2017.

[19] C. Ren, Y. J. Liang, X. J. Lu, H. B. Yan, "Research on the soil moisture sliding estimation method using the LS-SVM based on multi-satellite fusion," International Journal of Remote Sensing, vol. 40 no. 5-6, pp. 2104-2119, DOI: 10.1080/01431161.2018.1475778, 2019.

[20] S. Hai-jiao, L. Wei-ning, W. Jia-cheng, L. Pei-xun, L. Zhi-gen, D. Yao-hua, "Data fusion method in multi-sensors autonomous tracking," Chinese Journal of Liquid Crystals and Displays, vol. 31 no. 8, pp. 801-809, DOI: 10.3788/yjyxs20163108.0801, 2016.

[21] L. Mingyu, C. Chi, C. Ching-Hsiang, L. Wing, "A Gaussian process data modelling and maximum likelihood data fusion method for multi-sensor CMM measurement of freeform surfaces," Applied Sciences, vol. 6 no. 12, 2016.

[22] C. Li, X. Yang, "Multifocus image fusion method using discrete fractional wavelet transform and improved fusion rules," Journal of Modern Optics, vol. 68 no. 11,DOI: 10.1080/09500340.2021.1890250, 2021.

[23] X. S. Yang, S. W. Lee, "A study on clothing pattern optimization fusion method using 3D program," The Korean Society of Science & Art, vol. 38 no. 5, pp. 317-327, DOI: 10.17548/ksaf.2020.12.30.317, 2020.

[24] X. Jing, S. Li, J. Cheng, G. Junjun, "Multidimensional situational information fusion method for energy saving on campus," Journal of Intelligent and Fuzzy Systems, vol. 38 no. 4,DOI: 10.3233/jifs-191513, 2020.

[25] Q. Chen, Z. Wang, Y. Chai, "Multi-focus image fusion method based on improved VGG network," Journal of Applied Optics, vol. 41 no. 3, pp. 500-507, 2020.

[26] E. A. Veshkin, V. I. Postnov, V. V. Semenychev, A. A. Barannikov, "Study of the properties of carbon plastic samples formed by the infusion method," Industrial Laboratory. Diagnostics of Materials, vol. 86 no. 3, pp. 39-43, DOI: 10.26896/1028-6861-2020-86-3-39-43, 2020.

[27] G. Ravikanth, K. Sunitha, B. Eswara Reddy, "Location related signals with satellite image fusion method using visual image integration method," Computer Systems Science and Engineering, vol. 35 no. 5, pp. 385-393, DOI: 10.32604/csse.2020.35.385, 2020.

[28] H. Park, "An RGB-NIR image fusion method for improving feature matching," International Journal of Engineering and Technology Innovation, vol. 10 no. 3, pp. 225-234, DOI: 10.46604/ijeti.2020.5177, 2020.

[29] J. Cheng, C. Cai, X. Tang, V. S. Sheng, W. Guo, M. Li, "A DDoS attack information fusion method based on CNN for multi-element data," Computers, Materials & Continua, vol. 62 no. 3, pp. 131-150, DOI: 10.32604/cmc.2020.06175, 2020.

[30] T. Smets, T. De Keyser, T. Tousseyn, E. Waelkens, B. De Moor, "Correspondence-aware manifold learning for microscopic and spatial omics imaging: a novel data fusion method bringing mass spectrometry imaging to a cellular resolution," Analytical Chemistry, vol. 93 no. 7, pp. 3452-3460, DOI: 10.1021/acs.analchem.0c04759, 2021.