Content area
Safety is a challenge in human machine collaboration despite of the advantages in achieving efficiency, cost reduction and productivity in a collaborative scenario between human and machine/robot. During collaboration with machines, the user might not be able to follow the collaborative tasks as expected due to the cognitive burden causing potential safety concerns such as collision. Addressing this challenge, the aim of this paper is to explore the potential of on-body sensing systems in study of user experience and the psychological condition during the collaboration between machines and human. As the psychological condition is reflected in physiological signals, sensing technologies and signal processing techniques to extract features from physiological signals are explored with applicability in human machine collaboration scenarios. An experiment is designed utilising an industrial collaborative robot arm while quantitative and qualitative data is gathered for this purpose exploring the problem to study user experience and impact of mental strain and cognitive workload on user performance and experience during human machine collaboration. Results show that an adaptive machine to user experience measured by on-body sensing systems during the collaboration has the potential to address safety in human machine collaboration while improving performance and user experience.
Introduction
Human machine collaboration (HMC) in which human and machines share their skills in a collaborative setting is supported in Industry 4.0 aiming to achieve efficiency, cost reduction and productivity increase in automation; however, safety-related concerns in such collaborative environment would be a challenge. During HMC, machines/robots contribute from high levels of accuracy, speed and repeatability, while human workers contribute from flexibility and cognitive skills perspectives (Villani et al. 2018). Utilising the mentioned advantages of HMC, there is need to consider safety in the working environment. Physical or sensor-based barriers are used to be utilised ensuring the user’s safety; however, in collaborative scenario, such barriers are eliminated due to the shared working environment, but other safety mechanisms could be used to prevent harming the human user or collaborator.
To regulate the safety of industrial robots, safety standards are in place to provide requirements and design guidelines including International Organization for Standardization (ISO) 10218 which has two parts. ISO 10218–1 is related to the robot manufacturers’ safety requirements addressing the design of robot and its controller while ISO 10218–2 is intended for system integrators and defines the safety requirements for an industrial robot system (ISO 2011a; ISO 2011b). The standard is adopted by the European Community while the USA follows the American National Standard Institute/Robotic Industrial Association (ANSI/RIA) R15.06 and Canada follows Canadian Standards Association (CAN/CSA)-Z434 standard, which have been both updated with the two parts of ISO 10218.
In ISO 10218, four collaboration modes are identified including “Safety-rated Monitored Stop”, “Hand Guiding”, “Speed and Separation Monitoring” and “Power and Force Limiting” modes. For “Safety-rated Monitored Stop”, the human and robot work in a collaborative workspace but not at the same time; for “Hand Guiding”, the operator can move the robot to teach the robot positions without the need for an interface whereas the robot programme will be interrupted and once the operator leaves the collaborative area the robot resumes previously interrupted program. For the “Speed and Separation Monitoring” mode, the human presence is allowed within the robot space through safety-rated monitoring sensors enabling different operation speeds depending on the distance from the robot, the closer the distance the lower the speed. For the “Power and Force Limiting” mode, a human worker can work side-by-side with the robot while limiting motor power and force to handle collisions utilising force sensor to detect the contact (Villani et al. 2018).
Addressing safety in HMC scenarios, literature review reports the need to consider the cognitive workload of the user as there could be mental strain caused by close and long interaction with the robot during a collaborative robotic scenario (Landi et al. 2018). Mental strain and cognitive workload during the collaboration with the robot could cause the user not to be able to follow the collaborative tasks leading to potential safety concerns such as collision. Therefore, by considering the user experience into the robot control modes during the collaboration, an adaptive interaction will be enabled leading to less mental strain and better user experience.
User experience, which is closely related to the user’s psychological condition, is reflected in physiological signals, and can be predicted through analysing such bio-signal changes (Landi et al. 2018). For example, physiological arousal is used to measure attention, alertness, anxiety, frustration, stress and cognitive processing which is represented in bio-signals. The changes in bio-signals includes increase in blood pressure, respiration rate, constricted blood vessels, elevated body temperature, muscle spasms, raised blood flow to muscles and decreased blood flow to the skin (Matthews et al. 2020). Therefore, such parameter changes inform studies for the purpose of user experience recognition in interactive systems through detecting changes in arousal. The user’s physiological signals have the potential to be collected through the embedded sensors in wearable devices or the devices that the user commonly interact through them with the machine. Utilising wearable sensors to collect physiological signals for this purpose has the advantage of no need for camera or dependency to laboratory setup. In addition, the signals provide information for more reliable prediction of user psychological aspects in comparison to the other modalities such as facial expressions and body gesture and posture, which are commonly used in literature as social masking through these signals is impossible. For this purpose, wearability feature imposes the need for the weight and the size, which is required to be kept low and small, respectively (Haratian et al. 2016).
The aim of this paper is to explore the potential of on-body sensing systems in study of user experience addressing safety of human robot collaboration. For this purpose, the related safety mechanisms and standards were reviewed and how safety would be impacted by user experience including mental strain and cognitive workload during human robot collaboration were discussed. To study user experience closely related to psychological condition, which is reflected in physiological signals, different bio-signals of human body system will be reviewed along with the relevant sensor technologies to measure them with the potential to be utilised for addressing safety of HMC. The study includes a review of the relevant bio-signal processing techniques and the related features. A HMC scenario is designed to study impact of mental strain and cognitive workload on user performance and user experience in addressing safety during HMC informed by the background review of the study while quantitative and qualitative data are gathered for this purpose exploring the problem. The study shows that an adaptive machine to user experience measured by on-body sensing systems during collaboration has the potential to address safety in human machine collaboration while improving performance and user experience.
In this paper, after exploring the background regarding the sensor technologies in collecting physiological signals within the context of a human machine collaboration, the signal processing techniques to extract the features related to user experience are discussed in Sect. 2. Later, in the Methodology section, the approach to collect physiological signals and user experience in a designed human machine/robot collaboration scenario is explored. The collected qualitative and quantitative data regarding the collaboration scenario are analysed and discussed in the Result section and the paper is concluded in Sect. 5 summarising the findings.
Background of on-body sensor technologies and signal processing techniques
Human body’s signals (bio-signals) are collected and processed through various sensor technologies and signal processing techniques which are studied in this section in order to measure mental strain, and user experience with application in human machine collaboration. Psychological condition and experiences of the user interacting with machines including cognitive burden and distress are reflected in physical and physiological signals of body organ systems including nervous, cardiovascular, muscular, integumentary and respiratory system. To study the mutual interaction of psychological condition and physiological changes, there is need to consider the bio-signal changes related to each body organ system and how to measure them through appropriate sensor technologies and signal processing techniques which are explored in this section.
Nervous system
The nervous system collects, processes and transfers information from different body parts that includes the central nervous systems (CNS) and the peripheral nervous systems (PNS). CNS comprises the brain and spinal cord whereas the brain signals are collected through electroencephalography (EEG). EEG signals, which are electrical potentials at the surface of the brain or outer surface of the head, are collected through electrodes attached to the scalp. Electrical potentials of brain can also be measured through inserting electrodes into the brain using catheter whereas the approach is intrusive, risky and expensive due to the need for professionals to perform it in operation rooms or laboratories. Surface mount electrodes to measure EEG signals for the purpose of user experience recognition during human machine collaboration which requires to be wearable and non-intrusive could be considered as being wearable but cumbersome and might interfere with the user activities. EEG signals have commonly known features such as power spectral density (PSD), differential entropy (DE), differential asymmetry (DASM), rational asymmetry (RASM), asymmetry (ASM) and differential causality (DCAU) features. Specifically, PSD is computed using Short Time Fourier Transform (STFT); DE is equivalent to the logarithmic power spectral density for a fixed length EEG sequence; DASM and RASM features are the differences and ratios between the DE features of hemispheric asymmetry electrodes; ASM features are the direct concatenation of DASM and RASM features.
Further to the CNS in nervous system, the PNS comprises the nerves outside the brain and spinal cord and includes the autonomic (ANS) and the somatic nervous systems (SNS). The ANS operates between the CNS and various internal organs, such as heart, lungs, viscera and glands while SNS is associated with the voluntary control of body movements. Psychological conditions reflected in physiological signals are mainly in response to the CNS and the ANS of human body which is connected to different body organ systems including the cardiovascular, muscular, integumentary and respiratory systems (Saltzman 2015) which are studied in the following sub-sections.
Cardiovascular system
Cardiovascular system including heart and blood vessels circulates blood in the human body while psychological conditions could have impact on its operation reflected in the related physiological signals including heart signal and blood pressure. Traditionally, stethoscope is used to listen to the beating heart and sphygmomanometer is used to estimate the blood pressure within the body vessels which are requiring a skilled operator to worked with them. Heart functionality can be monitored using electrocardiography (ECG) to measure electrical potentials produced by cardiac muscle. The signal can be measured invasively by inserting recording electrodes mounted on a catheter into the heart which is dependent to laboratory and skilled professionals. Electrocardiography provides a non-invasive and reliable method to measure the cumulative cardiac electrical activity using skin electrodes attached to the body surface capturing the external currents around cardiac muscle cells. The ECG waveform is displayed as voltage and is a function of time that is analysed in segments comprising a waveform called PQRST wave: a small P wave, a short Q delay, a large QRS wave, a second delay and a small T wave whereas the phases of the cardiac cycle are correlated with the structure of the wave. Collecting ECG signals within the context of user experience recognition during human machine interaction could be considered due to wearability of the system but needs to consider that the surface mount probes could be uncomfortable to wear on the chest and might interfere with the user activities (Saltzman 2015).
Blood pressure, which is systolic pressure over diastolic pressure, provides information about the cardiovascular system. Systolic pressure is the maximum pressure exerted on the arterial walls during contraction of the ventricles of the heart while diastolic pressure occurs during the relaxation of the ventricles. It can be measured directly through catheterisation within a blood vessel by inserting catheter to transduces the atrial pressure to an external pressure transducer. The method is invasive and dependent to laboratory or operation room requiring highly skilled operator. Commonly blood pressure is measured using sphygmomanometer whereas an air-filled cuff is wrapped around the arm or wrist and the sound regarding systolic and diastolic phases are monitored either using stethoscope or microphone. Pressure sensors such as strain gauge or piezoelectric sensors placed in the cuff can detect fluctuation as the cuff is deflating. Within the context of user experience recognition during human machine collaboration, blood pressure measurement could be an option due to the wearability of the system but needs to consider that the cuff could be uncomfortable to wear on the arm and might interfere with the user activities.
Heartbeat rate can be measured through blood volume pulse using photoplethysmography (PPG), which estimates artery volume using light. The sensor radiates light onto the skin and the reflected light is detected by a photodiode. The amount of light that is reflected back depends on the volume of arteries near the skin’s surface in response to the blood pulsation. Red and infrared light moves through the skin easier leading to more in-depth penetration; however, blue light does not travel well into the body therefore reaching the small blood vessels at the surface where the pulse wave is at its weakest. Green light moves through the skin better than blue light, but worse than red which provides a better contrast signal which is easy to detect and is less sensitive to motion errors than red light. Within the context of user experience recognition during human machine collaboration, heart rate measurement using PPG could be considered due to wearability of the system and being comfortable to wear with less chance of interfering with the user activities.
Electrocardiogram (ECG) which is a measure of heart activity has features including heart rate (HR) and heart rate variability (HRV). (Raez et al. 2006) Heart rate is indicator of the number of times per minute that the heart contracts or beats whereas heart rate variability (HRV) is the variation between consecutive heart beats over time period and dependent on the regulation of the heart rate. It indicates the heart’s ability to adapt to the changing circumstances by detecting and quickly responding to unpredictable stimuli (Landi et al. 2018). The R peak of heart signals as the marker of each beat is denoted by Rk as the instant of occurrence of the kth heartbeat, whereas the RR series is derived from RRk = Rk+1 − Rk, k = 1,2, …. A low HRV can indicate a state of relaxation and an increased HRV can indicate a potential state of mental stress or frustration (Raez et al. 2006). Blood Volume Pulse (BVP) as a measure of blood flow and heart rate increases with negative valence emotions such as anxiety and fear.
HRV analysis is studied in time domain and frequency domain. In time domain, the features are RR mean value, standard deviation (SDRR), root mean square of the differences between consecutive RR intervals (RMSSD) and the percentage number of consecutive (normal) intervals differing more than 50 ms in the entire recording (pNN50) whereas under mental distress, RR, SDRR and RMSSD are decreased. In the frequency domain, the features are power spectral density (PSD) for different frequency bands including very low frequency (VLF, 0–0.04 Hz), low frequency (LF, 0.04–0.15 Hz) and high frequency (HF, 0.15–0.04 Hz). The most common frequency domain parameters include the powers of VLF, LF and HF bands in absolute and relative values, the normalised power of LF and HF bands, and the LF to HF ratio whereas under mental distress LF is reduced in mental stress condition, while HF is increased.
Muscular system
Muscular system is in charge of body movement with muscles and the electrical activity of body muscles can be measured using electromyography (EMG). The EMG measures electrical currents generated in muscles when contracted representing neuromuscular activities. EMG signal shows muscle response to neural stimulation since a muscle fibre depolarises as the signal propagates along its surface and the fibre contracts. It follows by a movement of ions generating electric field near each muscle fibre whereas EMG signal is correlated to the amount of muscle contraction and the number of contracted muscles.
To monitor muscular system activity, EMG setup requiring placing EMG probes to the skin providing robust information and even to detect invisible muscular activities. EMG has the potential to be used in portable and wearable systems with no need to be dependent to laboratory environment and is a non-invasive approach (Raez et al. 2006). Facial expressions are generated by the muscular activity of the face which is either visible or non-visible and such muscular activities have the potential to be recorded by EMG. In addition to EMG monitoring, the visible muscle activities including facial expressions, gesture and posture could be video recorded to recognise useful pattern and information from them. Within the context of user experience recognition in human machine collaboration, the signals related to muscular system could be reflected in gesture, posture and facial expressions which can be observed using a camera which is not wearable but inertial sensors could be utilised as wearable sensing devices to monitor gesture and posture. In addition, EMG could be used for this purpose as well as facial expressions; however, wearing the EMG probes on the face could be uncomfortable to wear.
Muscle activity and electromyography, EMG, as a measure of muscle activity correlate with negative valence emotions. EMG analysis is studied in time domain and frequency domain. In time domain, to extract the amplitude information, the linear envelope (LE) is used. Root mean square (RMS) amplitude of EMG signal is another feature which reflects the mean power and amplitude of signal. Such features are increased when subject to the fatigue or mental distress. In frequency domain, the power spectrum density (PSD) is used whereas the frequency is between 20 and 450 Hz for normal human muscles. Under mental distress, there is a shift of power towards lower frequencies which is quantified by mean frequency (MNF) and median frequency (MDF). MNF is sum product of the EMG power spectrum and frequency, divided by a total sum of spectrum intensity while MDF is frequency value at which the EMG power spectrum is divided into two regions with an equal integrated power computed (Muñoz et al. 2018).
Integumentary system
Integumentary system is forming the outermost layer of body and includes skin. Skin conductivity or galvanic skin response (GSR) is measured as a bio-signal changing based on the human psychological condition. Electrodermal activity (EDA) sensors are measuring skin surface resistance or conductivity by passing a microcurrent of electricity through a pair of electrodes located near one another, amplifying and registering current variation. This variation is possible as the skin resistance depends on skin humidity (sweating), the thickness of the outer layer of the skin (epidermis) and vasoconstriction, among other things. Sweating behaviour is sensitive to emotional stimulation due to the sweat glands being controlled by the autonomic nervous system (ANS), which controls the body’s other physiological responses such as heart rate, temperature and pupil diameter (Sanchez-Comas et al. 2021). The physiological response of the ANS can increase in the presence of stress and multiple stimuli. The higher the sweat response, the higher the conductivity and the lower the resistance. This physiological response behaviour links the galvanic skin response to measures of emotional valence, facing pleasant (positive valence) or unpleasant (negative valence) stimuli. In human machine collaboration, for the purpose of user experience recognition, electrodermal activity sensor has the potential to be utilised by attaching the probes to the skin which is wearable and less likely to interfere with the user activities.
Electrodermal activity (EDA), which is electrical changes on the skin surface receiving innervating brain signals, analysis is considered from tonic and phasic components. The tonic level or skin conductance level (SCL) relates to sweat gland activity and is the baseline for resting level of the EDA signal. The phasic component or galvanic skin response (GSR) includes episodes of sudden increases of conductance caused by purely sympathetic arousal generally generated by an external stimulus. GSR increases linearly with a person’s level of overall arousal or stress. In the events of experiencing emotional activation, increased cognitive workload or physical exertion, the brain sends signals to the skin to increase the level of sweating. The changes can be measured in many ways electrically including skin potential, resistance, conductance, admittance and impedance while the unit of measurement for conductance is microSiemen (μS).
Respiratory system
Respiratory system including the lungs is in charge of breathing and the level of oxygen in blood depends on its functionality. Blood oxygen is the level of oxygen content in blood expressed as percentage of O2 saturation: %SpO2 which is measured by pulse oximeter. Pulse oximeter also monitors the heart rate or pulse in fact measurement of pulsation of blood. The device consists of a light emitter probe coupled with a photodiode which measures the difference in light absorbance at two wavelengths including the red portion of the visible region and another in the infrared region. The measurement provides information on the ratio of oxyhaemoglobin (haemoglobin [Hb] bound to oxygen) to deoxyhaemoglobin (Hb without oxygen). Red light is not absorbed well by oxygenated blood, but IR light is absorbed, properly. By using photodiode to produce small light beams at these two wavelength and detectors to measure the fraction of each light beam that passes through the tissue, the device calculates the ratio of red to IR absorbance during pulsation that is correlated to the %SpO2. The device displays %SpO2 with the pulse rate and the ratio of red to IR of 0.5 is correlated to 100% SpO2. Such approach for measuring blood oxygen is easy to wear and has the potential to be utilised in human machine collaboration scenarios.
Respiration rate which is the rate of breathing is measured by counting the number of times that the chest is rising per minute. Respiration rate can be calculated directly or indirectly whereas in the direct approach the movement of the chest is measured using inductance plethysmography, capnography, piezoelectric or bioimpedance-based sensors. However, such approach requires wearing strap around the chest. Respiratory rate can be measured directly using acoustic sensors; however, the performance can be affected by environmental noise. Monitoring the chest skin stretching during breathing using a pair of wearable photodiode and photo detector is suggested in Singh et al. (2020). Respiratory rate can also be indirectly extracted from electrocardiography (ECG) or photo-plethysmograph (PPG) signals. However, these methods can suffer from accuracy issues despite advancements in signal processing techniques and being not easy to wear within the context of human machine collaboration. Respiration rate as a measure of breathing speed where slow and deep breathing indicates a relaxed resting state; irregular rhythm, corresponds to more aroused emotions and stress can affect and lead to lower oxygen levels in the blood. Regarding the related signal processing techniques, literature shows wavelet-based signal features and decomposition into sub-bands (SBs), and from each of SBs, extracted Shannon entropy (SE) as a measure of disorder, uncertainty or randomness in the given information (Sharma et al. 2022). Time domain features include minimum, maximum, mean and variance of SPO2 signals as well as the same for first, second and third derivatives, area under the dip level of SPO2, deviation of mean and median values from the maximum and minimum values of SPO2 which are studied in Koley and Dey (2014).
To summerise, bio-signals which are captured signals from body organs represent a physical variable of interest that has the potential to be measured and processed through various sensor technologies and processing techniques in studying user experience within the context of human machine/robot collaboration. As user experience is reflected in physiological parameter changes, the bio-signals related to different body organ systems are explored along with the sensor technologies to capture such changes. To process the signals, features from bio-signals are extracted in time domain, frequency domain or combination of both whereas the method of extraction could be periodically over fixed or adaptable moving windows, but some cases could have triggered based feature extraction method. Feature normalisation is required when each feature values is different in range and distribution to normalise their baseline and amplitude ensuring each feature contributing equally. The features are commonly either time or frequency domain whereas the time domain features are signals’ mean, standard deviation, minimum, maximum, mode, variance and range while the frequency domain features are spectral energy and entropy, mean, standard deviation and variance regarding the shape and amplitude of features. Feature selection methods to identify an optimal feature subset have wide range which are influencing the quality of data interpretation. Data window size is influencing the time frame the feature values are referred to and the required processing. The larger the time window the longer the execution time to extract features while the time window length is commonly set a priori but it could be adaptive based on the variance of previous feature values (Gravina et al. 2017). The captured and processed bio-signals are correlated with the user experience which can be studied through the discussed sensor technologies and signal processing techniques in this section.
Methodology
Addressing safety in human machine/robot collaboration, the methodology used in this paper considering the background review is to study the user experience and impact of mental strain and cognitive workload on user performance and experience during the collaboration and how it is reflected in physiological signals. To study user experience during a collaborative scenario with machines in which the speed of operation is changing, subjective and objective metrics are explored. Therefore, in this section, the experiment design procedure is present and the performance metrics considering objective and subjective ones are explored.
Experiment design
To study user experience during human machine collaboration and how it is reflected in physiological signals, a human robot collaboration scenario is designed. In this experiment, a human is collaborating with a robot arm in a pick and place scenario whereas the robot operational speed is changing during the collaboration. The robot is in charge of handing over a number of objects from an assumed inaccessible location to the user who is in charge of cognition and sorting the objects based on the colour. Such scenario simulates a simplified industrial collaborative task in which the operator is required to work with an industrial robot.
In this experiment, the objects are test tubes filled in with five different coloured liquid (green, blue, red, yellow, purple), six for each colour, making total of thirty test tubes randomly placed in a container in three rows of ten test tubes. The user is supposed to pick the test tubes from the robot arm and sort them out based on the colour and locate them in a container which is in 2.0 m distance from the robot as shown in Fig. 1 while the experiment was conducted in a quiet room with a standard room temperature of 21.0 degree centigrade.
Fig. 1 [Images not available. See PDF.]
Human robot collaboration experiment setup
During the designed collaborative scenario, the physiological signals of the user are collected through two different wristbands, one on right and the other on the left hand, which are EmotiBit and Empatica E4. Emotibit is a wearable sensor module for capturing physiological, and movement data with wireless data streaming or direct data recording to the built-in SD card with a fully open-source software (Emotibit 2022). Empatica E4 is equipped with sensors designed to gather high-quality data to combine EDA and PPG sensors, simultaneously enabling the measurement of sympathetic nervous system activity and heart rate (Empatica 2022). The use of the two wearable sensor modules is to be able compare and validate the results. After the experiment, each participant will answer a series of quantitative questions about their experience during the experiment and performance of the collaboration is measured through a series of objective and subjective metrices.
The robot arm utilised in handling the experiments is from ST robotics which is a 6-axes robot arm with high power micro-stepped hybrid stepping motors with reach of 500 mm in any direction and 360-degree waist rotation. The repeatability is 0.1 mm and the payload is of nominal 400 g whereas the maximum speed is approximately 800 mm/s. Maximum torque for pitch or yaw is 1 Nm and maximum torque for 6th axis roll is 0.25 Nm. The robot weight is 13.0 kg and controller is 11.0 kg while the required electrical power is 110/240v ac with noise approximately equal to 40–50 dB at 1 m (Robotics 2022).
In this experiment, 15 participants (4 females, 11 males) were involved age 22.5 ± 0.5 years old. The participants were informed about the experiment procedure through participant information sheet, and they signed consent form to take part in the experiment. Bournemouth University (BU) Ethical approval is in place and both the participant information sheet and the consent form are accepted by BU Ethics Panel with Ethics ID of 47085.
Each subject wore the wristbands while ensuring they are comfortably fitted and physiological signals are collected as expected. In this stage, subjects were in a resting state with minimum physical or mental activity. Then, the human robot collaboration experiment was started which lasts 5 min while the robot was handing over the randomly placed coloured test tubes to the subjects and for the subject to place them in a container which is in 2.0-m distance and sort them out based on the colour. During the experiment, the speed of robot increased whereas the robot would start with an initial speed at the start to pick the ten test tubes in first row and then the speed would be doubled to pick the ten test tubes in second row and speed would be triple of the initial one to pick the rest of ten test tubes in the last row (Fig. 1). By the end, a questionnaire was given to the subjects to collect their experience qualitatively as well as the quantitative data collected through the wristbands.
Performance metrics
Objective and subjective metrics were considered to assess the human machine/robot collaboration performance. Objective metrics consisted of performance-related indexes measured during the experiments by counting the number of mistakes occurred while carrying out the procedure by the machine/robot; the user mistakes include not picking the test tubes safely and sorting them out incorrectly while the machine mistake includes not picking and handing over the test tubes, correctly. The amount of the time needed to perform the task is also considered in the literature as a performance metrics (Villani et al. 2022) but the experiment duration is fixed in this paper. Subjective metrics consist of user experience questionnaire including physical working conditions, psychosocial working conditions and ethical aspects, as well as user’s satisfaction with the HMC scenario, including health and safety issues. The user experience questionnaire was designed based on the questionnaire designed and validated in Villani et al. (2022) in study of subjective metrics for an adaptive human machine interaction scenario.
The examples of the questions asked include “Do you think the physical working condition was appropriate” to consider the assessment could be influenced by some other external factors present in the environment, “Did you feel safe while interacting with the machine?” to address experienced safety, “Did you feel more distress while the robot speed was increasing over the time?” to address psychosocial working conditions and ethical aspects, and “Do you think experiencing distress during the interaction affected your performance to complete the task without mistake?” to address user’s satisfaction. Questions to understand user’s expectations and the desire to use the technological developments are “Do you think if the machine’s functionality is adjustable to your experience, you would feel less distress?”, “Do you think if adjustability to the user experience in machine functionality is developed, you would feel comfortable to use from ethical perspective?”, “Do you think if adjustability in machine functionality is developed, you would feel safer in the collaboration scenario?”, “Do you think if adjustability in machine functionality is developed, there would be less mistakes in the collaboration scenario?”, “Do you think it would be helpful if the machine’s functionality could be adjustable based on your experience?” and “Do you think wearing the wristband enabling the machine to predict your experience leading to adjusting its functionality is something that you would consider in a human machine collaboration scenario?”.
The approach in study of user experience has some limitations as user experience is subjective, and users could have different experiences during similar condition. Different parameters could have impact on user experience in addition to controlled one such as noise (conducting the experiment in a quiet room), temperature and humidity (controlled and fixed by an air conditioner) such as past experiences, knowledge, training, health status and competence. To address these limitations, both qualitative and quantitative data are gathered in having more informed study as represented in Figs. 2, 3 and 4 as well as Table 1.
Fig. 2 [Images not available. See PDF.]
Heart rate mean value collected during the human machine collaboration scenario through Empatica and Emotibit wristbands for subject 1 to 15
Fig. 3 [Images not available. See PDF.]
Electrodermal activity or GSR mean value collected during the human machine collaboration scenario through Empatica and Emotibit wristbands for subject 1 to 15
Results and discussion
To study the user experience, the monitored physiological signals during the experiment are heart signal and electrodermal activity (skin conductivity) collected by Empatica and Emotibit wristbands. The data is time normalised to be able to show the experiment time in percentage on horizontal axes. For heart signals, the extracted feature is the averaged heart rate over a fixed window size. The heart rate signal from Empatica is the average heart rate values computed in spans of 10 s with sampling frequency of 1 Hz while the signal from Emotibit is the average heart rate with sampling frequency of 25 Hz and window size of 150 samples. As it is shown in Fig. 2 for different subjects, it is evident that both the two wristbands represent similar trend in terms of changes in heart rate; however, the differences could be due to wearing the wristband on dominant or nondominant hand, the difference in processing speed of the wristbands and sampling rate, the difference in window size of the moving average, etc. The collected data shows that for subject 10 the Emotibit reading is not valid over the experiment time and the same applies for subject 14 that the data collection by Emotibit failed. Looking into the trend of the collected data, it can be seen that for some subjects the heart rate has increased during the experiment such as subject 8 but for some it is fluctuating such as subjects 1, 6, 9, 12 and 13. Such variation could be due to different experiences that the subjects had during the experiment; for some it might be more distressing when the speed was increased and for some they could adapt themselves with the changes in the speed after each step of increase in speed causing less distress after the adaptation. Electrodermal activity or galvanic skin response (GSR), which is electrical changes on the skin surface receiving innervating brain signals, is reflecting general changes in autonomic arousal. GSR signals collected from Empatica and Emotibit during the experiment are shown in Fig. 3 Sampling frequency for Emotibit is 15 Hz, and for Empatica, it is 4 Hz; therefore, the data is time normalised to be represented as percentage over the experiment time duration. The GSR data range could vary from 0.01 to 100 microSiemens, and to have readings, there is need to have coupling of the electrodes with the skin which could take around 10 to 15 min and influenced by the material of the electrodes, the position of the electrodes on the body and the environmental conditions during the recording (temperature and humidity of the room).
Fig. 4 [Images not available. See PDF.]
Responses (in percentage) to the questionnaire by the subjects attended the experiment
As shown in Fig. 3, it appears that the collected signals by Emotibit has not captured the electrodermal activity changes which could be due to lack of enough time for coupling of the electrodes with the skin which could be around 10 to 15 min. Therefore, for future experiments, a warmup and baseline period are needed to be considered. Looking into the trend of data, the figure also shows for some subjects there is increase in electrodermal activity such as subject 5, 12 and 15 but for some there are fluctuations which could be due to the different experiences that the subject had during the experiment. It can be observed that subjects had different experiences during the collaboration; therefore, different needs. Some found it stressful initially due to the inexperience and for some it was stressful when the speed increased and found it harder to organise the colours.
To assess the user performance, the considered metrics for the human machine collaboration performance includes objective ones such as performance-related indexes measured during the experiment by counting the number of mistakes occurred while carrying out the procedure by the machine/robot to pick the test tubes; the user mistakes include not picking the test tubes safely and then placing and sorting them out incorrectly. Such objectives are represented in Table 1 as “Robot Pick Success Rate” for the 30 test tubes, “Subject Pick Success Rate” for the successfully handed over test tubes by the robot, “Subject Place and Sort Success Rate” for the successfully placed and sorted test tubes for each subject and in average for the overall experiment. Table 1 shows that the robot had 99.5% success rate in handing over the test tubes whereas there were 93.2% success rate from the participants in collecting the test tubes from the robot with 99.5% success in placing the test tubes colour sorted.
Table 1. Success rate in the collaborative scenario for human and robot
Subject number | Robot pick success rate | Subject pick success rate | Subject place and sort success rate |
|---|---|---|---|
1 | 30/30 | 29/30 | 29/29 |
2 | 30/30 | 27/30 | 27/27 |
3 | 29/30 | 26/29 | 26/26 |
4 | 30/30 | 21/29 | 21/21 |
5 | 29/30 | 28/29 | 28/28 |
6 | 30/30 | 27/30 | 27/27 |
7 | 29/30 | 27/29 | 27/27 |
8 | 30/30 | 29/30 | 29/29 |
9 | 30/30 | 27/30 | 27/27 |
10 | 30/30 | 28/30 | 27/28 |
11 | 30/30 | 30/30 | 30/30 |
12 | 30/30 | 29/30 | 29/29 |
13 | 30/30 | 29/30 | 29/29 |
14 | 30/30 | 30/30 | 29/30 |
15 | 30/30 | 29/30 | 29/29 |
Total average % | 99.5% | 93.2% | 99.5% |
To study user experience through the qualitative approach, subjective metrics consist of user satisfaction questionnaire including physical working conditions, psychosocial working conditions and ethical aspects; and user’s satisfaction with the HMC, including health and safety issues. Looking into the questionnaire responses, 80% of participant perceived the physical working condition appropriate and 80% felt safe to a large extent and more while interacting with the machine. A total of 66.7% of participants felt more distress somewhat and more while the robot speed was increasing, and 33.3% of participants perceived the experienced distress affected their performance somewhat and more. When asking about whether they would feel less distress if the machines’ functionality was adjustable to their experience, 80% of participants agreed to somewhat and to a large extent while 86.7% of participants expressed that they would feel comfortable to use the system from ethical perspective to a large extent and more. Also, in response to if such feature developed 80.0% expressed, they would feel safer in the collaboration scenario to a large extent and more. A total of 86.7% mentioned if such feature developed, there would be less mistakes in the collaborative scenario and 86.6% of participants think such feature would be helpful to a large extent and more. Eighty percent of participants mentioned that they would consider to a large extent and more wearing a wristband to add adjustable functionality of machine feature (see Fig. 4). The subjects stated that they felt stressed when not being able to see the robot consistently and found it harder to organise the colours when the robot speed increased and took them time to adjust to the speed of the robot. They stated that “as the speed increases it felt more pressured which leaded to more mistakes and was more physically demanding” and some felt “confused” in the process of sorting out the colours when the speed increased. There were cases that the subjects stated once the rhythm of collaboration was found it was easier to perform the task while more stressful initially due to the inexperience. Some expressed that during the experiment, they wanted the robot speed to be decreased as felt tired and stressed. Furthermore, considering the responses, it appears if an adaptability feature based on user experience for the machines would be developed, the users could feel safer, and less distressed with more success rate in completing the task.
Conclusion
In this paper, the on-body sensing technologies and related signal processing techniques were explored in addressing safety in human machine collaboration in study of user experience and impact of mental strain and cognitive workload on user performance and experience. Considering cognitive burden during interaction with robots and its reflection in physiological signals, a human machine collaboration scenario was designed in which the robot was handing over coloured objects from assumed inaccessible location to the user to sort and place them in a predefined location while the speed of robot operation was increasing during the operation. The collaboration performance was measured through qualitative and quantitative approaches whereas the results showed that the users had different experiences during the experiments and different speed preferences potentially due to the different skills that they had. By increase in the speed of collaboration, some experienced more distress and more cognitive burden which could potentially lead to collision and safety-related concerns. However, some subjects preferred higher speeds of collaboration which means different users have different preferences; therefore, a mechanism adaptive to the user experience during collaboration would be helpful in improving user performance and experience which is the follow-up work.
Data Availability
The experiment data would be available upon request from the corresponding author.
Declarations
Research involving human participants and informed consent
The participants were informed about the experiment procedure through participant information sheet, and they signed consent form to take part in the experiment. Bournemouth University (BU) Ethical approval is in place and both the participant information sheet and the consent form are accepted by BU Ethics Panel with Ethics ID of 47085.
Conflict of interest
The author declares no competing interests.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
Emotibit, (2022), emotibit, Available at: https://www.emotibit.com/ (accessed 2nd December 2022)
Empatica, (2022), empatoca, Available at: https://www.empatica.com/ (accessed 2nd December 2022)
Gravina, R; Alinia, P; Ghasemzadeh, H; Fortino, G. Multi-sensor fusion in body sensor networks: state-of-the-art and research challenges. Inf Fusion; 2017; 35, pp. 68-80. [DOI: https://dx.doi.org/10.1016/j.inffus.2016.09.005]
Haratian R, Timotijevic T, Phillips C (2016) Reducing power and increasing accuracy of on-body sensing, IET Signal Proc J, 10(2):133–139. https://doi.org/10.1049/iet-spr.2014.0496
International Standard ISO 10218-1:2011 (2011a) Robots and robotic devices — Safety requirements for industrial robots — Part 1: RobotsEdition 22011–07. https://www.iso.org/standard/51330.html#:~:text=ISO%2010218%2D1%3A2011%20specifies,risks%20associated%20with%20these%20hazards
International Standard ISO 10218–2:2011 (2011b) Robots and robotic devices – safety requirements for industrial robots – part 2: robot systems and integration. Edition 12011–07. https://www.iso.org/standard/41571.html#:~:text=ISO%2010218%2D2%3A2011%20describes,risks%20associated%20with%20these%20hazards
Koley, BL; Dey, D. On-line detection of apnea/hypopnea events using SpO2 signal: a rule-based approach employing binary classifier models. IEEE J Biomed Health Inform; 2014; 18,
Landi, C; Villani, V; Ferraguti, F; Sabattini, L; Secchi, C; Fantuzzi, C. Relieving operators’ workload: towards affective robotics in industrial scenarios. Mechatronics; 2018; 54, pp. 144-154. [DOI: https://dx.doi.org/10.1016/j.mechatronics.2018.07.012]
Matthews O, Davies A, Vigo M, Harper S (2020) Unobtrusive arousal detection on the web using pupillary response. Int J Human-Comput Stud 136:1071–5819. https://doi.org/10.1016/j.ijhcs.2019.09.003
Muñoz, JE; Gouveia, ER; Cameirão, MS et al. PhysioLab - a multivariate physiological computing toolbox for ECG, EMG and EDA signals: a case of study of cardiorespiratory fitness assessment in the elderly population. Multimed Tools Appl; 2018; 77, pp. 11521-11546. [DOI: https://dx.doi.org/10.1007/s11042-017-5069-z]
Raez, MB; Hussain, MS; Mohd-Yasin, F. Techniques of EMG signal analysis: detection, processing, classification and applications. Biol Proced Online; 2006; 8, pp. 11-35. [DOI: https://dx.doi.org/10.1251/bpo115]
ST Robotics, (2022), ST Robotics Available at: https://www.strobotics.com (accessed 2nd December 2022)
Saltzman W (2015) Biomedical engineering: bridging medicine and technology (2nd ed., Cambridge Texts in Biomedical Engineering). Cambridge: Cambridge University Press. https://doi.org/10.1017/CBO9781139583831
Sanchez-Comas, A; Synnes, K; Molina-Estren, D; Troncoso-Palacio, A; Comas-González, Z. Correlation analysis of different measurement places of galvanic skin response in test groups facing pleasant and unpleasant stimuli. Sensors (Basel); 2021; 21,
Sharma, M; Kumbhani, D; Yadav, A et al. Automated sleep apnea detection using optimal duration-frequency concentrated wavelet-based features of pulse oximetry signals. Appl Intell; 2022; 52, pp. 1325-1337. [DOI: https://dx.doi.org/10.1007/s10489-021-02422-2]
Singh, G; Tee, A; Trakoolwilaiwan, T; Taha, A. Olivo M (2020) Method of respiratory rate measurement using a unique wearable platform and an adaptive optical-based approach. Intensive Care Med Exp; 2020; 8,
Villani V, Pini F, Leali F, Secchi C (2018) Survey on human–robot collaboration in industrial settings: safety, intuitive interfaces and applications, Mechatronics 55. https://doi.org/10.1016/j.mechatronics.2018.02.009
Villani, V et al. A user study for the evaluation of adaptive interaction systems for inclusive industrial workplaces. IEEE Trans Autom Sci Eng; 2022; 19,
Copyright Springer Nature B.V. Dec 2024