Effects of Introducing Speech Interaction Modality on Performance of Special Vehicle Crew Under Various Task Complexity Conditions

Abstract

An experiment with a two interaction modalities (traditional: touch; novel: touch–speech) × two task complexities (low: visual single task; high: audio–visual dual task) within-subjects design was conducted to observe alterations in crew performance (including task performance, subjective workload, and eye responses) in a typical planning task-based on a high-fidelity special vehicle simulation platform. The results revealed that (1) compared to the traditional interaction modality, the novel interaction modality significantly improved task performance, reduced subjective workload, increased mean peak saccade velocity, and decreased fixation entropy; (2) under high task complexity, subjective workload, mean pupil diameter, and the nearest neighbor index showed significant increases, while no significant decline in task performance was observed; (3) no interaction effect of crew performance was observed between interaction modality and task complexity. The foregoing results imply that (1) the novel interaction modality incorporating speech input exhibits advantages over the traditional touch-based modality in terms of enhancing task performance (over 45% improvement) and reducing cognitive workload; (2) leveraging dual-channel audio–visual information processing facilitates the maintenance of task performance under high task complexity and multi-tasking demands; (3) eye movement characteristics may serve as informative indicators for evaluating the benefits of speech-based interaction and the effectiveness of cognitive resource allocation under high-complexity task conditions. The results can provide a basis for the design of the display and control interface in special vehicles.