The more the better? Multisensory redundant mappings to convey information in abstract visualizations

Abstract

Immersive technology like Virtual Reality enables the integration of multisensory stimuli in data visualizations, enhancing comprehension and decision-making, especially when the visual channel is overloaded or ambiguous. While most Immersive Analytics (IA) applications rely on visual-centric approaches, multisensory feedback remains rare, and few studies have explored non-visual senses as conventional channels for data representation. Existing work often focuses on sensory substitution rather than leveraging the potential of redundant multisensory codification. This study addresses this gap by investigating the combined use of visual, auditory, and haptic stimuli for encoding data in a simple abstract visualization. We conducted a within-subjects study (n = 52) using a 3D point-cloud visualization to evaluate how redundant codifications (node radius, pitch, and vibration intensity) influence task efficiency, accuracy, and user workload. We also studied behaviour and user experience when presented with the different combinations of stimuli. Our findings show that redundant mappings positively affect performance metrics in situations where the visual modality might be compromised. The user experiences varied, underscoring the need for tailored training and adaptation to artificial sensory codifications such as vibration. While limited to simple visualizations, this study provides insights into designing effective multisensory redundant codifications.