Multimodal text-emoji fusion using deep neural networks for text-based emotion detection in online communication

Abstract

The task of emotion detection in online social communication has been explored extensively. However, these studies solely focus on textual cues. Nowadays, emojis have become increasingly popular, serving as a visual means to express emotions and ideas succinctly. These emojis can be used supportively or contrastively, even sarcastically, adding complexity to emotional interpretation. Therefore, incorporating emoji analysis is crucial for accurately extracting insights from social media content to support decision-making. This paper aims to investigate to what extent the usage of emojis can contribute to the automated detection of emotions in text messages with a focus on online social communication. We propose an emoji-aware hybrid deep learning framework for multimodal emotion detection. The proposed framework leverages the feature-level fusion of textual and emoji representations, incorporating conventional and recurrent neural networks, to learn the fused modalities. The proposed approach was extensively evaluated on the GoEmotions dataset with different performance metrics. The experimental results indicate that emoji features can significantly improve emotion classification accuracy, highlighting their potential for enriching emotion understanding in online social communication.