Full text

Turn on search term navigation

© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.

Abstract

With the popularization of better network access and the penetration of personal smartphones in today’s world, the explosion of multi-modal data, particularly opinionated video messages, has created urgent demands and immense opportunities for Multi-Modal Sentiment Analysis (MSA). Deep learning with the attention mechanism has served as the foundation technique for most state-of-the-art MSA models due to its ability to learn complex inter- and intra-relationships among different modalities embedded in video messages, both temporally and spatially. However, modal fusion is still a major challenge due to the vast feature space created by the interactions among different data modalities. To address the modal fusion challenge, we propose an MSA algorithm based on deep learning and the attention mechanism, namely the Mixture of Attention Variants for Modal Fusion (MAVMF). The MAVMF algorithm includes a two-stage process: in stage one, self-attention is applied to effectively extract image and text features, and the dependency relationships in the context of video discourse are captured by a bidirectional gated recurrent neural module; in stage two, four multi-modal attention variants are leveraged to learn the emotional contributions of important features from different modalities. Our proposed approach is end-to-end and has been shown to achieve a superior performance to the state-of-the-art algorithms when tested with two largest public datasets, CMU-MOSI and CMU-MOSEI.

Details

Title
Mixture of Attention Variants for Modal Fusion in Multi-Modal Sentiment Analysis
Author
He, Chao 1 ; Zhang, Xinghua 2 ; Song, Dongqing 3 ; Shen, Yingshan 4 ; Mao, Chengjie 3 ; Wen, Huosheng 5 ; Zhu, Dingju 5 ; Cai, Lihua 6 

 School of Computer Science, South China Normal University, Guangzhou 510631, China; [email protected] (C.H.); [email protected] (D.S.); [email protected] (C.M.); Aberdeen Institute of Data Science and Artificial Intelligence, South China Normal University, Guangzhou 528225, China; [email protected] 
 International United College, South China Normal University, Guangzhou 528225, China; [email protected] 
 School of Computer Science, South China Normal University, Guangzhou 510631, China; [email protected] (C.H.); [email protected] (D.S.); [email protected] (C.M.) 
 Aberdeen Institute of Data Science and Artificial Intelligence, South China Normal University, Guangzhou 528225, China; [email protected] 
 School of Software, South China Normal University, Guangzhou 528225, China; [email protected] 
 Aberdeen Institute of Data Science and Artificial Intelligence, South China Normal University, Guangzhou 528225, China; [email protected]; School of Software, South China Normal University, Guangzhou 528225, China; [email protected] 
First page
14
Publication year
2024
Publication date
2024
Publisher
MDPI AG
e-ISSN
25042289
Source type
Scholarly Journal
Language of publication
English
ProQuest document ID
2930507219
Copyright
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.