Multimodal Fusion for Image and Text Classification with Feature Selection and Dimension Reduction

Abstract

Internet has become an important information platform, and it is very important to accurately understand the multimedia information of the Internet. In this paper, our main task is to do classification based on pictures and texts collected from the Internet, which is a classification problem of multimodal fusion in practice. However, when multimodal data is put together, there may occur the dimension disaster problem. We apply feature selection (FS) and dimension reduction (DR) in feature levels both in later fusion and early fusion to solve this problem. The classification accuracies in different models obtain improvements in different levels respectively. We also discuss the relation between single modals and multimodal in later fusion. In our experiments, images and text can be classified by multimodal models under FS/DR, and of which with the help the multimedia information from the Internet can be analysed better to help enterprises provide better services and products, and then carry out better network marketing and promotion.