Facial emotion recognition using deep Siamese neural networks: multi-classifier fusion for single-emotion and multi-emotion models across age groups

Abstract

This research examines facial expressions as social cues in online platforms, focusing on online learning and remote work. Our high-accuracy emotion recognition framework is designed for post-session evaluation, aiding teaching strategies and identifying students needing support without real-time monitoring. By employing a sophisticated fusion of data, image, and feature-level analysis, complemented by multi-classifier systems, this study capitalizes on the strengths of Siamese networks to achieve a refined understanding of emotion recognition. The investigation spans various age groups and ethnicities, employing multiple datasets such as LIRIS-CSE, Cohn-Kanade, and Jaffe, alongside the author’s datasets for children and teens. This rigorous examination underlines the role of Information Fusion in enriching communication and collaboration within digital interfaces. The research underscores the use of advanced techniques in interpreting facial cues by merging Siamese networks with pretrained models such as VGG 19 and Inception Resnet V2. The paper compares Siamese networks with other architectures for remote work/play, and asserts that such networks are more flexible. Comparatively, networks with Siamese architecture use convolutional neural network and recurrent neural network for input branches while other networks such as VGG19 and Inception Resnet V2 use a single neural network. The SCNN-IRV2 model demonstrated impressive test accuracies, ranging from 96 to 99% for single-emotion models and 84–96% for multi-emotion models, reflecting an improvement of 1.05–10.13% over the Inception Resnet V2 CNN architecture. In a similar vein, the SCNN-VGG19 model achieved test accuracies between 95% and 99% for single-emotion models and 75–94% for multi-emotion models, surpassing the VGG19 CNN architecture by 207–422%. These findings highlight the role of advanced fusion techniques and thoughtful design in improving online education and remote work, fostering progress in emotional data analysis and human-computer interaction.