Lip reading of words with lip segmentation and deep learning

Abstract

Speech perception is recognized as a multimodal task, that is, it solicits more than one meaning. Lip reading, which superimposes visual signals to auditory signals, is useful and sometimes even necessary for understanding a message. Lip-reading is an area of great importance for a wide range of applications, such as silent dictation, speech recognition in noisy environment, improved hearing aids and biometrics. It is a difficult research subject in the field of computer vision, whose main purpose is to observe the movement of human lips from the video to identify the corresponding textual content. However, because of the limitations of lip changes and the richness of linguistic content, the increased difficulty of lip recognition slows down the development of lip language research topics. Recently, the development of deep learning in various fields gives us enough confidence to carry out the task of lip recognition. Unlike recognition of lip characteristics in traditional lip recognition, lip learning based on deep learning typically involves extracting features and understanding images using a network model. In this topic, we focus on the design of the acquisition, processing, and data recognition network framework for lip reading. In this work, we developed an accurate and robust algorithm, for lip reading. First, we extract the mouth region and segmented the mouth by using a proposed hybrid model with a new proposed edge based on a proposed filter, then we train our spatio-temporal model by the combination of Convolutional Neural Networks (CNN) and Bi-directional Gated Recurrent Units (Bi-GRU). Finally, we test our algorithm, and we get an evaluation of 90.38% of accuracy. The result shows the performance of our system by application of lip segmented as inputs to the proposed spatio-temporal model.