Facial Gesture Detection Using Ubiquitous Acoustic Sensing

Abstract

Facial gestures are defined as the movements of facial muscles that occur in areas such as the eyebrows, cheeks, and mouth, and are utilized by humans to convey a multitude of meanings in different situations. These meanings can range from expressing emotions through facial expressions, performing speech-related actions with the mouth and teeth, to conveying specific linguistic constructs. Consequently, detecting facial gestures has emerged as a popular research topic that has gained increasing interest from academia and industry. In addition, acoustic sensing has won significant popularity in hand tracking, gesture recognition, and silent speech recognition due to its inexpensive, widespread availability and compact size. With previous research highlighting the potentials of ubiquitous acoustic sensing and the growing usage of smart speakers and earphones, hence we investigate the feasibility of adopting acoustic sensing for recognizing various facial gestures in three critical application domains: detecting emotional facial expressions, recognizing silent speech, and identifying facial information for sign language translation. Accurately recognizing emotional facial expressions can be a valuable means of understanding audience feedback and engagement with entertainment content. In light of this, the dissertation presents SonicFace, the first-ever system that utilizes a smart speaker to identify various subtle emotional facial expressions. To ensure privacy protection during speech interactions and also enhance resistance to ambient noise, the dissertation introduces EarCommand, a new earphone-based, hands-free interaction technique that employs silent speech commands recognition by leveraging the connection between ear canal deformation and articulator movements. As sign language is the primary communication mode for individuals with hearing impairments, the dissertation proposes TransASL, an innovative smart glasses based system that simultaneously translates hand gestures and facial expressions into natural language by utilizing two pairs of microphone and speaker mounted on the earphones. One pair captures front-facing signals for hand gestures, while the other pair records inner-facing signals for recognizing facial expressions. All these scientific explorations in this dissertation are expected to lay a foundation for facial gesture recognition in emotional facial expression, silent speech recognition based on mouth motion and facial information in ASL recognition with a convenient and ubiquitous method.