Harnessing AI to Generate Indian Sign Language from Natural Speech and Text for Digital Inclusion and Accessibility

Abstract

Sign language is the fundamental mode of communication for those who are deaf and mute, as well as for individuals with hearing impairments. Regrettably, there has been a dearth of research on Indian Sign Language, primarily due to the lack of adequate grammar and regional variations in such language. Consequently, research in this area has been limited. The primary objective of our research is to develop a sophisticated speech/ text-to-Indian sign language conversion system that employs advanced 3D modeling techniques to display sign language motions. Our research is motivated by our desire to promote effective communication between hearing and hearing-impaired individuals in India. The proposed model integrates Automatic Speech Recognition (ASR) technology, which effectively transforms spoken words into text, and leverages 3D modeling techniques to generate corresponding sign language motions. We have conducted a comprehensive study of the grammar of Indian Sign Language, which includes identifying sentence structure and signs that represent the tense of the subject. It is noteworthy that the sentence structure of Indian Sign Language follows the Subject-Object-Verb sequence, in contrast to spoken language, which follows the Subject-Verb-Object structure. To enhance user experience as well as digital inclusion and accessibility, the research incorporates user-friendly and simple interfaces that allow individuals to interact effortlessly with the system intuitively. The model/ system is equipped to receive speech input through a microphone/ text and provide immediate feedback through 3D-modeled videos that display the generated sign language gestures and has achieved 99.2% accuracy. Our main goal is to promote digital inclusion and improve accessibility and enhance the user experience.