Content area
Full Text
Abstract
Recognition of American Sign Language (ASL) alphabet not only could bring benefits to the ASL users, but also could provide solutions for natural human-computer/robot interactions in many applications. In this paper, we propose a method for ASL alphabet recognition with use of a Leap Motion Controller (LMC). The skeleton data from the native LMC API is transformed by a skeleton module into a vector of the angle features. Meanwhile, two raw infraredradiation (IR) images are captured and each of them is fed into a vision module using a Convolutional Neural Network (CNN) for visual feature extraction, which results in two feature vectors. Those three feature vectors are then fed into a fusion neural network to output the predicted label. An ASL alphabet dataset is established, on which the proposed model is evaluated. The results show that our proposed method achieves the prediction accuracies of 80.1% and 99.7% in the leave-one-out and the half-half experiments, respectively.
Keywords
American Sign Language, Leap Motion Controller, Convolutional Neural Network, deep learning, machine learning.
1.Introduction
American Sign Language (ASL) alphabet refers to 26 finger-spelled letters including 24 static ones and 2 dynamic ones 'J and 'Z' [1]. Automatic recognition of these signs not only could bring benefits to the ASL users, but also could provide solutions for natural human-computer/robot interactions in a wide range of applications such as using gesture control devices. To tackle this challenging task, different kinds of sensors have been explored. Cyber gloves can directly sense the finger bending but are inconvenient for daily use [2, 3]. Microsoft Kinect, which was first released in 2010, has been applied in this task intensively because it is able to perceive the depth information [4, 5]. Leap Motion Controller (LMC) was first released in 2012 and it can provide the hand skeleton information. This information has been used for sign recognition tasks [6, 7]. Different machine learning methods have been applied, such as support vector machine, k-nearest neighbor, and random forest [4, 8]. Deep learning methods, such as Convolutional Neural Networks (CNN), have been prevalent recently for image recognition and Natural Language Processing (NLP) tasks [9].
In the present study, we use a LMC as the sensing device to capture human hands for sign recognition. An overview of...