A non-anatomical graph structure for boundary detection in continuous sign language

Abstract

Recently, the challenge of the boundary detection of isolated signs in a continuous sign video has been studied by researchers. To enhance the model performance, replace the handcrafted feature extractor, and also consider the hand structure in these models, we propose a deep learning-based approach, including a combination of the Graph Convolutional Network (GCN) and the Transformer models, along with a post-processing mechanism for final boundary detection. More specifically, the proposed approach includes two main steps: Pre-training on the isolated sign videos and Deploying on the continuous sign videos. In the first step, the enriched spatial features obtained from the GCN model are fed to the Transformer model to push the temporal information in the video stream. This model in pre-trained only using the pre-processed isolated sign videos with same frame lengths. During the second step, the sliding window method with the pre-defined window size is moved on the continuous sign video, including the un-processed isolated sign videos with different frame lengths. More concretely, the content of each window is processed using the pre-trained model obtained from the first step and the class probabilities of the Fully Connected (FC) layer embedded in the Transformer model are fed to the post-processing module, which aims to detect the accurate boundary of the un-processed isolated signs. In addition, we propose to present a non-anatomical graph structure to better present the hand joints movements and relations during the signing. Relying on the proposed non-anatomical hand graph structure as well as the self-attention mechanism in the Transformer model, the proposed model can successfully tackle the challenges of boundary detection in continuous sign videos. Experimental results on two datasets show the superiority of the proposed model in dealing with isolated sign boundary detection in continuous sign sequences.