An intent recognition pipeline for conversational AI

Abstract

Natural Language Processing (NLP) is one of the Artificial Intelligence applications that is entitled to allow computers to process and understand human language. These models are utilized to analyze large volumes of text and also support aspects like text summarization, language translation, context modeling, and sentiment analysis. Natural language, a subset of Natural Language Understanding (NLU), turns natural language into structured data. NLU accomplishes intent classification and entity extraction. The paper focuses on a pipeline to maximize the coverage of a conversational AI (chatbot) by extracting maximum meaningful intents from a data corpus. A conversational AI can best answer queries with respect to the dataset if it is trained on the maximum number of intents that can be gathered from the dataset which is what we focus on getting in this paper. The higher the intent we gather from the dataset, the more of the dataset we cover in training the conversational AI. The pipeline is modularized into three broad categories - Gathering the intents from the corpus, finding misspellings and synonyms of the intents, and finally deciding the order of intents to be picked up for training any classifier ML model. Several heuristic and machine-learning approaches have been considered for optimum results. For finding misspellings and synonyms, they are extracted through text vector neural network-based algorithms. Then the system concludes with a suggestive priority list of intents that should be fed to a classification model. In the end, an example of three intents from the corpus is picked, and their order is suggested for the optimum functioning of the pipeline. This paper attempts to pick intents in descending order of their coverage in the corpus in the most optimal way possible.