MaQA: A Manual Text-Based Approach for Car-Specific Question Answering

Abstract

In the past few years, intelligent virtual assistant technology has had a significant impact on our daily lives, enabling us to easily access information through simple voice commands. In this paper, we present MaQA (Manual Question Answering), an approach for solving domain-specific question answering in the automobile sector, aiming to broaden the impact of virtual assistants by integrating them into the automotive domain. The goal of MaQA is to accurately classify car-related questions into one of the 1526 FAQ categories. To tackle this task, we constructed the MaQA dataset in both Korean and English, each consisting of 45,641 questions, and propose an end-to-end neural FAQ model. Our approach achieved accuracy scores of 85.24% and 82.95% on the Korean and English datasets, respectively, with 12× and 13.4× faster inference, outperforming the BERT-based model in both accuracy and latency.

1. Introduction

The closed-domain question answering (QA) task takes an input query from a specific domain and outputs the correct answer to the question. Recently, various studies have explored open-domain QA with Machine Reading Comprehension (MRC) techniques, using Wikipedia as the knowledge base [1,2,3]. While there have been extensive works in the field of general-purpose open-domain QA [3,4,5], the direct application of these advancements to specific domains, such as technical manuals and legal documents, gives unsatisfactory results. QA systems in specific domains are in high demand as such systems are essential in building real-world applications and services [6,7,8]. Specifically, QA in the automobile domain, which is the target of this paper, if well-integrated with voice assistants, can benefit numerous daily car users.

There are three main aspects of the automobile domain-specific QA when it comes to car users referring to a car manual book for solving questions under a driving circumstance: (1) car manuals are comprehensive yet too dense in information; (2) users are required to know domain-specific terminologies, which makes car manuals even more inaccessible; and, last but not least, (3) car manual books are not a great source of information when users are seeking a real-time response while driving. One straightforward approach to solving the car domain-specific QA is employing MRC-based QA methods. However, MRC-based QA comes with a high computational cost, which makes it not efficient enough for real-time services, and an expensive time-consuming data annotation process.

Considering these aspects, we bring a different perspective to solve the car domain-specific QA task. Instead of taking existing MRC-based QA approaches, we reformulate an automobile domain-specific QA problem as a multi-class classification problem of Frequently Asked Questions (FAQs) [9,10]. Answers for FAQs in a specific domain are easily retrieved through the pre-defined FAQ and matching answer pairs [7,11]. Reformulating a domain-specific QA task as a FAQ classification problem [12,13] gives advantages in reducing computational cost and time-consuming data annotation. Furthermore, FAQ classification suits the environment of real-world services as we anticipate our application to be exposed to users using voice assistants, where the single correct answer is delivered through the system [14,15,16].

We built on the above observation by constructing a new QA dataset, where questions were created by a domain expert, crowd workers, and real users, and answers were manually generated from a car manual. Our task possesses its strength in dealing with queries from actual car users, which, in turn, reflects an accurate representation of users’ needs, and unanswerable questions. We show that our method works equally well with multiple languages by constructing both Korean and English datasets.

The main goal of our MaQA task is to classify an input question to the corresponding FAQ pair among 1526 question–answer pairs to extract the answer, where representative questions for each FAQ class are defined by the following rules: (a) the inclusion of car problems, functions, safety, and maintenance; and (b) the addition of different forms of unique queries. In addition to the FAQ classification, we suggest two additional sub-tasks. One is a Named Entity Recognition (NER) task [17,18] on car domain-specific terminologies, and the other is a classification task on questions with corresponding topic types as their target categories.

To encode input sequences globally and sequentially, our model employs Transformer [19] and Gated Recurrent Units (GRUs, [20]). Our model trains the topic type classification along with the NER task using multi-task learning (MTL) [21,22] to improve the performance of FAQs. When classifying topics, we apply variational inference [23] to model questions by their topic types. Furthermore, to enable real-time inference, we refrained from using any additional linguistic features or pre-trained language models such as Bidirectional Encoder Representation using Transformer (BERT) [24].

Our contributions can be summarized as follows:

• We suggest a new comprehensive car-domain FAQ classification task and the first end-to-end neural FAQ model for the car domain.

• We achieve high accuracy on a real-world, user-generated test set with multi-task learning and variational inference.

• Our proposed model is 12× and 13.4× faster during inference in the Korean and English datasets, compared to our model with BERT.

2. Related Work

2.1. Closed-Domain QA (CDQA)

Our proposed task is an example of closed-domain QA, or subject-limited QA problems, which seems to be in contrast to the contemporary QA trends [25] and the goal to reach human-level NLP. When the QA field was first developed, many of the first QA systems, such as BASEBALL [26] and LUNAR [27], were closed-domain systems. BASEBALL was one of the earliest automatic question-answering systems, developed to answer structured, domain-specific queries about baseball statistics. It utilized a restricted natural language interface and a structured database, allowing users to retrieve information based on pre-defined patterns. Although effective for its time, its capabilities were limited to its narrowly defined domain and structured data, lacking the flexibility to handle unstructured or ambiguous queries. LUNAR expanded on these ideas by applying natural language understanding to a more complex domain: lunar geology. The system was designed to interpret queries about lunar rock samples, leveraging a combination of structured data and domain-specific terminology. Compared to how resources were less abundant and methods less effective at the time, applying recent advancements and using new, comprehensive resources for more complex domains will lead to unexplored, meaningful results. Additionally, as modern devices become more technologically complex and sophisticated, manuals and FAQs will in turn become denser, furthering this need even more.

Much progress has been made in open-domain settings, but such improvements will not fully suit closed-domain settings. Open-domain tasks tend to search for a short, precise factoid answer, in contrast to the long, detailed, and frequently procedural answers sought in closed-domain tasks. Therefore, the broad information sources and reading comprehension methods that are often used in open-domain tasks are less helpful in studies of closed domains—domain-specific knowledge, datasets, and methods will be required, but these needs remain under-explored. However, development in document retrieval, originally developed for open-domain challenges, is more transferable, as shown later in this paper.

This transferability shows the potential of studying closed-domain settings in the modern QA research environment—other advantages include practical reasons such as the need to reflect genuine user needs and real-world applicability in the car domain.

2.2. Car Domain and QA Systems

General dialogue systems and assistants are not novel in the car domain, both in literature [28,29] and applications, but relevant literature for FAQ systems specifically in the car domain are sparse. Previous work includes a Chinese-language car FAQ assistant developed by [30], but the dataset is too general and too small with only 123 common question–answer pairs, while our dataset is larger by a factor of nearly 400, allowing for large-scale multi-label classification.

An effective CDQA system is particularly necessary for the car domain for several reasons. Cars are important and costly, yet service centers and general maintenance are not always accessible, considering time, place, and monetary cost. Accurate, timely, and hands-free answers are regularly needed yet often unavailable. As car technology improves in areas such as car connectivity and autonomous driving, comprehensive in-car assistants will complement such changes. Lastly, consumers already extensively use in-car voice assistants, but many are general assistants (e.g., Alexa or Google Assistant) or smartphone mirror applications (e.g., Android Auto or Apple CarPlay) and not specific to the car, which indicates the need for further development of model-specific car assistants that can be integrated into current systems. General assistants still lack detailed information and answers to procedural questions; in-house closed domain assistants will serve that need instead. Smartphone mirror applications extend smartphone applications to the car interface, while some general assistants are equipped with a limited number of car-specific commands that are made in specific partnerships with individual car companies.

2.3. Machine-Translated Parallel Dataset

Multilingual QA research is growing, but there is still a relative lack of QA resources and datasets for non-English languages. Most non-English QA works translate from original English sources or generate datasets and models based on original methodologies, but, in our work, the dataset and the model were initially generated for the Korean language and then translated and adapted for the English language.

For the Korean language, KorQuAD (Korean Question Answering Dataset) [31] and K-QuAD [32] are two SQuAD-derived datasets, but are less relevant to our work as they study open domains. KorQuAD is a large-scale Korean dataset based on SQuAD, designed for evaluating machine comprehension models in Korean. It primarily focuses on extractive question-answering tasks. K-QuAD extends the scope of question-answering in Korean by incorporating a variety of question types, including multiple-choice and free-text responses, thereby enabling broader model evaluations. While these datasets are valuable benchmarks for Korean language understanding, they do not address domain-specific FAQ tasks or multi-task learning approaches, which are key contributions of MaQA.

Our dataset studies a closed domain and was translated from Korean to English by an in-house neural translation model that is specialized for the car domain. The resulting English dataset was additionally double-checked and edited by human translators. We also create three non-edited parallel English datasets, translated by our translation model and two other translation applications (i.e., Google Translate and Papago). We ran each non-checked dataset through our end-to-end model and compared results to confirm the effectiveness of our translation model.

3. Background

Traditionally, FAQs solve a QA task with a retrieval approach, calculating the similarity between an input query and question–answer pair dataset. The learning-to-rank [33] method, utilizing ranking, calculates similarity by training a binary classification using a simple neural network and retrieves an answer that is similar to an input query. The BERT-based FAQ retrieval system [34] is an approach to calculating query-question (Q-Q) similarity and BERT-based query-answer (Q-A) relevance together and outperforms baseline retrieval systems on benchmark IR datasets. Recently, there have been a number of fully unsupervised FAQ approaches [35,36]. Firstly, reference [35] calculates similarity through Q-Q and Q-A matching scores using BERT without the fine-tuning stage and alleviates the missing labeled data by automatically generating question paraphrases. A hybrid approach with a voting mechanism for FAQ retrieval, combining BM25 (Best Matching 25) [37] and contextual language models, has been proposed in prior work [36]. The BM25 is a widely used ranking function in search engines for assessing the relevance of documents to a specific search query. Developed within the probabilistic information retrieval framework, it serves as a fundamental tool for determining the most relevant results in response to user queries.

MTL [21] is a valuable method in machine learning that helps to establish state-of-the-art results in a variety of tasks by acting as a regularizer. Similarly, in Natural Language Processing (NLP), MTL has had promising outcomes in many tasks  [22,38,39,40,41]. In our implementation of MTL, we combined NER loss, topic type loss, and FAQ pair loss, trained using hard parameter sharing, which led to an improvement in our results.

Variational inference is an optimization problem used extensively in many machine learning problems and in NLP studies  [42,43,44,45,46]. Variational inference has been successfully applied to both generative document modeling and supervised question-answering tasks, demonstrating exceeding performance on two question–answer benchmark datasets [47]. A neural variational question model has been proposed to encode question types, successfully grouping question vectors by type [48]. Inspired by this approach, we used variational inference in our model to aid in clustering the questions.

4. Car Domain QA Dataset Based on Manual Texts

This section describes the construction process of the MaQA dataset. We designed a structured task that was carried out by a car domain expert and ten crowd workers. First, we built the MaQA dataset in Korean and then created a parallel dataset in English with crowd workers. The dataset was augmented with NER tags and topic types that allowed us to construct classification tasks for three targets: FAQ, topic type, and NER.

4.1. Parallel Dataset Construction

First, a car domain expert constructed 1526 representative question-and-answer pairs (a set of FAQs), by referring to the owner’s manual of a specific car model. This collection of FAQs was the set of classes that input sequences were classified into. Our set of representative questions included a question that served as a ‘no-answer’ case to account for unanswerable questions. We adopted the same methodology used for constructing the Korean MaQA dataset to create its English counterpart, resulting in a parallel dataset across both languages.

4.2. Definition of the Sub-Domain

Prior to establishing the representative questions, we defined three sub-domains that sufficiently encompassed the car domain and subsequently guided the process of building the list of representative questions. The three sub-domains were Manual, Troubleshooting, and MyCar. For each sub-domain, the domain expert considered the following four cases: (1) car problems, (2) car functions, (3) car safety, and (4) car maintenance. The Manual domain refers to general information about the car, the Troubleshooting domain deals with questions regarding unexpected emergency situations or problems, and the MyCar domain contains a user’s vehicle-related questions. Our MaQA dataset consisted of 39,449 training samples, 7718 development samples, and 481 test samples.

4.3. Definition of the Sub-Task

The MaQA task had two sub-tasks of topic type classification and NER in the car domain. Domain experts constructed FAQ pairs of the MaQA dataset according to the contents of the car manual book. They took the contents of the car manual as topics of FAQ pairs. Since the FAQ pairs were built on the contents of the manual, we hypothesized that there exists a dependency between topics and FAQ pairs. Therefore, we categorized each FAQ pair by the corresponding topic.

NER is a task of classifying an entity word to a pre-defined entity label and generally identifies entity names of person, location, and organization. NER plays an important role when distinguishing the surrounding text of a given question. Particularly, in the car domain, vocabularies are often ambiguous as there exist various combinations of similar words such as engine, engine oil, and brake oil. Therefore, we can take advantage of NER to distinctly disambiguate the car domain terms.

All representative and extended questions were annotated with thirteen topic types, such as Dashboard, and MyCarInfo, and NER labels, which covered thirteen entity types for the car domain.

4.4. Question Types and Sampling Strategies

To enrich our dataset, we employed several sampling strategies to gather diverse question types. First, ten crowd workers expanded each representative question by creating ten to twenty similar yet distinct paraphrased questions. This expansion helped the model handle varied real-world phrasings and contexts. Additionally, we generated unanswerable questions by asking crowd workers to create questions using six basic interrogatives (who, what, when, where, why, and how) with added topic types and NER labels, enabling the model to identify plausible but unanswerable queries.

Finally, for testing, we collected 481 natural questions directly from car users with different driving experience levels. These questions, created independently from any reference material, reflected authentic user needs. Among them, 77.6% were answerable, while 22.4% were unanswerable, further challenging the model to handle real-world ambiguity.

5. Dataset Analysis

To summarize the construction of the MaQA dataset without a test set, we had 1526 representative questions that acted as a set of classes for test questions to be classified into. A total of 45,641 additional questions were added to the original 1526. A total of 47,167 questions were classified into topic types and tagged with NER labels. This resulting structure allowed us to perform three classification tasks that were titled (1) FAQ, or classifying input sequences into one of the 1526 FAQ classes, (2) topic type, or classifying input sequences into one of the 13 topic types, and (3) NER, or classifying unique words of the input sequences into NER labels.

The topic type labels included common, basic guidelines and information, safety precautions, safety system, dashboard, convenient features, starting and driving vehicle, emergency situations, maintenance, troubleshooting, check trouble, how to maintain, and  my car info.

In NER, the base label set comprised car type, body, chassis, safety, traffic safety, instrument panel, infotainment, air conditioning, convenience function, driving, maintenance, personal data, and transaction.

5.1. Details of Question–Answer Pair Definition

The criteria for selecting representative questions are the same as functions, terminology, common sense, laws, and so on for automobiles, and domain experts created questions in consideration of the following guidelines. The style of question should be written in colloquial style rather than written expression, and the voice agent should maintain the style of explaining like a vehicle guide with correct answers.

The content of the question was as follows: (a) a question must be related to a function; (b) use ‘official terms’ for words, but, in general, if there is a word that the majority of users understand, it takes precedence, such as 스티어링 휠(steering wheel) → 핸들(handle) in Korean; (c) in order to facilitate future expansion, questions should be asked in representative terms or expressions such as the representative question “What is the cruise control function?”, and the extension question “What is the function that keeps the speed constant even without stepping on the accelerator?”; and (d) exclamation and adverbs that are not directly related to the core content are excluded from the representative question and answer.

5.2. Naturalness

We see in Table 1 that there are a variety of both simple and complex sentence structures, from longer, situational, and procedural questions to shorter commands and even sentence fragments. To illustrate further, the question in the Natural row of Table 1 is an example of how we accounted for natural language (e.g., heated steering wheel is referred to as the hot hands function).

5.3. Details of Topic Type Definition

As shown Table 2, MaQA has thirteen topic types. The topic types were defined as high-level topics, vehicle unit functions, and detail functions, each of which is as follows: (1) the vehicle unit function is selected as a unique subject word and questions are grouped into units of subject word; (2) the high-level topics encompassing the vehicle unit functions are defined in consideration of the topic type or knowledge domain of the manual; and (3) if the vehicle unit functions need to be divided into the detail functions, it is defined under the vehicle unit functions.

5.4. Details of Question Type Definition

In Table 3, we show question type frequency only for MaQA’s answerable questions. The MaQA dataset has three domains such as Manual, Troubleshooting, and MyCar. Since the Troubleshooting and MyCar domains mainly have questions about how to solve a car problem and how to use a function, the ‘How to’ question types are most often distributed in the dataset. The ‘What is’ is the second most common question type, generally due to FAQs on vehicle status, functions, names, and so on in the Manual and MyCar domains. When creating a representative question-and-answer pair, we considered attributes such as the abbreviation, condition, definition, install, notice, spec, and turn on/off, and the rest of the question types were FAQs defined according to the attributes.

5.5. Details of Annotation Instruction

Crowd workers extended the defined representative questions to similar questions and annotated them by considering the following conditions: (1) Extension questions were constructed in questions and plain text. For example, the question text is “What are the engine room components?”, and plain text is “Tell me about the engine room components”. (2) When constructing an extension question, they clearly included the meaning of the representative question, and an entity of question target must have one. (3) The question to be constructed included both colloquial and literary styles. (4) The workers combined synonyms, abbreviations, endings, and word order changes. (5) The workers created extension questions considering patterns such as how to, what is, where, why, and when. (6) The workers did not generate sentences other than questions such as commands for operation and control. We demonstrate overall statistics of the MaQA dataset in Table 4 for Korean and English.

6. End-to-End FAQ Model with MTL

The MaQA dataset was structured for a multi-class classification task. Our goal was to output the correct answer corresponding to an input query by classifying the FAQ pair. We propose an end-to-end model with multi-task learning to efficiently approach this task. The total loss function of our method is a linear combination of three losses: FAQ pair loss, topic type loss, and NER loss.

6.1. Task Formulation

The input sequence $Q=\{q_1,q_2,\dots ,q_n\}$ is a single question where n is the length of the question. Our model has three outputs: FAQ pair, topic type, and NER predictions. The FAQ and topic-type tasks are multi-class classification problems. The FAQ pair output is denoted as $y^q\in {\mathbb{R}}^l$ where l is the number of FAQ pair classes. The topic type output is denoted as $y^c\in {\mathbb{R}}^m$ where m is the number of topic type labels. NER is a sequence labeling task where the output of an input sequence is $Y_e=\{y_1^e,y_2^e,\dots ,y_n^e\}$ with $y_i^e$ representing the NER label for the $i^{th}$ token of the input sequence.

6.2. Model Architecture

The MaQA model in Figure 1 uses a MTL framework and lightweight optimization strategies to deliver effective and accurate FAQ classification, especially for domain-specific applications like vehicle manuals. The design starts with an input processing step that uses language-appropriate tokenization techniques, such as subword-based approaches for English and morphological tokenization for Korean, to enable multilingual data, notably Korean and English. In order to extract linguistic and semantic subtleties across languages, a shared encoder embeds the input data into a low-dimensional feature space, capturing contextualized representations. The MTL framework, which is at the heart of the design, concurrently completes three related tasks: topic type categorization, NER, and FAQ classification. The NER effort finds important things like names, dates, or locations that are specific to the linguistic intricacies of both Korean and English, while the FAQ classification task assigns queries to predetermined categories based on their aim and content. In addition, the topic classification task divides queries into more general thematic categories like maintenance, troubleshooting, or vehicle operations. Job-specific optimization is ensured via a dedicated decoder for each job, which is softmax-based for FAQ classification, sequence labeling-based for NER, and employs topic-focused classifiers for thematic grouping. The model also uses variational inference to improve generalization and handling of uncertainty, especially in situations with limited data. Predictions for thematic topics, extracted entities, and FAQ classes are given by the output layer. This integration of tasks, coupled with its multilingual capability and optimized inference speed, positions MaQA as an advanced solution for domain-specific FAQ retrieval, offering both high accuracy and practical applicability in real-world scenarios.

As seen in Figure 1, in our MaQA model architecture, instead of using BERT, we embed input tokens using the pre-trained word and character embeddings, which increases the efficiency of the model. After concatenating embedding vectors, the resulting dense vectors are passed into an encoder of two contextual encoding layers: (1) a globally contextual encoding layer based on Transformer and (2) a sequentially contextual encoding layer based on GRUs. The output of the sequentially contextual encoding layer is used to classify the NER label. For FAQ pair and topic type classification, we take the output from two encoding layers and pass the output to the feed-forward neural network (FFNN) layer. FFNN layer uses shared parameters for both FAQ pair and topic type classifications. The output of the sequential encoding layer is passed on to the conditional random field (CRF) [49], which acts as a regularizer to output the NER loss. The topic type score is determined by passing the output of the FFNN layer into a variational inference layer and based on the calculated score, an input sequence is classified into one of thirteen topic types. The FAQ classifier takes the sentence embedding of the input sequence from sentence BERT [50] and the output of the FFNN layer to produce FAQ pair loss value, which is used to classify the input sequence into FAQ pair classes.

6.3. Encoder

6.3.1. Input Representations

We represent our input sequence with word embeddings $e_i^w={\mathrm{emb}}_w\left(q_i\right)$ and character embeddings $e_i^c={\mathrm{CNN}}_c\left(q_i\right)$, CNN is a Convolutional Neural Network, without using additional linguistic features. We concatenate the embeddings to create a dense vector e. For Korean, the word embeddings are pre-trained by NNLM [51,52] and the character embeddings are pre-trained by GloVe [53]. For English, we use FastText [54] for the pre-training for both word and character embeddings. Following the setting of BERT, we add the special token, [CLS], at the beginning of the input sequence, and the hidden state of the [CLS] token is used for FAQ pair and topic type classifications.

6.3.2. Contextual Layers

Transformer receives the concatenated vector $\mathbf{e}$ of a word and character embeddings and globally encodes the vector using multi-head attention. The encoded hidden states ${\mathbf{s}}_i$ from Transformer are passed into the GRU layer as an input, which encodes the hidden states in a bi-directional manner to create a sequentially encoded context vector ${\mathbf{r}}_i$.

(1)$\begin{array}{c}\hfill {\mathbf{s}}_i=\mathrm{Transformer}\left({\mathbf{e}}_i\right)\\ \hfill {\mathbf{r}}_i=\mathrm{BiGRU}({\mathbf{r}}_{i-1},{\mathbf{s}}_i),\end{array}$

where the outputs, ${\mathbf{s}}_i$ and ${\mathbf{r}}_i$, of these encoders are used in two different ways. First, ${\mathbf{r}}_i$ is passed on to the CRF for the NER classification. Second, for the topic type and FAQ classifications, we concatenate ${\mathbf{s}}_{\left[\mathrm{CLS}\right]}$ and ${\mathbf{r}}_n$, and feed it into the FFNN layer, as shown in Equation (2). The ${\mathbf{s}}_{\left[\mathrm{CLS}\right]}$ represents the hidden state of the Transformer corresponding to the [CLS] token. The ${\mathbf{r}}_n$ of the bidirectional GRU corresponds to the last position of the forward and backward passes.

(2)$\begin{array}{c}\hfill \mathbf{h}=\mathrm{FFNN}\left([{\mathbf{s}}_{\left[\mathrm{CLS}\right]};{\mathbf{r}}_n]\right)\end{array}$

6.4. Multi-Task Learning Objectives

We perform multi-task learning using shared parameters to transfer information when training our model for FAQ pair classification. Our secondary tasks are NER tagging, which involves training entity classification for terms related to automobiles and sharing term information on latent variables, and question topic type classification, which learns the topic of the question and shares the topic type information of the question when training FAQs.

6.4.1. Named Entity Recognition for Car Domain

We define and construct a NER dataset for the automotive domain. To approach the NER task, each token of the input sequence is labeled with BIO tags  [55]. We use CRF to predict the sequence $y_e$ of the entity label and pass the hidden states $\mathbf{r}$ of the previous GRU to the CRF to determine the NER label corresponding to each token. We use the negative log-likelihood as the NER loss.

6.4.2. Topic Classification with Variational Network

The topic type module classifies the questions into one of thirteen topic types. We apply variational inference to model the semantic information of the input question as a latent variable. We use the hidden state $\mathbf{h}$ of the shared-parameter FFNN as an input to the variational inference network. Vectors sampled from the variational inference network output the result in the topic type classifier.

(3)$\begin{array}{c}\hfill {\mathbf{h}}_z=\mathrm{ReLU}({\mathbf{W}}_z\mathbf{h}+{\mathbf{b}}_z)\\ \hfill \mathit{\mu }=l_{\mu }\left({\mathbf{h}}_z\right),\mathit{\sigma }=l_{\sigma }\left({\mathbf{h}}_z\right)\\ \hfill \mathbf{z}=\mathit{\mu }+\mathit{\sigma }\odot ϵ\\ \hfill \tilde{\mathbf{h}}=\sigma ({\mathbf{W}}_2\mathrm{ReLU}({\mathbf{W}}_1\mathbf{z}+{\mathbf{b}}_1)+{\mathbf{b}}_2)\end{array}$

To create a vector ${\mathbf{h}}_z$, the input vector $\mathbf{h}$ computes a nonlinear operation with ReLU (Rectifier Linear Unit) [56]. Sampling $z\sim q_{\pi }\left(z\right|\mathbf{h})$ is calculated with the reparameterization trick. In Equation (3), a linear function $l(·)$ is applied to ${\mathbf{h}}_z$ to generate Gaussian distribution parameter vectors $\mathit{\mu }$ and $\mathit{\sigma }$. We also define the normal distribution variable $ϵ$, perform element-wise multiplication with $\mathit{\sigma }$, and create the sampled $\mathbf{z}$ by adding the resulting product to $\mathit{\mu }$. Finally, we compute $\mathbf{z}$ via a nonlinear operation with ReLU and $\sigma$ (sigmoid) to generate $\tilde{\mathbf{h}}$, encoded and decoded with variational inference. In the final topic type classifier, the output $y_c$ is predicted with a softmax function performed on $\tilde{\mathbf{h}}$. We refer to the visualization of variational inference for topic types in Section 9.1.

6.4.3. FAQ Pair Classification

MaQA contains 1526 FAQ pairs that act as respective classes for an input sequence. The FAQ pair module consists of a simple classifier that uses a linear function; its parameters are shared with the topic type module via the FFNN layer. We extract the pre-trained sentence vector corresponding to the representative FAQ’s question from Sentence BERT and embed it in the classifier. Finally, we use softmax to match the input sequence to its respective FAQ pair.

6.4.4. Loss with Multi-Task Learning

The topic type and FAQ classifiers use cross-entropy for their loss functions, ${\mathcal{L}}_c$ and ${\mathcal{L}}_q$, respectively. The topic type module uses the Kullback–Leibler divergence [57] to control the generation of a latent variable through Gaussian distribution. We determine the total loss with a weighted sum:

(4)$\begin{array}{c}\hfill \mathcal{L}=\alpha {\mathcal{L}}_q+\beta {\mathcal{L}}_c+\gamma {\mathcal{L}}_e\end{array}$

where the hyperparameters $\alpha$, $\beta$, and $\gamma$ are optimized weights for the FAQ, topic type, and NER losses, respectively.

7. Experiments

7.1. Implementation

We present details of our model training process for MaQA including hyperparameter settings in this section. We trained our model on the MaQA dataset for 100 epochs with a batch size of 32 per GPU (a single Tesla V100). Our encoding layers were based on Transformer and GRU. We used a 5-layer Transformer with a layer dimension size of 256 and 2 stacks of GRU with a layer dimension size of 300. We set the dropout probability to 0.2 for GRU layers and 0.1 for Transformer. We used adamax [58] and set the learning rate to $1\times {10}^{-3}$ for the Korean dataset and $1\times {10}^{-2}$ for the English dataset, and linearly decayed the learning rate by 50% in case of no improvement for five consecutive epochs. When utilizing BERT-Base to compare performance, we used Adam [58] and set the learning rate to $1\times {10}^{-4}$ and hyperparameters for the rest of the training following the text classification examples of Huggingface’s Transformers (https://github.com/huggingface/transformers/), accessed on 5 November 2024.

Lastly, we set $\alpha$, $\beta$, and $\gamma$ from Equation (4) to 0.9, 0.1, and 0.2, respectively.

7.2. Results

In this section, we present experimental results in both Korean and English FAQ datasets. We denote the proposed end-to-end neural FAQ model as MaQA in the following section.

7.2.1. Overall Score

Table 5 compares our MaQA model with other models in the Korean and English MaQA datasets. We report classification accuracy for the FAQ task, the main task of MaQA, and for topic type classification. For the sub-task NER task, we report F1 [17] scores using the CoNLL evaluation script (https://www.clips.uantwerpen.be/conll2000/chunking/output.html, accessed on 5 November 2024).

For encoding input sequences, the proposed model includes a global context layer and a sequential context layer. First, we utilize the GRU, which is a simple layer for encoding sequential context, to validate the accuracy of the FAQ. The GRU demonstrates 75.88 and 78.79 on FAQ pair classification on the Korean and English datasets, respectively. We employ a BERT, BERT-Base, which encodes input sequences in parallel, to actually prove that it works well for globally contextual encoding. Since the BERT is pre-trained based on the Transformer, it has the benefit of being able to leverage pre-trained context information for fine-tuning. BERT-Base shows 3.54 and 1.67 improved accuracy on the Korean and the English datasets, respectively, compared to the GRU baseline model. It should be noted that we evaluate our the Korean dataset using Google multi-lingual BERT and the English dataset using BERT-Base.

Unlike the two aforementioned models, our MaQA model performs multi-task learning using the main task FAQ and sub-tasks for topic classification and NER. When our model is trained, it back-propagates a loss from all three tasks at the same time, and the model parameters are optimized by sharing the knowledge of the three tasks. In Table 5, our MaQA model achieves the best FAQ performance on the Korean and the English datasets. Furthermore, the MaQA may output not only the results of the FAQ but also the topic classification and entity name results of the question. It is worth pointing out that MaQA shows higher performance on the task than when incorporating the pre-training process of BERT (MaQA + BERT from Table 5) instead of the global context encoding layer with the word embedding in our model. Since BERT is pre-trained with large data corresponding to the general domain, it is difficult to apply it to domain-specific tasks.

In the English dataset, MaQA + BERT shows 7.48 of higher accuracy in topic type and 3.98 higher F1 in NER than MaQA. This is assumed to be because, unlike the Korean model, which uses multi-lingual BERT, the English model, by using the BERT pre-trained with large-capacity English data, can utilize the context information more abundantly. Furthermore, MaQA shows 12× and 13.4× faster inference speeds in both datasets compared to MaQA + BERT.

7.2.2. Performance in Korean vs. English

We carried out an investigation concentrating on language distinctions, the multi-task learning framework, dataset properties, and mistake patterns in order to better comprehend MaQA’s accuracy gains in Korean and English contexts in Table 5.

Disparities in Language

As an agglutinative language with a head-final structure, Korean presents significant challenges due to its numerous morphological variations, including the use of suffixes and postpositions. These linguistic features increase the complexity of tokenization and semantic representation. MaQA effectively addresses these problems with its NER and subject categorization sub-tasks, which are designed to capture context-sensitive dependencies and Korean-specific language peculiarities. However, because English is a subject–verb–object (SVO) language, its morphological elements are simpler and its word order is rather consistent. Tokenization and feature extraction procedures are made simpler by these features. This simplicity is leveraged by the approach to enable simpler intent mapping and semantic interpretation in English inquiry.

Framework for Multi-Task Learning

MaQA’s MTL architecture permits task-specific fine-tuning while facilitating shared representation learning across tasks. With this method, the model can successfully adjust to linguistic variances: The NER sub-task enhances the model’s contextual awareness for Korean by resolving ambiguity in questions with intricate morphological patterns. The subject categorization job for English gains from the syntactic clarity of the language, which raises overall accuracy.

Features of the Dataset

Language-specific question formats and contextual factors are included in the MaQA dataset. Implicit contextual dependencies are frequently used in Korean FAQs. Strong contextual awareness is necessary for questions such as “이 기능을 어떻게 켜나요?” (“How do I turn on this function?”). Because English FAQs are usually more specific, the model can more easily map tokens to specified intentions.

7.3. Component Ablation Study

Three important components of the MaQA model are MTL, variational inference, and pre-trained embedding. As shown in Table 6, if the variational inference is removed from the topic type module, the FAQ accuracy decreases. If the entire topic type module, including variational inference, is removed from the MaQA model, the resulting FAQ accuracy decreases by 0.25 and 0.61, respectively, on both datasets. Our model trains NER for automotive domain terms as part of MTL. Removing NER results in a decreased accuracy, showing that training NER for automotive terms is helpful for FAQs. When removing both topic type and NER modules of MTL, the performance significantly drops. Comparing using pre-trained embeddings with randomly initialized character embeddings, we claim that using pre-trained character embedding is meaningful. Lastly, the last row of Table 6 confirms that using sentence embedding further helps the training process.

7.4. Multi-Task Learning Study

The proposed model performs MTL to improve the performance of the FAQ by training the FAQ-related tasks jointly through parameter sharing. MTL trains with one total loss by adding all the losses of FAQs, topic types, and NERs. At this time, the loss ratio is determined by applying $\alpha$, $\beta$, and $\gamma$ to each task loss.

Table 7 shows the performance change according to the loss ratio, and we set the ratio in the order of $\alpha$, $\beta$, and $\gamma$. Since the main task is FAQs, we apply a larger weight to FAQs than to other tasks and check the correlation between tasks by adjusting the ratio of the remaining tasks. As a result of the experiment, in Korean, the ratio of 7:1:2 shows the best performance with the FAQ accuracy of 90.74, and, in English, the ratio of 9:1:1 shows the accuracy of FAQs as 91.37.

In Table 7, we find the characteristic that the performance changes proportionally according to the weighted ratio. From the FAQ’s point of view, the aforementioned ratio seems to be optimized, but, at the 6:2:2 ratio for both Korean and English, the accuracy of the FAQ decreases by 0.3~0.4, while we find that it maintains meaningful performance in topic type and NER tasks.

7.5. Sentence Embedding Performance

We generated embeddings of an input question and its corresponding answer with the sentence BERT model in the FAQ module—the sentence embedding is used as the initial vector of the classifier. Table 8 shows the difference in performance when using different sentence types (random initialization, question, answer, and combination of question and answer). It shows that the model benefits the most from the sentence BERT embedding when the model is initialized only with a question sentence.

8. Hyperparameter Tuning

We used Transformer for globally contextual encoding and RNN for sequentially contextual encoding. In this chapter, we show the optimizing of hyperparameters of Transformer and RNN for Korean and English, respectively, on the MaQA development set. The Transformer optimizes the dimension size of the hidden layer, the number of encoder stacks, the number of attention heads, and the activation function. The RNN optimizes the dimension size of the hidden layer and compares RNN types.

8.1. Optimizing Dimension Size of Hidden Layer

The optimization of the number of hidden layer dimensions for Transformer and RNN is as shown in Table 9; we use biGRU for the default setting. We first increase the dimension size proportionally from 64 to 1024 to two times for both RNN and Transformer and set the hidden state size of wide-FFNN to two times larger. In this case, when the dimension size of the hidden layer of RNN and Transformer is 256, the best accuracy in both Korean and English is shown. Then, if the optimized dimension size 256 of the Transformer is fixed and the number of RNN hidden states is optimized, the FAQ accuracy is 90.59 for 300 dimension size in Korean and 91.26 for 400 dimension size in English. We used an optimized hyperparameter in subsequent optimization experiments.

8.2. Optimizing the Number of Encoder Stacks

In Table 10, we optimize the number of Transformer encoder stacks from two-layer to eight-layer. Korean shows the best performance with an FAQ accuracy of 90.71 when the stack size is a five-layer, while English is optimized to 91.37 when the stack size is a relatively small three-layer.

8.3. Optimizing the Number of Heads

Self-attention, one of the important elements of Transformer, is extended to multi-head attention. Table 11 shows the optimization of the number of attention heads, and both Korean and English show the best performance when the number of heads is two.

8.4. ReLU vs. GELU

We compared the performance of Transformer activation function using ReLU [56] used in  [19] and GELU (Gaussian Error Linear Unit) [59] used in BERT [24]. As a result of experiments in Table 12, both languages showed better performance when using ReLU.

8.5. Optimizing RNN Type and the Number of RNN Stacks

As shown in the Table 13, we finally performed optimization on the RNN type and the number of layer stacks of the RNN in encoder hyperparameter optimization. We used GRU, long short-term memory (LSTM, [60]), and simple recurrent units (SRUs, [61]) as the RNN type, compared the number of layers for GRU, and LSTM was stacked from one to three, and the SRU was stacked from one to five in consideration of the characteristics of each RNN type. All RNN types utilize a bidirectional manner. As a result of the experiment, using GRU showed the best performance at 90.71 for Korean and 91.37 for English in one-layer. In the case of Korean, the performance was close to the best score in the SRU four-layer stack, and, in the case of English, the performance was similar to the best score in the LSTM one-layer.

8.6. Variational Inference Model Optimization

In this chapter, we perform the optimization of dimension sizes for input, output, and latent variables. We utilized variational inference to model semantic information of topic type as a latent variable. Since this model samples the parameter vectors $\mathit{\mu }$ and $\mathit{\sigma }$ according to Gaussian distribution by using a linear function, the layers constituting this model should be optimized.

8.6.1. Optimizing Dimension Size of Input and Output

The variational inference network has an input and output layer, and we optimize the dimension size for the input and output layer in Table 14. As a result of the experiment, both Korean and English are optimized when the dimension size of the input and output layers is 90.

8.6.2. Optimizing Dimension Size of Latent Variable

The variational inference network uses a trainable linear function to sample latent variables $\mathit{\mu }$ and $\mathit{\sigma }$ according to Gaussian distribution. Table 15 shows the dimension size optimization for the layer of the latent variable. The latent variable also shows the best performance in both Korean and English when the dimension size is 90.

9. Analysis

9.1. Analysis for Latent Space of Topic Type

We used variational inference, which is a generative model, to classify a more robust topic type. By sampling a Gaussian distribution with a prior distribution, variational inference trained our model. The visualization product of the latent space representation for each topic label is shown in Figure 2. Thirteen topic types are expressed in different colors in the development dataset, and it can be seen that the questions are well clustered with the same topic type.

9.2. Quantitative Error Analysis

We carried out a thorough quantitative error analysis to have a better understanding of the MaQA model’s limitations. Finding trends in the model’s inaccurate predictions is the main goal of the research, especially when it comes to rare or ambiguous question types.

The errors in MaQA’s performance were categorized into four key groups for analysis. Insufficiently contextualized questions with ambiguous or implicit meanings like “How do I fix this?” were examples of ambiguity in inquiry intent. Underrepresented situations that posed a difficulty for the generalization of the model were represented by rare question categories, like “When” inquiries, which make up just 0.1% of the sample. Complex linguistic structures included Korean queries with rare morphological variations and English queries with complex syntactic constructs. Lastly, instances where the ground truth annotations were imprecise or inconsistent, resulting in misclassifications, were referred to as dataset noise. These categories offered a methodical comprehension of the model’s shortcomings and possible enhancement areas.

9.3. Qualitative Analysis

Table 16 analyzes four characteristics of a question: Extrapolation, Fragment, Natural, and Unanswerable. The FAQ pair, a representative question and answer, refers to the given input query. The first row of Table 16 shows that the model is capable of extracting the definition of Auto Hold correctly, mapping it to “A function that does not keep the braking system for a long time”. Our model also shows robustness for a fragmented input sentence, correctly classifying the input Open a manual trunk, inside into the FAQ pair, “How do I open the manual trunk from inside the car?”. In the circumstance of using in-car voice assistants, the input is mostly colloquial, and users are more likely to use abbreviations or nicknames than formal terms. To make the model more robust to informal terms, we integrate NER in the automobile domain using MTL. This helps the model to identify and classify real-world questions flexibly. Finally, the last row of Table 16 shows that our model is capable of dealing with unanswerable questions, which is an important ability, especially for the voice assistant applications for automobiles. We used no-question data examples, comprising up to 22.4% of the total data examples, to make the model more robust to unanswerable questions. As a result, our model outputs “I do not know” to unanswerable questions instead of an incorrect answer, which shows that our model is capable of avoiding confusion.

9.4. Weaknesses

We present two example cases where our model fails to classify an input query correctly. Despite the fact that our MaQA dataset is large-scale in its number of categories, the size of the dataset is comparably small and carries biases such as the imbalance in question types (refer to Section 5). Questions starting with ‘When’ compose only 0.1% of our dataset and, when the model is given such questions, it fails to output the correct answer.

Implicit and Natural Language Questions

The second row of Table 17 shows an example of implicit questions, meaning the input query is given as a form of command rather than a question. A sentence like I want to turn off the automatic opening of the trunk puzzles the model with a wrong impression of a car control-related command. For the other example, this issue has to do with implicit queries and natural language variations, such as commands that are not framed as explicit questions, like “I want to stop the trunk opening automatically”. These cases often result in incorrect classifications, highlighting the need for improved handling of colloquial or implied intents.

Unanswerable Questions

To work around these issues, we collected unanswerable questions, making up to 22.4% of the entire dataset, so that the model defends unanswerable questions better. As a result, our proposed method predicts the no-answer class (“I do not know”) rather than predicting an incorrect answer. Although the inclusion of unanswerable questions enhances the model’s ability to avoid providing incorrect answers, the overall performance in identifying “no-answer” scenarios could still be improved. The current system occasionally fails to confidently classify ambiguous or intentionally vague queries.

Our future works lie in the scope of making the proposed model more robust to the corner cases that are mentioned above.

9.5. Future Works

How to Improve Weaknesses

Future work should focus on expanding the dataset to include a more balanced distribution of question types, especially underrepresented categories like “When”. This would help the model generalize better across diverse query types and improve robustness in real-world applications.

(1) Enhanced handling of implicit queries: to address the challenges posed by implicit and natural language questions, incorporating conversational context and advanced intent recognition mechanisms could improve the model’s ability to interpret and classify such queries accurately. (2) Cross-domain and cross-lingual extensions: Extending the MaQA framework to other domains (e.g., healthcare or finance) and languages (e.g., Spanish or Mandarin) would significantly enhance its utility and generalizability. Employing multilingual pre-trained models or transfer learning techniques could facilitate this expansion. (3) Dynamic and real-time adaptation: Incorporating mechanisms for continual learning or dynamic adaptation to evolving user needs and data could enable the model to maintain high performance in changing contexts and scenarios, such as new car features or updated manuals. (4) MaQA achieves promising results; it faces challenges in handling diverse question types and implicit relationships. Leveraging advanced optimization techniques, such as Activated Gradients, could potentially address these limitations by dynamically prioritizing task-relevant features during training.

Future Work on Model and Benchmark

Although the efficiency and accuracy improvements of MaQA have been demonstrated over a BERT-based baseline, more exploration is necessary to fully assess its relative effectiveness. In particular, comparisons with well-established FAQ retrieval models, such as Dense Passage Retrieval (DPR), T5-based retrieval frameworks, and hybrid methods that combine sparse and dense retrieval approaches, would offer deeper insights into MaQA’s performance across diverse metrics. Such comparisons could elucidate MaQA’s strengths in areas including inference speed, handling of diverse question types, multilingual capabilities, and its practical applicability in real-world scenarios. Moreover, benchmarking MaQA against industry-standard datasets or widely used public FAQ retrieval datasets would provide a clearer picture of its generalizability and robustness beyond the automotive domain.

In future work, we aim to expand our evaluation to include these comparisons, focusing on both domain-specific and cross-domain contexts. By addressing these aspects, we seek to further establish MaQA as a state-of-the-art solution in FAQ retrieval and classification tasks, demonstrating its scalability and versatility across a broader spectrum of applications.

10. Conclusions

In this paper, we suggest a different perspective of reformulating the car domain-specific QA task as a FAQ classification task. Based on this consideration, we propose a FAQ-based manual question-answering task, MaQA, in a closed-domain setting for a specific car and constructed the MaQA dataset, which includes the main task of FAQ classification and sub-tasks of question topic type classification and NER tagging. To tackle the MaQA dataset, we present an end-to-end neural FAQ classification method for an automobile domain. To the best of our knowledge, we are the first to propose a FAQ task for the car manual in real time. Employing multi-task learning to train three tasks, one main task, and two sub-tasks, and variational inference for question topic type module that works as a regularizer, our proposed model showed remarkable performance in both of the MaQA datasets in Korean and English. Furthermore, our method improved the inference time compared to both the proposed model with BERT and BERT-Base. The MaQA task can not only be extended and generalized well to various car types at a low cost but also be applied to various productions that have user manuals included.

Author Contributions

Conceptualization, C.P. and J.K.; Methodology, C.P.; Software, C.P. and S.J.; Validation, C.P. and S.J.; Investigation, S.J and J.K.; Data curation, C.P., S.J. and J.K.; Writing—original draft, C.P.; Writing—review & editing, S.J.; Supervision, J.K. All authors have read and agreed to the published version of the manuscript.

Data Availability Statement

The datasets used in this article are not readily available because they are part of a study in Hyndai Motor Company and are subject to commercial restrictions, which prohibit public sharing. The original data to which the proposed method were obtained from the website https://www.hyundai.com/kr/ko/digital-customer-support/helpdesk/download-center, accessed on 5 November 2024.

Conflicts of Interest

Author Seohyeong Jeong was employed by the company Hyundai Motor Group. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Abbreviations

The following abbreviations are used in this manuscript:

BERT	Bidirectional Encoder Representation from Transformers
BM25	Best Matching 25
CRF	Conditional Random Fields
DPR	Dense Passage Retrieval
FAQ	Frequently Asked Question and Answering
FastText	A library for efficient learning of word representations
GELU	Gaussian Error Linear Unit
GloVe	Global Vectors for Word Representation
GRU	Gated Recurrent Unit
KorQuAD	Korean Question Answering Dataset  [31]
K-QuAD	Korean Question Answering Dataset  [32]
MRC	Machine Reading Comprehension
MTL	Multi-task learning
NER	Named Entity Recognition
NNLM	Neural Network Language Model
QA	Question answering
ReLU	Rectified Linear Unit
SQuAD	Stanford Question Answering Dataset
T5	Text-to-Text Transfer Transformer

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figures and Tables

Figure 1. Structure of our neural end-to-end MaQA model. The input sentence is “암레스트 어떻게 움직여” in Korean, which translates to “How can I move the armrest”. Our model outputs NER, topic type, and FAQ predictions with MTL for the input question.

Figure 2. UMAP visualization of the question representations achieved on the held-out development dataset. The questions are gathered from 13 different topic groups, which correspond to the points in the figure with different colors.

Table 1

Test set question variety examples.

Naturalness Type	Examples
Long	What should I set to keep the kids from opening the window from behind?I think the remote smart parking assist function is not working well, what should I do?
Short	Let me know what EPB is.Guide me on how to set up the head-up display.
Fragments	How to turn the dash cam offTo watch the black box recording again?
Natural	What is the way to turn on the hot hands function?Don’t you have an emergency key or something on a smart key?

Table 2

Topic type frequency for the overall dataset of MaQA.

Topic Type	%	Example
None	22.2	When should I switch the car to a larger model?
Basic guidelines and information	0.6	What are the benefits of a short wheelbase?
Safety precautions	2.0	Can I take a call while driving?
Safety system	6.8	Since the car has airbags, can I not wear a seat belt?
Dashboard	9.8	What should I do if the brake warning light is on?
Convenient features	32.9	How do I set up the Auto Defog?
Starting and driving vehicle	10.3	How can I park while in neutral gear?
Emergency situations	2.6	Tell me how to tow a car?
Maintenance	8.5	Where do I go to check the vehicle?
Troubleshooting	0.3	What should I do if an engine is damaged?
Check trouble	1.2	What’s wrong with the ABS?
How to maintain	1.3	When was the coolant changed recently?
My car info	1.6	What is the fuel economy of my car?

Table 3

Question type frequency for MaQA without unanswerable questions.

Question Type	%	Example
What is	23.7	What is an electro-chromic mirror?
Where	1.8	Which direction is the fuel inlet located?
When	0.1	When can children use the seat belts attached to the vehicle?
Why	4.2	Why can’t I unlock the doors other than the driver’s door with a smart key?
How to	57.8	How do I set up the instrument panel menu?
Check	2.4	There was a warning on the brakes. Why is that?
General	9.9	Is the high beam on now?

Table 4

Overall statistics of the MaQA dataset.

	Korean	English
Representative FAQ pair	1526	1526
Extended questions	45,641	45,641
Words/question	5.54	9.37
Words/answer	17.38	23.23
Character/question	19.53	47.88
Character/answer	73.59	131.44
Answerable %	77.6	77.6
Unanswerable %	22.4	22.4

Table 5

Experimental results on the test set of the MaQA dataset. Bold values indicate best performance.

Model	Parameters	Korean				English
		FAQ acc	Topic acc	NER F1	sent/s	FAQ acc	Topic acc	NER F1	sent/s
GRU (baseline)	6 M	75.88	–	–	4208	78.79	–	–	4810
BERT-Base (only)	111 M	79.42	–	–	392	80.46	–	–	465
MaQA	11 M	85.24	87.94	74.79	3308	82.95	79.42	73.56	3642
MaQA + BERT	113 M	82.33	87.53	74.63	275	80.25	86.90	77.54	271

Table 6

Component ablation study on the dev set.

Components	Korean	$\mathbf{\Delta }$	English	$\mathbf{\Delta }$
MaQA	90.74	–	91.27	–
−VI	90.54	$-0.20$	90.85	$-0.42$
−topic type (w/VI)	90.49	$-0.25$	90.66	$-0.61$
−character emb.	90.45	$-0.29$	90.67	$-0.60$
−NER	90.40	$-0.34$	90.67	$-0.60$
−topic and NER	90.02	$-0.72$	90.32	$-0.95$
−sentence emb.	89.01	$-1.73$	89.56	$-1.71$

Table 7

Performance change according to loss ratio for each task on dev set. Bold values indicate best performance.

Loss Ratio	Korean			English
$\mathit{\alpha }$:$\mathit{\beta }$:$\mathit{\gamma }$	FAQ acc	Cate. acc	NER F1	FAQ acc	Cate. acc	NER F1
9:1:1	90.71	93.16	75.07	91.37	88.03	80.99
8:1:1	90.58	92.87	75.50	90.78	88.61	81.90
7:1:2	90.74	93.40	83.91	91.01	89.42	84.67
6:1:3	90.52	93.91	84.63	90.48	89.59	85.37
5:1:4	90.13	92.60	86.00	90.58	90.00	85.85
7:2:1	90.52	93.73	77.28	90.83	91.71	82.34
6:3:1	90.63	94.04	80.91	90.93	93.01	82.76
5:4:1	90.45	94.49	81.90	90.71	92.77	82.53
6:2:2	90.40	93.88	84.02	90.85	91.96	85.11

Table 8

Performance change according to sentence type using sentence embedding. Bold values indicate best performance.

Sentence Type	FAQ acc
	Korean	English
Random init.	89.01	89.56
Question	90.74	91.27
Answer	89.86	90.25
Question + answer	90.66	90.56

Table 9

Optimizing dimension size of encoder hidden layer. Bold values indicate best performance.

RNN Hidden State	TransformerHidden State	Wide-FFNNHidden State	FAQ acc
			Korean	English
64	64	128	84.58	86.80
128	128	256	88.71	90.08
256	256	512	90.42	91.19
512	512	1024	90.35	90.51
768	768	1536	88.35	73.69
1024	1024	2048	89.84	38.28
50	256	512	86.65	87.77
100	256	512	89.30	90.12
200	256	512	90.21	90.58
300	256	512	90.59	91.00
400	256	512	90.57	91.26
500	256	512	90.52	91.19

Table 10

Optimizing the number of encoder stacks of Transformer. Bold values indicate best performance.

Number of Stacks	FAQ acc
	Korean	English
2	90.59	91.26
3	90.11	91.37
4	90.61	90.41
5	90.71	89.42
6	89.80	–
7	90.22	–
8	89.98	–

Table 11

Optimizing the number of heads of Transformer. Bold values indicate best performance.

Number of Heads	FAQ acc
	Korean	English
2	90.71	91.37
4	90.33	91.19
8	89.95	91.15
16	89.98	90.34
32	89.93	90.76

Table 12

Optimizing activation function of Transformer. Bold values indicate best performance.

Activation	FAQ acc
	Korean	English
ReLU	90.71	91.37
GELU	90.22	90.93

Table 13

Optimizing RNN type and the number of RNN stacks. Bold values indicate best performance.

RNN Type	Numberof Stacks	FAQ acc
		Korean	English
GRU	1	90.71	91.37
	2	90.45	90.62
	3	90.15	90.56
LSTM	1	89.72	91.27
	2	87.60	90.66
	3	87.58	90.61
SRU	1	89.77	90.73
	2	90.48	90.88
	3	90.00	90.69
	4	90.54	90.52
	5	90.43	88.87

Table 14

Optimizing dimension size of variational inference’s input and output. Bold values indicate best performance.

Hidden Sizeof Input/Output	FAQ acc
	Korean	English
50	90.62	90.82
90	90.71	91.37
100	90.35	91.00
200	90.35	90.97
400	90.39	91.02
600	90.59	90.89
800	90.42	90.78
1000	90.57	91.09

Table 15

Optimizing dimension size of latent variable. Bold values indicate best performance.

Hidden Sizeof Latent Variable	FAQ acc
	Korean	English
50	90.65	91.15
90	90.71	91.37
100	90.59	91.02
200	90.38	90.97
400	90.21	91.09
600	90.56	91.17
800	90.38	91.24
1000	90.40	91.02

Table 16

Qualitative analysis: examples of predictions from the test data. Our model classifies the FAQ pair, such as representative question (Rep. question) and Answer, which refers to the input query.

Extrapolation	Input query	What’s the function that I don’t need to keep braking?
	Rep. question	What is the Auto Hold?
	Answer	Auto hold is a function that maintains a stopped state even if you take yourfoot off the brake after the vehicle stops.
Fragments	Input query	Open a manual trunk, inside
	Rep. question	How do I open the manual trunk from inside the car?
	Answer	To open the trunk from inside the car, press the trunk open button which is locatednear the driver’s lap.
Natural	Input query	I want to turn on the hand heater, what should I do?
	Rep. question	How do I turn on the heated steering wheel?
	Answer	The steering wheel becomes warm if you press the steering wheelwarmer button after the ignition.
Unanswerable	Input query	Where can I check the fuse box manual?
	Rep. question	No-answer
	Answer	I do not know.

Table 17

Weakness analysis: examples of predictions from the test data.

Imbalance inquestion type	Input query	When does it automatically switch to the fresh air mode?
	Output answer	I do not know.
	Correct answer	The control to prevent the inflow of outdoor air is a function that keeps the indoor airpleasant by automatically closing the window and changing the air conditioning system tothe ventilation cycle when passing through the tunnel or the non-clean area in conjunctionwith the navigation system.
Implicitquestion	Input query	I want to stop the trunk opening automatically
	Output answer	I do not know.
	Correct answer	To use the Auto Open Smart Trunk, close and lock all the car doors, and then after aboutfifteen seconds pass and wait behind the trunk with the smart key for about three seconds.

References

1. Rajpurkar, P.; Zhang, J.; Lopyrev, K.; Liang, P. SQuAD: 100,000+ Questions for Machine Comprehension of Text. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing; Austin, TX, USA, 1–5 November 2016; Su, J.; Duh, K.; Carreras, X. Association for Computational Linguistics: Stroudsburg, PA, USA, 2016; pp. 2383-2392. [DOI: https://dx.doi.org/10.18653/v1/D16-1264]

2. Yang, Y.; Yih, W.t.; Meek, C. WikiQA: A Challenge Dataset for Open-Domain Question Answering. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing; Lisbon, Portugal, 17–21 September 2015; pp. 2013-2018. [DOI: https://dx.doi.org/10.18653/v1/D15-1237]

3. Kwiatkowski, T.; Palomaki, J.; Redfield, O.; Collins, M.; Parikh, A.; Alberti, C.; Epstein, D.; Polosukhin, I.; Devlin, J.; Lee, K. et al. Natural Questions: A Benchmark for Question Answering Research. Trans. Assoc. Comput. Linguist.; 2019; 7, pp. 453-466. [DOI: https://dx.doi.org/10.1162/tacl_a_00276]

4. Chen, D.; Yih, W.t. Open-Domain Question Answering. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts; Online, 5–10 July 2020; Savary, A.; Zhang, Y. Association for Computational Linguistics: Stroudsburg, PA, USA, 2020; pp. 34-37. [DOI: https://dx.doi.org/10.18653/v1/2020.acl-tutorials.8]

5. Lee, J.; Seo, M.; Hajishirzi, H.; Kang, J. Contextualized Sparse Representations for Real-Time Open-Domain Question Answering. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics; Online, 5–10 July 2020; Jurafsky, D.; Chai, J.; Schluter, N.; Tetreault, J. Association for Computational Linguistics: Stroudsburg, PA, USA, 2020; pp. 912-919. [DOI: https://dx.doi.org/10.18653/v1/2020.acl-main.85]

6. Chung, H.; Song, Y.I.; Han, K.S.; Yoon, D.S.; Lee, J.Y.; Rim, H.C.; Kim, S.H. A Practical QA System in Restricted Domains. Proceedings of the Conference on Question Answering in Restricted Domains; Barcelona, Spain, 25–26 July 2004; pp. 39-45.

7. Campos, J.A.; Otegi, A.; Soroa, A.; Deriu, J.; Cieliebak, M.; Agirre, E. DoQA—Accessing Domain-Specific FAQs via Conversational QA. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics; Online, 5–10 July 2020; Jurafsky, D.; Chai, J.; Schluter, N.; Tetreault, J. Association for Computational Linguistics: Stroudsburg, PA, USA, 2020; pp. 7302-7314. [DOI: https://dx.doi.org/10.18653/v1/2020.acl-main.652]

8. Sheng, M.; Li, A.; Bu, Y.; Dong, J.; Zhang, Y.; Li, X.; Li, C.; Xing, C. DSQA: A Domain Specific QA System for Smart Health Based on Knowledge Graph. Proceedings of the Web Information Systems and Applications: 17th International Conference, WISA 2020; Guangzhou, China, 23–25 September 2020; Proceedings Springer: Berlin, Heidelberg, 2020; pp. 215-222. [DOI: https://dx.doi.org/10.1007/978-3-030-60029-7_20]

9. Tepper, N.; Goldbraich, E.; Zwerdling, N.; Kour, G.; Anaby Tavor, A.; Carmeli, B. Balancing via Generation for Multi-Class Text Classification Improvement. Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2020; Online, 16–20 November 2020; Cohn, T.; He, Y.; Liu, Y. Association for Computational Linguistics: Stroudsburg, PA, USA, 2020; pp. 1440-1452. [DOI: https://dx.doi.org/10.18653/v1/2020.findings-emnlp.130]

10. Xu, D.; Jansen, P.; Martin, J.; Xie, Z.; Yadav, V.; Tayyar Madabushi, H.; Tafjord, O.; Clark, P. Multi-class Hierarchical Question Classification for Multiple Choice Science Exams. Proceedings of the Twelfth Language Resources and Evaluation Conference; Marseille, France, 11–16 May 2020; Calzolari, N.; Béchet, F.; Blache, P.; Choukri, K.; Cieri, C.; Declerck, T.; Goggi, S.; Isahara, H.; Maegaard, B.; Mariani, J. et al. European Language Resources Association: Paris, France, 2020; pp. 5370-5382.

11. Sun, S.; Sedoc, J. An Analysis of BERT FAQ Retrieval Models for COVID-19 Infobot. Proceedings of the ACL 2020 Workshop NLP-COVID; Online, 5–10 July 2020.

12. Niu, Y.; Yang, L.; Dong, R.; Zhang, Y. Learning to Generalize for Cross-domain QA. Proceedings of the Findings of the Association for Computational Linguistics: ACL 2023; Toronto, ON, Canada, 9–14 July 2023; Rogers, A.; Boyd-Graber, J.; Okazaki, N. Association for Computational Linguistics: Stroudsburg, PA, USA, 2023; pp. 1298-1313. [DOI: https://dx.doi.org/10.18653/v1/2023.findings-acl.84]

13. Sadat, M.; Zhou, Z.; Lange, L.; Araki, J.; Gundroo, A.; Wang, B.; Menon, R.R.; Parvez, M.R.; Feng, Z. DelucionQA: Detecting Hallucinations in Domain-specific Question Answering. arXiv; 2023; arXiv: 2312.05200

14. Oewel, B.; Ammari, T.; Brewer, R.N. Voice Assistant Use in Long-Term Care. Proceedings of the 5th International Conference on Conversational User Interfaces; New York, NY, USA, 19–21 July 2023; CUI ’23 [DOI: https://dx.doi.org/10.1145/3571884.3597135]

15. Woszczyk, D.; Lee, A.; Demetriou, S. Open, Sesame! Introducing Access Control to Voice Services. Proceedings of the 1st Workshop on Security and Privacy for Mobile AI; New York, NY, USA, 25 June 2021; MAISP’21 pp. 7-12. [DOI: https://dx.doi.org/10.1145/3469261.3469405]

16. Miksik, O.; Munasinghe, I.; Asensio-Cubero, J.; Bethi, S.R.; Huang, S.T.; Zylfo, S.; Liu, X.; Nica, T.; Mitrocsak, A.; Mezza, S. et al. Building Proactive Voice Assistants: When and How (not) to Interact. arXiv; 2020; arXiv: 2005.01322

17. Grishman, R.; Sundheim, B. Message Understanding Conference- 6: A Brief History. Proceedings of the COLING 1996 Volume 1: The 16th International Conference on Computational Linguistics; Copenhagen, Denmark, 5–9 August 1996.

18. Nadeau, D.; Sekine, S. A survey of named entity recognition and classification. Lingvisticae Investig.; 2007; 30, pp. 3-26. [DOI: https://dx.doi.org/10.1075/li.30.1.03nad]

19. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, L.u.; Polosukhin, I. Attention is All you Need. Proceedings of the Advances in Neural Information Processing Systems; Guyon, I.; Luxburg, U.V.; Bengio, S.; Wallach, H.; Fergus, R.; Vishwanathan, S.; Garnett, R. Curran Associates, Inc.: New York, NY, USA, 2017; Volume 30, pp. 5998-6008.

20. Cho, K.; van Merrienboer, B.; Gülçehre, Ç.; Bahdanau, D.; Bougares, F.; Schwenk, H.; Bengio, Y. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014; Doha, Qatar, 25–29 October 2014; A meeting of SIGDAT, a Special Interest Group of the ACL Moschitti, A.; Pang, B.; Daelemans, W. ACL: Stroudsburg, PA, USA, 2014; pp. 1724-1734. [DOI: https://dx.doi.org/10.3115/v1/d14-1179]

21. Caruana, R. Multitask Learning: A Knowledge-Based Source of Inductive Bias. Proceedings of the Machine Learning, Proceedings of the Tenth International Conference, University of Massachusetts; Amherst, MA, USA, 27–29 June 1993; Utgoff, P.E. Morgan Kaufmann: Burlington, MA, USA, 1993; pp. 41-48. [DOI: https://dx.doi.org/10.1016/b978-1-55860-307-3.50012-5]

22. Bingel, J.; Søgaard, A. Identifying beneficial task relations for multi-task learning in deep neural networks. Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers; Valencia, Spain, 3–7 April 2017; Lapata, M.; Blunsom, P.; Koller, A. Association for Computational Linguistics: Stroudsburg, PA, USA, 2017; pp. 164-169.

23. Kingma, D.P.; Welling, M. Auto-Encoding Variational Bayes. Proceedings of the 2nd International Conference on Learning Representations, ICLR 2014; Banff, AB, Canada, 14–16 April 2014; Conference Track Proceedings 2014; Available online: https://arXiv.org/abs/1312.6114v10 (accessed on 5 November 2024).

24. Devlin, J.; Chang, M.W.; Lee, K.; Toutanova, K. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers); Minneapolis, MN, USA, 2–7 June 2019; Burstein, J.; Doran, C.; Solorio, T. Association for Computational Linguistics: Stroudsburg, PA, USA, 2019; pp. 4171-4186. [DOI: https://dx.doi.org/10.18653/v1/N19-1423]

25. Kodra, L. A Review on Neural Network Question Answering Systems. Int. J. Artif. Intell. Appl.; 2017; 8, pp. 59-74. [DOI: https://dx.doi.org/10.5121/ijaia.2017.8206]

26. Green, B.F.; Wolf, A.K.; Chomsky, C.; Laughery, K. Baseball: An Automatic Question-Answerer. Proceedings of the Papers Presented at the 9–11 May 1961, Western Joint IRE-AIEE-ACM Computer Conference; New York, NY, USA, 9–11 May 1961; IRE-AIEE-ACM ’61 (Western) pp. 219-224. [DOI: https://dx.doi.org/10.1145/1460690.1460714]

27. Woods, W.A. Progress in Natural Language Understanding: An Application to Lunar Geology. Proceedings of the 4–8 June 1973, National Computer Conference and Exposition; New York, NY, USA, 4–8 June 1973; AFIPS ’73 pp. 441-450. [DOI: https://dx.doi.org/10.1145/1499586.1499695]

28. Becker, T.; Blaylock, N.; Gerstenberger, C.V.; Kruijff-Korbayova, I.; Korthauer, A.; Pinkal, M.; Pitz, M.; Poller, P.; Schehl, J. Natural and Intuitive Multimodal Dialogue for In-Car Applications: The SAMMIE System. ECAI; 2006; 141, pp. 612-616.

29. Lemon, O.; Georgila, K.; Henderson, J.; Stuttle, M. An ISU Dialogue System Exhibiting Reinforcement Learning of Dialogue Policies: Generic Slot-Filling in the TALK In-car System. Proceedings of the Demonstrations; Trento, Italy, 5–6 April 2006.

30. Jin, B.; Jin, Z. Car FAQ Assistant Based on BiLSTM-Siamese Network. OALib; 2019; 06, pp. 1-7. [DOI: https://dx.doi.org/10.4236/oalib.1105817]

31. Lim, S.; Kim, M.; Lee, J. KorQuAD1.0: Korean QA Dataset for Machine Reading Comprehension. arXiv; 2019; arXiv: 1909.07005

32. Lee, K.; Yoon, K.; Park, S.; Hwang, S.w. Semi-supervised Training Data Generation for Multilingual Question Answering. Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018); Miyazaki, Japan, 7–12 May 2018.

33. Wang, Z.; Ittycheriah, A. Faq-based question answering via word alignment. arXiv; 2015; arXiv: 1507.02628

34. Sakata, W.; Shibata, T.; Tanaka, R.; Kurohashi, S. FAQ Retrieval Using Query-Question Similarity and BERT-Based Query-Answer Relevance. Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval; New York, NY, USA, 21–25 July 2019; SIGIR’19 pp. 1113-1116. [DOI: https://dx.doi.org/10.1145/3331184.3331326]

35. Mass, Y.; Carmeli, B.; Roitman, H.; Konopnicki, D. Unsupervised FAQ Retrieval with Question Generation and BERT. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics; Online, 5–10 July 2020; pp. 807-812. [DOI: https://dx.doi.org/10.18653/v1/2020.acl-main.74]

36. Tseng, W.T.; Hsu, Y.C.; Chen, B. Effective FAQ Retrieval and Question Matching Tasks with Unsupervised Knowledge Injection. Proceedings of the Text, Speech, and Dialogue; Ekštein, K.; Pártl, F.; Konopík, M. Springer: Cham, Switzerland, 2021; pp. 124-134.

37. Robertson, S.; Zaragoza, H. The Probabilistic Relevance Framework: BM25 and Beyond. Found. Trends Inf. Retr.; 2009; 3, pp. 333-389. [DOI: https://dx.doi.org/10.1561/1500000019]

38. Xu, Y.; Liu, X.; Shen, Y.; Liu, J.; Gao, J. Multi-task Learning with Sample Re-weighting for Machine Reading Comprehension. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers); Minneapolis, MN, USA, 2–7 June 2019; pp. 2644-2655. [DOI: https://dx.doi.org/10.18653/v1/N19-1271]

39. Shao, B.; Gong, Y.; Bao, J.; Ji, J.; Cao, G.; Lin, X.; Duan, N. Weakly Supervised Multi-task Learning for Semantic Parsing. Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19; Macao, China, 10–16 August 2019; Volume 7, pp. 3375-3381. [DOI: https://dx.doi.org/10.24963/ijcai.2019/468]

40. Worsham, J.; Kalita, J. Multi-task learning for natural language processing in the 2020s: Where are we going?. Pattern Recognit. Lett.; 2020; 136, pp. 120-126. [DOI: https://dx.doi.org/10.1016/j.patrec.2020.05.031]

41. Alqahtani, S.; Mishra, A.; Diab, M. A Multitask Learning Approach for Diacritic Restoration. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics; Online, 5–10 July 2020; pp. 8238-8247. [DOI: https://dx.doi.org/10.18653/v1/2020.acl-main.732]

42. Blei, D.M.; Kucukelbir, A.; McAuliffe, J.D. Variational Inference: A Review for Statisticians. J. Am. Stat. Assoc.; 2017; 112, pp. 859-877. [DOI: https://dx.doi.org/10.1080/01621459.2017.1285773]

43. Deng, Y.; Kim, Y.; Chiu, J.; Guo, D.; Rush, A.M. Latent Alignment and Variational Attention. Proceedings of the 32nd International Conference on Neural Information Processing Systems; Red Hook, NY, USA, 3–8 December 2018; NIPS’18 pp. 9735-9747.

44. Lippe, P.; Gavves, E. Categorical Normalizing Flows via Continuous Transformations. Proceedings of the 9th International Conference on Learning Representations, ICLR 2021; Virtual Event, Austria, 3–7 May 2021; Available online: OpenReview.net (accessed on 5 November 2024).

45. Rush, A.; Kim, Y.; Wiseman, S. Deep Latent Variable Models of Natural Language. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts; Melbourne, Australia, 4 November 2018; Mausam,; Wang, L. Association for Computational Linguistics: Stroudsburg, PA, USA, 2018.

46. Cheng, X.; Xu, W.; Wang, T.; Chu, W.; Huang, W.; Chen, K.; Hu, J. Variational Semi-Supervised Aspect-Term Sentiment Analysis via Transformer. Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL); Hong Kong, China, 3–4 November 2019; pp. 961-969. [DOI: https://dx.doi.org/10.18653/v1/K19-1090]

47. Miao, Y.; Yu, L.; Blunsom, P. Neural variational inference for text processing. Proceedings of the International Conference on Machine Learning, PMLR; New York, NY, USA, 20–22 June 2016; pp. 1727-1736.

48. Park, C.; Lee, C.; Song, H. VS³-NET: Neural variational inference model for machine-reading comprehension. ETRI J.; 2019; 41, pp. 771-781. [DOI: https://dx.doi.org/10.4218/etrij.2018-0467]

49. Lample, G.; Ballesteros, M.; Subramanian, S.; Kawakami, K.; Dyer, C. Neural Architectures for Named Entity Recognition. Proceedings of the NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies; San Diego, CA, USA, 12–17 June 2016; Knight, K.; Nenkova, A.; Rambow, O. The Association for Computational Linguistics: Stroudsburg, PA, USA, 2016; pp. 260-270. [DOI: https://dx.doi.org/10.18653/v1/n16-1030]

50. Reimers, N.; Gurevych, I. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP); Hong Kong, China, 7 November 2019; Inui, K.; Jiang, J.; Ng, V.; Wan, X. Association for Computational Linguistics: Stroudsburg, PA, USA, 2019; pp. 3982-3992. [DOI: https://dx.doi.org/10.18653/v1/D19-1410]

51. Bengio, Y.; Ducharme, R.; Vincent, P.; Jauvin, C. A neural probabilistic language model. J. Mach. Learn. Res.; 2003; 3, pp. 1137-1155.

52. Lee, C.; Kim, J.; Kim, J. Korean Dependency Parsing using Deep Learning. Proceedings of the 24th Annual Conference on Human and Cognitive Language Technology; Busan, Republic of Korea, 18–20 December 2014; pp. 87-91.

53. Pennington, J.; Socher, R.; Manning, C.D. GloVe: Global Vectors for Word Representation. Proceedings of the Empirical Methods in Natural Language Processing (EMNLP); Doha, Qatar, 25–29 October 2014; pp. 1532-1543.

54. Bojanowski, P.; Grave, E.; Joulin, A.; Mikolov, T. Enriching Word Vectors with Subword Information. Trans. Assoc. Comput. Linguist.; 2017; 5, pp. 135-146. [DOI: https://dx.doi.org/10.1162/tacl_a_00051]

55. Tjong Kim Sang, E.F.; De Meulder, F. Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition. Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003; Edmonton, AB, Canada, 31 May–1 June 2003; pp. 142-147.

56. Agarap, A.F. Deep learning using rectified linear units (relu). arXiv; 2018; arXiv: 1803.08375

57. Kullback, S.; Leibler, R.A. On information and sufficiency. Ann. Math. Stat.; 1951; 22, pp. 79-86. [DOI: https://dx.doi.org/10.1214/aoms/1177729694]

58. Kingma, D.P.; Ba, J. Adam: A method for stochastic optimization. arXiv; 2014; arXiv: 1412.6980

59. Hendrycks, D.; Gimpel, K. Gaussian error linear units (gelus). arXiv; 2016; arXiv: 1606.08415

60. Hochreiter, S.; Schmidhuber, J. Long short-term memory. Neural Comput.; 1997; 9, pp. 1735-1780. [DOI: https://dx.doi.org/10.1162/neco.1997.9.8.1735] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/9377276]

61. Lei, T.; Zhang, Y.; Wang, S.I.; Dai, H.; Artzi, Y. Simple Recurrent Units for Highly Parallelizable Recurrence. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing; Brussels, Belgium, 4 November 2018; Riloff, E.; Chiang, D.; Hockenmaier, J.; Tsujii, J. Association for Computational Linguistics: Stroudsburg, PA, USA, 2018; pp. 4470-4481. [DOI: https://dx.doi.org/10.18653/v1/D18-1477]