Content area
Abstract
As the carrier of information, language is the principal and most natural communication system used by humans. However, language is not directly understandable and executable for machines. Therefore, a key task in artificial intelligence is to create a map between natural language text and machine-interpretable meaning representations. Abstract Meaning Representation (AMR) is one such representation with a wide range of applications. AMR encodes the meaning of a natural language sentence as a rooted, directed, and labeled graph, where nodes represent concepts and edges represent relations.
In the first part of the thesis, we present an algorithm for automatically transforming natural language texts into AMR (i.e., AMR Parsing). This task is challenging for it encompasses a rich set of traditional tasks such as Named Entity Recognition (NER), Semantic Role Labeling (SRL), Word Sense Disambiguation (WSD), and Coreference Resolution. Our proposed method constructs a parse graph incrementally in a top-down fashion. The output graph spans the nodes by the distance to the root, following the intuition of first grasping the main ideas then digging into more details. The core semantic first principle emphasizes capturing the main ideas of a sentence, which is of great interest. Experiments show that our parser is especially good at obtaining the core semantics. The proposed method is then further enhanced by an iterative inference design. We explicitly characterize each spanning step as the efforts for finding which part of the input sequence to abstract, and where in the partially constructed output graph to construct. The iterative process helps achieve better answers to both questions, leading to greatly improved parsing accuracy.
In the second part of the thesis, we present an algorithm for mapping AMR to natural language text (i.e., AMR-to-text Generation). The algorithm uses a new neural network architecture, named Graph Transformer, for graph representation learning. Unlike traditional graph neural networks that restrict the information exchange between immediate neighborhoods, Graph Transformer uses explicit relation encoding and allows direct communication between two distant nodes. It provides a more efficient way for global graph structure modeling. Experiments show that our method largely outperforms previous state-of-the-art methods for AMR-to-text generation. We also show that the algorithm can be used to improve syntax-based machine translation.
In the third part of the thesis, we explore multilingual AMR parsing. To date, most resources of AMR are associated with English. The annotations of AMR for other languages can be very expensive. It is therefore challenging to develop a multilingual AMR parser. We tackle this problem from the perspective of knowledge distillation, where the aim is to learn and improve a multilingual AMR parser by using an existing English parser as its teacher. The complete training process consists of multiple pre-training and fine-tuning stages. As a result, we obtain one multilingual AMR parser whose performances surpass all previously published results on four different languages, including German, Spanish, Italian, and Chinese, by large margins.
This thesis also discusses the semantic transformations between different natural languages (i.e., machine translation). We propose a new framework for neural machine translation (NMT) that uses monolingual data in the target language as translation memory (TM) and performs learnable cross-lingual memory retrieval. First, the cross-lingual memory retriever allows abundant monolingual data to be TM. Second, the memory retriever and NMT model can be jointly optimized for the ultimate translation goal. Experiments show that the proposed model even outperforms strong TM-augmented NMT baselines using bilingual TM. Moreover, our framework also demonstrates effectiveness in low-resource and domain adaptation scenarios.