Natural Language Processing Technology Used in Artificial Intelligence Scene of Law for Human Behavior

Abstract

In order to study the application of natural language processing (NLP) technology in artificial intelligence (AI) scene of law, NLP technology is used to construct a legal AI retrieval system and further simulate the system. Then, by inputting the subject matter of the case into the system, the system’s accuracy, recall rate, and error rate and other related indicators are evaluated, to analyze the performance of the legal retrieval system. The results show that in the case analysis of a single theme, the accuracy rate of the case with the theme of “impeding police enforcement” is low, and the accuracy rate of the other theme cases is over 70%, and the highest accuracy rate even reaches 95%. In the case retrieval analysis of multitheme, the accuracy rate of case retrieval is improved, higher than 75%, and the zero-detection rate is significantly reduced with the increase in keywords. In the analysis of network case retrieval, the average correct rate of the overall case retrieval will be nearly 65%. Further tests on its reliability show that during the continuous week of the retrieval test, the system has no faults and passed the reliability test. Therefore, through this study, it is found that the application of NLP technology in the legal AI retrieval system has a reliable accuracy, which meets the expectation of this paper.

This work is licensed under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.

1. Introduction

With the rapid development of science and technology, the living standards for human beings have become increasingly intelligent; while liberating numerous labors, it makes people energetic to do things they like. Artificial intelligence (AI), as the most important technological advancement in human history, affects all areas of the living environment with an unstoppable attitude. Similarly, the judiciary is an indispensable field in daily lives, and its intelligent development is also in full swing. In the era of deepening system reform in the judicial field of China, to respond to the call for building the judicial civilization, the introduction of AI into the practice of the court is an urgent problem to be solved [1]. The developing process of the judiciary in China is distinguished from the mature technology, legislation, theory, and practice of foreign countries. In China, it is the application scenario that the AI technology is introduced into the court from top to bottom after the call of the central government and subsequent guidance from the internal organs of the judiciary [2, 3]. However, in practice, whether the artificially intelligent judicial system can meet the current needs of society and increase the efficiency of the court has become the focus of scientific researches.

Natural language processing (NLP) is a branch of AI from the macroperspective. It mainly refers to a process of processing natural language communication between humans and computers through computer application of natural language processing technology, which is an embodiment of the research on a human-computer interaction process [4]. The goal of NLP technology is to enable people to use natural language to interact with computer systems to achieve information and to ensure the efficient processing of information in the process of human-computer interaction. The application of NLP technology is mainly focused on the automatic differentiation of phrases, corpus construction, and grammar research, which makes it widely used in data mining, information retrieval, and machine translation [5, 6]. With the steady advancement of Chinese legal society, the open system of the judicial field has gradually improved. When the court’s referee is issued, the disclosure of the referee’s documents will provide people with sufficient data sources. The judgment documents are analyzed by NLP technology to realize the function of retrieving similar case referee results that match the user’s case, which can be critical for both the legal workers and the litigants in the case handling and litigation procedures [7]. In the process of handling the cases in the court, the litigants can obtain the judgment documents and results of similar cases by inputting their cases without the relevant legal expertise. It is very important for improving the legal awareness and quality of the citizens [8].

In summary, in the country of system reform, the construction of the judicial field is not perfect, and there are not many applications of NLP technology in the judicial field. Therefore, in this study, a legal artificial intelligence retrieval system is constructed by using NLP technology. Then, the system is simulated, and its retrieval performance and reliability are analyzed, which provides a reference for the application of AI technologies in the legal field.

2. Related Works

2.1. An Exploration of NLP Application

Currently, with the development of science and technology, the development of all walks of life is becoming increasingly intelligent. NLP technology, as a research manifestation of human-computer interaction, is an efficient guarantee for information processing capabilities. The extraction of diagnostic images and large amounts of digital content in radiology reports by Pons et al. is an extremely difficult challenge for the average patients, which used the NLP technology to convert the texts into structured language for patient to information, providing great help and significances [9]. Soysal et al. have developed a general clinical NLP system for the extraction of clinical text information and a clinical NLP toolkit, which not only provides the most advanced NLP components but also provides a user-friendly graphical user interface that helps users quickly build custom NLP pipelines for their applications, achieving good performance in entity naming recognition and concept coding [10]. The limitations of research by Parbhu et al. on administrative coded data made it difficult to accurately represent the problems with surgical indicators and results, which used the NLP technology to process images. In actual works, the efficiency is significantly improved compared with the manual survey, and the final performance of NLP tools in identifying indications of the process is excellent [11]. Furthermore, Northuis et al. have studied the NLP machine learning algorithms that used neuroimaging reports to classify stroke cases and subtypes of stroke. It is finally found that the technology could effectively extract the information reported by neuroimaging, and the performance of the constructed system is outstanding [12]. In the same year, Ebietomere and Ekuobase applied language processing technology to the legal system for the retrieval of related cases. Compared with other algorithms, the method had obvious advantages in the literature search in the legal field [13].

2.2. The Progression of the Legal System

In the process of development, a country cannot do without the constraints of legal rules on the behaviors of citizens. Otherwise, various illegal acts in society will be very embarrassing. Law as a stripe that can constrain the behaviors of people is also in the dynamic process of continuous improvement. Tileubergenov et al. discussed the implementation of natural science and technological achievements in the criminal justice system. In the context of growing crime predictions, the technological situations of the system are analyzed and determined, making the iconic system be determined and applied to practices [14]. McNamee and Ciara studied the learning barriers prevalent in the criminal justice system and emphasized the common learning disabilities among criminals and preexisting risk factors for them to become criminals to train them through relevant education. Of course, the most important is still the national guidelines [15]. Nissi et al. constructed a two-stage packet analysis model, and the related topics are examined to improve the judicial efficiency of Italy and to analyze the regional differences [16]. Marciano and Ramello introduced the microproblems in dealing with social issues and researched the latest progression, thereby deepening the understanding of the ordinary people on the judicial system [17].

Through the discussion of the development of NLP technology and law, if the NLP technology is applied to the judicial field, the improvement of the law and the understanding of relevant cases by nonprofessionals will be greatly beneficial. Therefore, in this study, the NLP technology is applied to the legal AI scenarios, which provides great help for the relevant people to understand the cases.

3. The Proposed Method

3.1. NLP Technology

NLP technology refers to a series of processing of human language by means of a computer to realize the use of natural language for communication between humans and computers [18–20]. In the process of implementation, NLP technology needs to be crossed by multiple disciplines such as linguistics, basic computer science, mathematics, statistics, and human brain science to achieve the ultimate interaction between humans and computers [21–24]. NLP is an indispensable technology in modern research on information science and technology. It includes the study of word sense disambiguation, entity naming recognition, syntactic analysis, automatic summarization, and information extraction. Its application range is also wide, such as sentiment analysis, email spam identification, automatic translation, information retrieval, and automatic question and answer. The application scenarios of NLP technology are shown in Figure 1.

[figure omitted; refer to PDF]

NLP technology usually includes the aspects of text classification, question and answer system, machine translation, and information retrieval. The text classification indicates the process of automatically determining the text category according to the text content under the specified classification system; however, it is difficult when it comes to sentence similarity. The question-and-answer system have emerged in the case that someone wants to obtain certain information accurately and quickly from massive information, and the calculation of sentence similarity is the key technology of the question-and-answer system. The process of machine translation is to transform a natural language into another natural language using the computer system, which is usually the translation of sentences or full text. The information retrieval is the information retrieval mode in language processing technology. During the query, the understanding of the meaning of the user query statement is the key to ensure the accuracy of the query information.

3.2. Information Retrieval Technology

Information retrieval is a massive field in computer science. Its purpose is to maximize the ratio of user satisfaction to its efforts. It mainly includes three kinds, i.e., the Boolean model, the vector space model, and the probability retrieval model. The Boolean model is the simplest. In the process of information retrieval, the selection of feature values is extremely important. The weight calculation of feature words is based on the text representation model. Usually, the text representation model includes the vector space model and the Boolean model. The calculation methods of feature weight mainly include a simple word frequency method, Boolean function method, root function method, and Tf-idf function method. The simple word frequency method considers that the number of occurrences of the featured item in the document, i.e., the frequency of the featured item, can express the importance degree of the feature word in the document, i.e., the weight value. The equation is as follows: $$\begin{array}{}\text{(1)}& w_{ij}=t_{\mathit{\text{if}}},\end{array}$$

where $w_{ij}$ indicates the weight of the featured item $t_i$ on the document $d_j$ and $t_{\mathit{\text{if}}}$ indicates the frequency of occurrence of the featured item. The Boolean function method is simpler than the simple word frequency method. In the function, only the feature items in the document are considered, and the number of occurrences is negligible. Therefore, the weight value of the featured items can be 0 and 1 in the algorithm. The equation is as follows: $$\begin{array}{}\text{(2)}& w_{ij}=\begin{array}{cc}0,\hfill & {tf}_{\mathit{\text{if}}}>0,\hfill \\ 1,\hfill & {tf}_{\mathit{\text{if}}}\le 0.\hfill \end{array}\end{array}$$

In addition, the root function method is an improvement on the simple word frequency method. The weight of the featured item is expressed by the square root of the number of occurrences of the featured item. The equation is as follows: $$\begin{array}{}\text{(3)}& w_{ij}=\sqrt{{tf}_{\mathit{\text{if}}}}.\end{array}$$

Because of the above-mentioned several feature weight calculation methods, the discrimination degree of the document is low, and thus, the method of tf-idf appears. In the tf-idf function method, the ratio of the word frequency of the featured item in the document to the total number of words of the document tf, the number of all documents, and the number of documents idf containing the featured item are fully considered, wherein idf is used as the featured item. The distinguishing factor is inversely proportional to the weight value of the featured item. The equation is as follows: $$\begin{array}{}\text{(4)}& w_{ik}={tf}_{ij}\ast {idf}_{ik}={tf}_{ik}\ast \log \frac{N}{n_k+0.001}{df}_i,\end{array}$$

where $t{f}_{ij}$ is the number of occurrences of feature items $t_i$ in the document $d_j$, $d{f}_i$ is the number of feature items $t_i$ included in all documents, and $N$ represents the total amount of all documents.

3.3. The Combination of AI and Law

In the process of the gradual development of the law, the AI system has been gradually applied in judicial practice such as courts. However, it is still necessary to explore the integration of AI and law, as well as the implementation of the basic technology of the court office system. AI is a cross-discipline technology, which includes machine learning, NLP, image recognition, and human-computer interaction. It makes machines or systems have similar or even beyond human independent thinking ability and ultimately achieve more efficient and refined decision-making abilities than humans. The integration of deep learning makes it a breakthrough in the field of computer vision. The development of AI includes three major spans, i.e., the weak AI, the strong AI, and the super AI. At present, experts in the field of AI generally believe that it is still in the stage of weak AI.

With the addition of supercomputing, big data, and data training, AI has an important impact on the law, legal industry, and judicial activities. Also, the characteristics of the law are in line with the advantages of AI. Some scholars have summarized the characteristics of the law into the following points: the law has clear rules, strict procedures, and clear standards of proof; the law has vast legal provisions, cases, theories, and principles; the rigorous logic of law can be proven by different methods of reasoning and classification; the law has multiple tasks and functions, such as guidance, evaluation, prediction, education, and coercion; the law demonstrates the facts of the case through the process of confrontations; the law has an open structure; the law has a high degree of self-reflection mechanism that is characterized by innovation and advancing with the times. The technical characteristics of the law not only promote the combination of AI and law but also provide the potential for development. The characteristics of legal argumentation create an extremely suitable application condition for the introduction of AI into the judicial field, as shown in Figure 2.

[figure omitted; refer to PDF]

The basic operation process of the system is as follows. First, when the user inputs the query, the system automatically analyzes the query, including word segmentation, entity identification, keyword information extraction, and related word meaning expansion processing. After processing, the words to be queried are submitted to the query module, and the query is performed in the system’s case index library. Finally, the query result is returned according to the filtered keyword ranking mechanism.

The main core of the system in the running process includes four functional modules, which are word segmentation module, keyword extraction module, keyword expansion module, and retrieval module. The word segmentation module mainly performs word segmentation on the case, calls the relevant word segmentation tool to segment the input case, and introduces a custom legal dictionary into the process to improve the result of the word segmentation. The keyword extraction module extracts the vocabulary related to the subject of the case from the case, filters the irrelevant vocabulary, and extracts the keywords that can be queried for the main information. The keyword expansion module mainly expands the keywords extracted for the case inquiry and adds related words. After inputting the extension, the query is enriched, and the semantics of the case are enriched. The retrieval module is mainly used for searching the legal terms, and the expanded keywords are searched and matched with the index library built by the legal provisions. Then, the query is obtained, and the cases are rated by matching relevance and legal terms.

3.5. Simulation and Performance Evaluation

In the development process of this system, the development language is JAVA for the keyword extraction and extension phase, and the experimental environment is Windows 7. The retrieval module is implemented on the MyEclipse platform. The specific system development environment is shown in Table 1. The search development kit LUCENE 5.0 full-text search development library is the 2015 version of LUCENE announced by Apache.

Table 1

The development environment of the system.

Development platform	Windows7+JDK8
Development language	JAVA+JAVA WEB
Development tool	Eclipse4.9+Myeclipse2017+Tomcat7
Development kit of retrieval	LIJCENE5.0

In this paper, the system retrieval performance is retrieved by using the Text Retrieval Conference (TREC). The Text Retrieval Conference uses objective indicators such as precision, recall, and $F$-measure to measure the effects of the constructed retrieval system. The equations are as follows: $$\begin{array}{}\text{(5)}& \text{Accuracy}\text{rate}P=\frac{\text{number}\text{of}\text{correctly}\text{retrieved}\text{law}\text{provisions}}{\text{number}\text{of}\text{retrieved}\text{law}\text{provisions}},\text{(6)}& \text{Recall}\text{rate}R=\frac{\text{number}\text{of}\text{correctly}\text{retrieved}\text{law}\text{provisions}}{\text{number}\text{of}\text{law}\text{provisions}\text{related}\text{to}\text{the}\text{case}},\text{(7)}& \text{Metric}F=\frac{2\ast P\ast R}{P+R}.\end{array}$$

The performance test is still in the stage of simulation. The legal search power supply of this system is compared with the general information retrieval system. The number of legal terms related to the index library is still limited. Therefore, in the system evaluation, based on the correct rate, another error rate and zero-detection rate are added. The equations are as follows: $$\begin{array}{}\text{(8)}& \text{Accuracy}\text{rate}P=\frac{\text{number}\text{of}\text{correctly}\text{retrieved}\text{cases}}{\text{total}\text{case}\text{number}},\text{(9)}& \text{Error}\text{Rate}E=\frac{\text{numbers}\text{of}\text{incorrectly}\text{retrieved}\text{cases}}{\text{total}\text{case}\text{number}},\text{(10)}& \text{Zero}\text{retrieval}\text{rate}Z=\frac{\text{number}\text{of}\text{nonretrieved}\text{cases}}{\text{total}\text{case}\text{number}}.\end{array}$$

In this paper, the test cases are randomly selected from the case index database as a test set to conduct a search test on the system. Finally, the three indexes of the correct rate, error rate, and zero retrieval rate are used to evaluate the legal retrieval system designed in this study. In addition, the real individual subject cases, real multiple subject cases, network scenarios, and system reliability are tested.

4. Results and Discussion

4.1. Real Single-Subject Case Retrieval Performance Analysis

A single subject means that there is a corresponding legal clause for each case that is binding on it. In this study, seven cases are selected, including assault, theft, and interference of law enforcement. The statistics are shown in Table 2 and Figure 4. It can be seen from Table 2 that the search system of this study can understand the legal terms and the total number of cases corresponding to each topic. Further statistics on its correct rate, error rate, and accuracy rate are shown in Figure 4. It is found that the correct detection rate of other cases is lower than 70%, except for the case of “interference of law enforcement.” The reason is that such descriptions involve many aspects of illegal behaviors, resulting in a lower rate of correct detection. Therefore, the retrieval system of this study can achieve more than 70% of the retrieval ability of a single subject case.

Table 2

Search results of real single-subject cases.

Number	Cases	Legal provision	Total number of cases
a	Assault	Article 43	453
b	Theft	Article 49	728
c	Interference of law enforcement	Article 50	47
d	Illegal business operations	Article 54	69
e	Hotel administration	Article 56	211
f	Prostitution	Article 66	458
g	Gambling	Article 70	1719
h	Drug taking or making others take drugs	Article 72	637

[figures omitted; refer to PDF]

4.2. Real Multiple-Subject Case Retrieval Performance Analysis

Multiple subjects indicate binding on each case with multiple corresponding legal provisions. In this study, three cases are selected, including assault and theft, theft and gambling, and drug taking and assault. The statistics are shown in Table 3 and Figure 5. Through the above experimental results, it can be observed that the multitopic case has a significant increase in the correct detection rate in the legal retrieval system of this study, both of which are higher than 75%. The reason may be that more content is detected after including multiple topics, and thus, the zero retrieval rate is lower and the correct detection rate is higher. Therefore, the retrieval system of this study is more than 75% in the retrieval ability of multiple-subject cases.

Table 3

Search results of real multiple-subject cases.

Number	Cases	Total number of cases	Accuracy rate	Error rate	Zero retrieval rate
1	Assault and theft	59	0.761	0.239	0
2	Theft and gambling	72	0.827	0.112	0.062
3	Drug taking and assault	117	0.819	0.109	0.071

[figure omitted; refer to PDF]

4.3. Test and Analysis of Online Cases

The characteristics of the online cases are more complicated, and the record may be the comments on the design terms through oral expression and manual methods. During the test process, a total of 174 cases are obtained from the network. The search results obtained by the test are shown in Figure 6. The correct rate is about 65%, the error rate is as high as 30%, and the zero retrieval rate is 4.7%. Therefore, the retrieval system of this research can reach more than 65% of the online case retrieval ability.

[figure omitted; refer to PDF]

4.4. System Reliability Analysis

In the test of system reliability, the effect is observed by performing a continuous search for one week. Finally, it is found that during the test period, the system can run stably. Also, no reliability failure has occurred, and the system has good robustness. Therefore, the retrieval system of this research has passed the reliability test.

5. Conclusions

During the developmental process of law, science and technology are also developing. In particular, when reviewing the cases, it is very important to make nonprofessionals understand and facilitate these cases. In order to study the application of NLP technology in legal AI scenarios, in this system, a legal retrieval system is constructed by using NLP technology. The core of system operation includes a word segmentation module, keyword extraction module, keyword expansion module, and retrieval module. When the user inputs the word or sentence to be queried, the keyword extraction and keyword amplification are performed so that the defect that the result is different due to the user input agreeing to different words is compensated. Finally, the performance of the retrieval system is developed, simulated, and tested. In the real case search, it is found that the case search accuracy of multiple-subject cases and single-subject cases has increased significantly, and the zero retrieval rate is avoided to a greater extent. Further analysis of the network case has revealed that the correct rate is reduced. It may be since the case is often referred to as oral and manual expressions. At the same time, the reliability of the system is analyzed. It is found that no retrieval failure occurs in the continuous retrieval process, and the system has good reliability.

In summary, through the research of this study, it is found that the application of NLP technology in the legal AI retrieval system has a stable accuracy rate and reliability, which meets the expectations of this study and further develops the natural language technology in the legal AI scenarios, providing an experimental basis. There are also certain deficiencies in the research process. For example, the experiment is still in the stage of development and simulation. However, in actual lives, people will encounter more emergencies. Thus, it should be further applied to the actual legal scenarios in the follow-up researches, which will provide a more reliable basis for the intellectualization of the legal AI scenarios.

References

[1] N. Aletras, D. Tsarapatsanis, D. Preoţiuc-Pietro, V. Lampos, "Predicting judicial decisions of the European Court of Human Rights: a natural language processing perspective," PeerJ Computer Science, vol. 2, article e93,DOI: 10.7717/peerj-cs.93, 2016.

[2] B. Alarie, A. Niblett, A. H. Yoon, "How artificial intelligence will affect the practice of law," University of Toronto Law Journal, vol. 68, pp. 106-124, DOI: 10.3138/utlj.2017-0052, 2018.

[3] T. Kerikmäe, T. Hoffmann, A. Chochia, "Legal technology for law firms: determining roadmaps for innovation," Croatian International Relations Review, vol. 24 no. 81, pp. 91-112, DOI: 10.2478/cirr-2018-0005, 2018.

[4] S. Pandey, S. K. Pandey, L. Miller, "Measuring innovativeness of public organizations: using natural language processing techniques in computer-aided textual analysis," International Public Management Journal, vol. 20 no. 1, pp. 78-107, DOI: 10.1080/10967494.2016.1143424, 2017.

[5] M. Conway, M. Hu, W. W. Chapman, "Recent advances in using natural language processing to address public health research questions using social media and consumer generated data," Yearbook of Medical Informatics, vol. 28 no. 1, pp. 208-217, DOI: 10.1055/s-0039-1677918, 2019.

[6] L. B. Ramsey, S. L. Aldrich, E. Poweleit, C. A. Prows, L. J. Martin, J. R. Strawn, "Racial differences in escitalopram/citalopram-related weight gain in children and adolescents: a natural language processing-based electronic medical record study," Journal of Child and Adolescent Psychopharmacology, vol. 29 no. 2, pp. 162-163, DOI: 10.1089/cap.2018.0110, 2019.

[7] M. Parackal, D. Mather, D. Holdsworth, "Value-based prediction of election results using natural language processing: a case of the New Zealand general election," International Journal of Market Research, vol. 60 no. 2, pp. 156-168, DOI: 10.1177/1470785318762234, 2018.

[8] C. Lindvall, E. J. Lilley, S. N. Zupanc, I. Chien, B. V. Udelsman, A. Walling, J. A. Tulsky, "Natural language processing to assess end-of-life quality indicators in cancer patients receiving palliative surgery," Journal of Palliative Medicine, vol. 22 no. 2, pp. 183-187, DOI: 10.1089/jpm.2018.0326, 2019.

[9] E. Pons, L. M. Braun, M. M. Hunink, J. A. Kors, "Natural language processing in radiology: a systematic review," Radiology, vol. 279 no. 2, pp. 329-343, DOI: 10.1148/radiol.16142770, 2016.

[10] E. Soysal, J. Wang, M. Jiang, Y. Wu, S. Pakhomov, H. Liu, H. Xu, "CLAMP–a toolkit for efficiently building customized clinical natural language processing pipelines," Journal of the American Medical Informatics Association, vol. 25 no. 3, pp. 331-336, DOI: 10.1093/jamia/ocx132, 2018.

[11] S. K. Parbhu, G. Cole, M. Sossenheimer, A. Gawron, "Natural language processing of EGD procedure notes more accurately identifies GERD indications than ICD administrative codes," American Journal of Gastroenterology, vol. 113, pp. S207-S208, DOI: 10.14309/00000434-201810001-00362, 2018.

[12] C. Northuis, M. Michalowski, K. Lakshminarayan, "Abstract WP382: using natural language processing algorithms to identify stroke cases and stroke subtypes from neuroimaging reports," Stroke, vol. 50,DOI: 10.1161/str.50.suppl_1.WP382, 2019.

[13] E. P. Ebietomere, G. O. Ekuobase, "A semantic retrieval system for case law," Applied Computer Systems, vol. 24 no. 1, pp. 38-48, DOI: 10.2478/acss-2019-0006, 2019.

[14] E. M. Tileubergenov, S. I. Pelevin, A. M. Vasiliev, G. Khalidullin, "Theoretical problems of implementing achievements of natural and technical sciences into the criminal judicial system," Journal of Advanced Research in Law and Economics, vol. 7 no. 1, pp. 132-139, 2016.

[15] G. McNamee, T. Ciara, "Supports for offenders with learning disabilities in the Irish judicial system: a critical review," Irish Journal of Applied Social Studies, vol. 17 no. 1, 2017.

[16] E. Nissi, M. Giacalone, C. Cusatelli, "The efficiency of the Italian judicial system: a two stage data envelopment analysis approach," Social Indicators Research, vol. 146, pp. 395-407, DOI: 10.1007/s11205-018-1892-5, 2018.

[17] A. Marciano, G. B. Ramello, "Introduction to the symposium on the empirics of judicial institutions," Journal of Institutional Economics, vol. 15 no. 1, pp. 73-80, DOI: 10.1017/S1744137418000322, 2019.

[18] E. Eckhaus, N. Davidovitch, "Improving academic conferences–criticism and suggestions utilizing natural language processing," European Journal of Educational Research, vol. 7 no. 3, pp. 445-450, DOI: 10.12973/eu-jer.7.3.445, 2018.

[19] B. Udelsman, I. Chien, K. Ouchi, K. Brizzi, J. A. Tulsky, C. Lindvall, "Needle in a haystack: natural language processing to identify serious illness," Journal of Palliative Medicine, vol. 22 no. 2, pp. 179-182, DOI: 10.1089/jpm.2018.0294, 2019.

[20] N. Davidovitch, E. Eckhaus, "The influence of birth country on selection of conference destination - employing natural language processing," Higher Education Studies, vol. 8 no. 2, pp. 92-96, DOI: 10.5539/hes.v8n2p92, 2018.

[21] C. Nath, M. S. Albaghdadi, S. R. Jonnalagadda, "A natural language processing tool for large-scale data extraction from echocardiography reports," PLoS One, vol. 11 no. 4, article e0153749,DOI: 10.1371/journal.pone.0153749, 2016.

[22] R. R. Romeo, J. A. Leonard, S. T. Robinson, M. R. West, A. P. Mackey, M. L. Rowe, J. D. Gabrieli, "Beyond the 30-million-word gap: children’s conversational exposure is associated with language-related brain function," Psychological Science, vol. 29 no. 5, pp. 700-710, DOI: 10.1177/0956797617742725, 2018.

[23] S. Sheikhalishahi, R. Miotto, J. T. Dudley, A. Lavelli, F. Rinaldi, V. Osmani, "Natural language processing of clinical notes on chronic diseases: systematic review," JMIR Medical Informatics, vol. 7 no. 2, article e12239,DOI: 10.2196/12239, 2019.

[24] J. Mariani, G. Francopoulo, P. Paroubek, "Reuse and plagiarism in speech and natural language processing publications," International Journal on Digital Libraries, vol. 19 no. 2-3, pp. 113-126, DOI: 10.1007/s00799-017-0211-0, 2018.