The Innovative Use of Intelligent Chatbot for Sustainable Health Education Admission Process: Learnt Lessons and Good Practices

Abstract

This article presents the methodology of creation of an innovative used by intelligent chatbots which support the admission process in universities. The lifecycle of the ontology, unlike the classical lifecycles, has six stages: conceptualization, formalization, development, testing, production and maintenance. This leads to sustainability of the chatbot, called Ana, which has been implemented at the “Iuliu Hatieganu” University of Medicine and Pharmacy from Cluj-Napoca during the admission session throughout July–September 2022, for international candidates. The sustainability of the chatbot comes from the continuous maintenance and updates of the ontology, based on candidates’ interraction with the system and updates of the admission procedures. Over time, the chatbot is able to answer the questions according to the present situation of the admission and the real needs of the candidates. Ana had a huge impact, succeeding to resolve a number of 5173 applicants requests, and only 809 messages was transferred to the human operators, statistics which show a high cost-benefit improvement in terms of reducing the travel expenses for the candidates and also for the university. The article also summarizes the good practices in developing and use of such an intelligent chatbot.

1. Introduction

In the yearly university admission period, universities are overwhelmed by the huge numbers of candidates visiting the admission departments of each university, with questions about the university, admission requirements and related documents [1]. This poses a challenge for most admission teams, as the staff involved in the admission usually partake in one to one conversations with candidates. This is not a cost effective method of communication in such situations, and in some cases physically visiting the admission centers, waiting in queues in order to get answers from the admission team might be seen too costly and time consuming for candidates in order to received the required answers and consider applying for a specific university.

1.1. Background

Our study will focus on international candidates and the digitization of the traditional international university admission process, which provides a specific set of challenges to candidates, such as sending the application form and admission documents through national and international postal services to the admission center, which can be costly, stressful, and in some cases bear the risk of losing the admission documents through postal services, or the risk that candidates cannot timely sent the necessary or additional documents required by the admission staff in the imposed deadline, due to slow postal operators, strikes, customs, etc.

Moreover communication efforts in the pre-admission and admission period in universities are usually conducted traditionally, which requires the availability of a large number of university administrative staff in order to communicate on an individual basis, with a large number of candidates in a short period of time [2]. Communication methods for university admissions has evolved from personal physical discussion between candidates and the admission academic staff, to emails and call centers, as competition between universities pushes them to accept multiple forms of communication, in order to attract candidates.

Simultaneously with the increased competition between universities in order to attract candidates, seen with the acceptance of multiple forms of communication, the recent COVID-19 pandemic has accelerated the worldwide transition from the traditional physical admission to the modern web based online admission systems [3], such as Socrates, developed by Holisun (http://www.holisun.com/en/, accessed on 15 January 2023), in order to respond to the social distance rules imposed by the pandemic. As in the admission period of 2019–2020 most of the university admissions were online, an overwhelming number of candidates started using online forms of communication with the admission staff, those online forms of communication provides a large database with the questions addressed to the admission staff, most times those questions being repetitive ones.

Even if most universities are using Q&A public documents addressing the most frequently asked questions, the majority of candidates still prefer using personal forms of communication, such as writing emails to the administrative staff in order to get answers to their inquiries, as the majority of candidates seem to not be interested in reading Q&A documents. This creates a problem for universities across the world, as answering those repetitive questions by the academic staff requires significant labor hours and the availability of a large admission staff, but now is possible to digitize the most frequent Q&A, associated with the admission process, by defining those questions and answers as a natural language processing problem [4], that can be used as a system of dialogues between candidates and a Q&A database through an automated chat box, used primarily to provide an interface of communication between candidates and the Q&A database, this technology is most widely and commonly known as “chatbot” [5].

1.2. Aim of the Research

This paper looks into the way chatbots are starting to be used in a multicultural and bilingual environment by Romanian universities, the first one being the University of Medicine and Pharmacy “Iuliu Hatieganu” from Cluj-Napoca (UMF Cluj). This leads to cost reduction and high sustainability in terms of resources allocated to the whole admission process (human resources and time).

The aim of the paper is to explore how Ana, as the chatbot is called, can innovate the university admission support process by answering questions from international candidates that otherwise would ask those questions at the admission office or by other means of one to one communication methods that entails large costs in terms of labor hours from the admission staff. The chatbot’s name Ana was chosen by the UMF Cluj Admission Staff following the well established practice of giving popular and internationally recognized human names for chatbots in order to make their interaction with humans feel more natural, as a human to human conversation does.

1.3. Research Questions

The research questions the article answers are:

• RQ1. What is an appropriate structure of the ontology of the chatbot?;

• RQ2. How and to what extent a chatbot can contribute to the candidate support during the admission period?;

• RQ3. What are good practices in desinging such a chatbot?;

• RQ4. What is the innovation and added value of such a chatbot to the admission process?;

• RQ5. What is the acceptance of such a chatbot and what is its (perceived) impact?;

• RQ6. What actions are needed for making the chatbot sustainable over more admission sessions?

1.4. Novel Contribution

As an element of novelty, the Ana chatbot was designed and developed to be used in English and French conversations with international candidates applying for the University of Medicine and Pharmacy “Iuliu Hatieganu” from Cluj-Napoca, one of the largest medical universities in Romania. Ana, developed initially for UMF Cluj, is the first chatbot to be used in Romanian universities for admission purposes and the chosen chatbot name does not relate to other chatbots on the market that might use the same popular human name of “Ana”.

As one of the most documented challenges of any institution interested in maximizing chatbots acceptance by the end user is developing social intelligence features [6] through simulating humanlike communication characteristics in order to make the interaction feel more natural [7], the challenge and novelty in our study, if compared with the works of [8] is not only developing bilingual chatbots and adapting their database to the youth social language [9] but to see if chatbots can be effectively used is a highly multicultural environment, which implies overcoming communication issues associated with 3332 international candidates from UMF Cluj who applied from 76 different countries and 5 continents (Europe, Africa, Asia, North America, South America).

In this sense our study will also provide an insight into how a dedicated university chatbot performs not only in a highly multicultural environment, but also on how it performs on candidates with a human science profile.

1.5. Organization of the Article

This paper is organized as follows: Section 1 Introduction introduces the main aspects related to developing and using chatbots in higher education, with focus on the international university admission process. Section 2 Related Work presents other chatbots and ontologies used for admissions at universities. Section 3.1 Context provides the context in which the Ana chatbot has been developed, namely for the admission process of “Iuliu Hatieganu” University of Medicine and Pharmacy, Cluj Napoca, Romania. Further, Section 3 Methodology presents the methodology for developing the ontology of Ana. Section 3.3 Software Architecture describes the Software Architecture of the Chatbot. Section 3.4 Improved lifecycle of the chatbot refers to all the lifecycle steps improved within the chatbot. The results of using Ana are reported in Section 4 Results and challenges related to the chatbot lifecycle. Then we draw some good practices and recommendations in Section 5 Recommendation, and finally the conclusions and possible future developments are stated in Section 6 Conclusions.

2. Related Work

The general idea of using artificial intelligence (A.I.) techniques in order to simulate human conversation can be traced back to 1950’s, when the famous Alan Turing, best known for its Cryptanalyst merits during World War II [10], developed the famous Turing Test [11], in which we can determine if the entity with whom we are having a conversation is a real personal or a computer program.

2.1. First Intelligent Chatbots

The first chatbot called ELIZA [12] was developed in 1960’s, and used keyword matching algorithms in order to match user input, but at this time the technology was in its early stages, without any significant practical uses in higher education.

With the arrival of the graphical user interfaces and the development of natural language algorithms, a new generation of more complex chatbots were developed, with applications in the public, private and higher education areas, such as MegaHall [13], CONVERSE [14], ELIZABETH and HEXBOT [15], ALICE [16] and Siri [17], developed by the Apple Corporation, that alongside Alexa from Amazon, Cortana from Microsoft and Google Assistant from Google are using advanced software, data mining and machine learning to create the most popular and advanced speech conversation chatbots in the world. The chatbot market is rapidly developing and is predicted to reach 1,23 billion dollars by 2025, according to [18].

In order to reduce labor costs, chatbots using natural language processing and machine learning algorithms are developed in order to assure a chat like interface for common questions addressed by candidates and are designed to provide answers similarly to how humans do, but automatically and on a much larger scale, without any human intervention.

2.2. Chatbots in Higher Education

Chatbots in higher education vary in complexity, communication between a candidate and the chatbot can be in text, audio, pictures and video formats. Chatbots can be installed on most online communication channels, such as the most well known messaging and social media platforms, but can also function on websites and various software applications.

Chatbots are becoming increasingly popular not only in the online admission process, where according to MainStay, formerly known as AdmitHub [19] can generally answer more than 65% of candidate and student questions instantly, 24/7. Advanced chatbots can also be used by universities in order to reduce summer melt [20], increasing access to financial aid and improving student engagement.

Chatbots are developed by large and smaller universities alike, such as Georgia State University (USA) that has one of the most advanced university chatbots in the world [20], and other universities which are using less advanced chatbots such as Staffordshire University (UK) [21], Telkom University (Indonesia) [5] and even Jamhuriya University (Somalia) [22], from one of the poorest countries from the world.

2.3. Conversational Chatbots

Yet, back in 2011 Al-Zubaide and Issa  [23] developed OntBot, a chatbot which uses appropriate mapping technique to transform ontologies and knowledge into relational database and then use that knowledge to drive its chats. Vegesna et al. [24] propose a chatbot that handles queries from users for an E-commerce website. The Ontology template is developed using Protégé which stores the knowledge acquired from the website APIs.

Nazir et al. [23] presented a novel incremental approach for building a chatbot for fashion brands based on semantic web, by organizing a dataset of 5000 question and answers of top10 brands in the fashion domain, which covers the information about new arrivals, sales, packages, discounts, exchange/return policies, etc. Ranoliya et al. [25] describes the design of a chatbot, which provides an efficient and accurate answer for any query based on the dataset of FAQs using Artificial Intelligence Markup Language (AIML) and Latent Semantic Analysis (LSA).

Agus Santoso et al. [26] propose the development of Chatbot, called Dinus Intelligent Assistance (DINA), which acts as a conversation agent that can play a role of as student candidate service. The source of the knowledge base is taken from Universitas Dian Nuswantoro (UDINUS) guest book. It contains of questions and answers about UDINUS admission services. Testing of this system is done by entering questions. From 166 intents, the author tested it using ten random sample questions. Among them, it got eight tested questions answered correctly.

Colace et al. [27] presents the realization of a chatbot prototype in the educational domain, which was developed a system to provide support to university students on some courses. The initial purpose has focused on the design of the specific architecture, model to manage communication and furnish the right answers to the student. Dimitriadis shows in  [28] that Pounce Chatbot used within Georgia University increased the enrolment with 3.9% and reduced the summer melt of the students with 21.4%.

3. Research and Development Methodology

3.1. Context

“Iuliu Hatieganu” University of Medicine and Pharmacy (UMF) is the continuation of the Romanian Faculty of Medicine founded in 1919, as part of the University of “Upper Dacia”. The “Iuliu Hațieganu” University has over 6000 students, 2400 resident and over 1100 teachers and researchers divided between its 3 faculties: Medicine, Dental Medicine and Pharmacy.

As 600 of the total 1266 admission vacancies, or 47% are allocated to international candidates, there is a business continuity need to assure that the number of international candidates does not drop, as 76% of the total annual tuition fees collected by the university for students in the first year of study, meaning 3,600,000 euros out of the total of 4,697,065 euros collected were from international students. The need to assure better online and interpersonal forms of communication with international candidates first emerged during the 2020 international admission, when due to international travel restrictions and social distancing heath regulations, UMF Cluj implemented Socrates (https://holisun.com/produse/socrate-admitere-universitati, accessed on 15 January 2023), an online admission platform, in order to assure the continuation of the international admission session in the COVID-19 pandemic period of 2020.

The Ana chatbot implemented in the 2022 international admission of UMF Cluj is the first chatbot implemented by a Romanian university for admission purposes and is based on the AIDA.AI natural language algorithms and machine learning methods developed by Holisun, with the purpose of creating and training the Ana chatbot using data consisting of the most frequent Q&A’s pairs, with questions and answers that have a tree-type complexities, with multiple branches of answers, depending on the context of the discussion carried out by the candidates with the Ana chatbot, developed to be used in English and French chat conversations.

3.2. The Admission Process

The whole admission process has four steps, as depicted in Figure 1. The first step of the whole process is the one in which prospects (visitors of the admission platform) can apply for educational offers of the universities. In the second phase, the applications are evaluated based on certain assessment criteria. After the evaluation phase, the candidates are ranked. The admitted ones have to register for becoming students.

Ana is meant to be used only for the university application stage (see Figure 1), as that is the step where candidates have the most significant number of questions and need the most support and tutoring.

3.3. Software Architecture

The software architecture of Ana chatbot, detailed in Figure 2, comprises of several modules:

• Domain Abstract Ontology: Is the schema of the general ontology with domain independent concepts, able to be instantiated for each particular authority. This ontology refers to public information, such as deadlines, student opportunities and educational offer.

• Specific Ontology: represents concepts which belong or are particular to a specific candidate, be it country origin, last courses taken. The ontology is structured in three sub-ontologies, namely:

  • -. Public information: refers to information for public use, such as number of available seats, specialisation etc.

  • -. Adaptive information is still general information, but provided to the candidates based on their profile, e.g., different countries issue different documents related to graduation (Abitur, Bacalaureate, GMAT etc.), depending on bilateral agreements, foreigners have access to education based on various fees. This adaptive feature is highly recommended by inclusive communication guidelines [29].

  • -. Personal information is closely related to the candidate and contains a significant amount of personal data which falls under GDPR regulations: data related to birth and identity, grades, medical certificates and educational options. Therefore strict access policies will be in place, both technically as well as administratively.

• AI Engine: includes the algorithms used in processing the knowledge acquisition, and modeling correct answers for the candidates. This module makes the connection between the two parties of the conversation, respectively feeds the candidate with very specific, personalized, and accurate information from the ontology. Moreover, the information is explained, so that the candidate has no doubt about the accuracy and conformity of the knowledge which is provided. This module is also able to learn based on user interaction and feedback so that the chatbot improves its knowledge, responsiveness, and usability with each conversation.

• Knowledge Acquisition: collects information from different sources such databases, cloud, documents, other software, and local data.

Figure 2

Ana Chatbot Architecture.

[Figure omitted. See PDF]

Of course, the Ana chatbot security is paramount as any university must protect the privacy of the candidates, respectively the personal data and preferences. That has to be done not only for GDPR compliance but also because people need to trust the Ana chatbot and the University of Medicine and Pharmacy "Iuliu Hatieganu" as well. The security is structured on three levels: (1) data security which regards the safety and reliability of the data, database and data flow; (2) ontology security which focuses on the consistency accuracy and quality of the ontology; (3) third party security which assures a proper, correct and responsible integration of the Ana chatbot with other applications.

3.4. Improved Lifecycle of the Chatbot

In this section we detail and expand the lifecycle briefly described in Section 3 along with the results for each stage.

The chatbot ontology lifecycle follows closely the classical ontological development lifecycle [30]. However, we include two more steps, namely deployment into production and maintenance because the chatbots are meant to be intensively used in production environments and have a life of their own. The six steps of the chatbot lifecycle are (see also Figure 3):

• conceptualization aims at identifying and defining the objects, concepts and their relationships referred by the ontology. It produces an agreed upon meaning for a concept for the purposes of research.

• formalization tries to bring the previously identified concepts to a canonical form, eventually structured on several abstraction levels.

• development implies the creation of the ontology using the formal concepts.

• testing assures that the ontology is sound and robust. The step is mainly technical, focused on the quantity of knowledge, rather than on its quality and is performed by ontology experts.

• deployment into production is the phase in which the chatbot interacts with the prospects and provides them with the expected, pertinent and correct answers.

• maintenance is required because a chatbot, like any other software can be considered a live agent which needs to be continuously updated as the information regarding admission process changes or the process itself evolves.

Figure 3

The improved chatbot lifecycle.

[Figure omitted. See PDF]

However, the chatbot has to be very adaptive, therefore the lifecycle is agile [31], with continuous updates based on the immediate needs of the users, rather than frozen in a rigid structure, such as waterfall [32].

3.4.1. Conceptualization

The conceptualization starts before developing the ontology of the chatbot, based on previous questions of the users. They are to be collated and then canonicalized, as there are many variations of the same question or the answers are slightly similar. And, finally, the statements are to be very personalized, appealing to the user and stated from her perspective.

The corpus of questions comes partially from the questions raised by the prospects in the previous admission session and partially from the clerks managing the process. All the specifications are structured in a table manner, as shown in Table 1 and Table 2. For each question (hop, node), we defined:

• node ID, as xxx.yyy.zzz, where xxx means a main node, xxx.yyy means a sub-node, xxx.yyy.zzz means a sub-sub-bode and so on. This is displayed in column ID.

• the content (information, action), shown in column Content.

• keywords used for accessing the respective info when the user types in her question. These are listed in column Keywords. The keywords concept is designed to admit possible typing errors like skipping a letter, or inversion of two letters. For this situation we used the * symbol which allows the keyword to have as many errors as we indicate in the beginning.

• related nodes, represented by their ID’s in column Related nodes.

Table 1

The questions known by the chatbot (I).

ID	Content	Keywords	Related Nodes
100	Welcome to the International Students Department
200	My name is Ana, how can I help you?
300	Please agree with GDPR policy	gdpr
300.100	If you did not agree with GDPR policy
400	Main menu	start
400.100	Connect to an operator	operator
400.100.100	Operator available
400.100.200	Operator unavailable	working, hours, work, hour
400.200	Admission platform
400.200.100	Video presentation	admission*, video*, tutorial*	600
400.200.200	How many pages can I upload?	pdf, page, pages	400.200.100, 600
400.200.300	What does “pending payment” mean?	Pending, Payment*, Pay	400.200.100, 600
400.300	Clarification regarding the application file
400.300.100	Multiple application files	file*, option*, application*	400.400.300, 600
400.300.200	Documents issued in a non-European country	non-EU, non-UE, non-European*, Hague, Apostille	400.300.100, 600
400.300.300	The medical certificate	Medical, Specialist*, Healt*	400.300.400, 600
400.300.400	Psychological examination	Psychological	400.300.300, 600
400.300.500	Expired passport	Passport, pasport, Valid, Expir*	600
400.400	Admission, education, taxes and prerequisites
400.400.100	What is your question related to language	Lang*
400.400.100.100	In what language are the courses taught	cours*, Corses, Corse	400.400.100, 600
400.400.100.200	How can applicants demonstrate their language proficiency?	proficienc*, certif*, prficiency	400.400.100.300, 400.400.100.400, 600

Table 2

The questions known by the chatbot (II).

ID	Content	Keywords	Related Nodes
400.400.100.300	If I studied in English/French, do I still need a Language Proficiency Certificate?		600
400.400.100.400	Can I pass a language proficiency examination at your university, before applying?	exam*, exmination	600
400.400.200	How many places are available?	place*, seat seats, medicin*, pharm*, farma*, dental, dent, dentistry, dentist	400.400.300, 600
400.400.300	What are the tuition fees and costs of the application process?	tuition*, fee, fees, cost, costs	600
400.400.400	How can I apply?	method*, metodology, ethodology, apply, appl, aply	400.200.100, 600
400.400.500	What is the deadline for submitting the application files and when are the results?	deadline, date, dates, dead, Result*, reslts	600
400.400.600	If I was not admitted in the Early Admission, is my file still in competition?	Early, erly	400.400.500, 600
500	Unfortunately, I do not understand your question.		400
600	Close the chat. Go to the main options. Connect to an operator	thank*	400, 400.100

3.4.2. Formalisation

We want the candidates to have the best user experience while using Ana chatbot. This means a feeling of interacting with a human and optimal access to knowledge, which is achieved in four ways. Firstly, the chatbot is called Ana, name ranked the 16th in the top of the most popular human names [33]. The name is feminine as there is a gender bias expectation in such applications, as reported by Feine et al. in [34] and McDonnell and Baxter in  [35]. Secondly, once in a while, the chatbot simulates the typing delay of a human operator, which means that the chatbot waits a couple of seconds before prompting its reply. Thirdly, for a complete user experience the chatbot interacts in several modes such as buttons to be selected by the user and free text. And fourthly the ontology is structured in a tree manner with optimal access to knowledge. The tree is built as suggested by Singer et al. in [36]. For each level of the tree, the information gain is computed according to the formula:

(1)$IG(S,a)=H\left(S\right)-H\left(S\right|a)$

where $IG(S,a)$ is the information for the dataset S for the variable a for a random variable, $H\left(S\right)$ is the entropy for the dataset before any change (described above) and $H\left(S\right|a)$ is the conditional entropy for the dataset given the variable a. In other words, the information gain is the difference in entropy of the information of the whole ontology minus the entropy of the sub-ontologies split by the attribute a [37].

The entropy is defined as [38]:

(2)$H\left(S\right)=-\sum _{i=1}^{n}P\left(s_i\right)logP\left(s_i\right)$

where $H\left(S\right)$ is the information entropy of the set $S=s_1,s_2,\cdots ,s_n$. $log$ is the logarithm, the choice of base varying between different applications. Base 2 gives the unit of bits, while base e gives the “natural units” nat, and base 10 gives a unit called “dits”.

The attribute with the highest discrimination power (which induces the largest information gain) is set as node [36].

For ethical and legal reasons, however the first replies of the chatbot are self-introductory and presenting the terms and conditions of use.

3.4.3. Development

The structure of the chatbot ontology is depicted in Figure 4. The semantics of the nodes are collated in Table 1 and Table 2. There are 6n main nodes corresponding to the main steps a conversation could have. From here on the nodes are represented by their ID’s, as displaying the actual content would make the presentation, explanations and graphs cumbersome.

The first three nodes (100, 200, 300) contain introductory sentences such as Welcome to the International Students Department. My name is Ana, your virtual assistant. The transition between these nodes (100, 200, 300) is automatic and the links are marked with the purple arrow. The flow between 300 and 400 is conditioned by the acceptance of GDPR policy by the user, otherwise the chat closed.

The node 400 represents the main menu containing the main options. From statement 100 to 400.*.*.*, the ontology is formalised as a tree (see Section 3.4.2). All the leaves of this tree (400.*.*.*) should provide answers to the user, but we also take into account that the answer is not the proper one. Therefore from those leaves, the user is provided with three options, marked as node 600:

• to go to the main menu (400), by typing start;

• to contact an operator (400.100), by typing operator;

• to close the chat, by typing thank you.

If a text input by the user is not understood (does not match sufficiently any of the ontological nodes), the bot outputs the text Unfortunately, I do not understand your question and sends an email to the knowledge expert, so that she can improve the ontology continuously based on real needs of the candidates.

3.4.4. Testing

The trust in chatbots is very important from a social perspective [39], therefore the tests were extensive and addressed the ontology as well as the business logic. The stage of the lifecycle has been performed by a very competent team consisting of all people involved in admission process over the years.

As the chatbot is to be used by people with various cultural and educational background, a special attention was given to the structure of the ontology and the way it is displayed, therefore we performed five sets of usability tests, aiming at:

• The users’ perception of human-like chatbot;

• The proper way to interact with a user;

• The depth of the ontological tree;

• The optimal number of available options;

• The number of interconnections between; ontological nodes;

The usability tests have been performed on 28 people (admission staff and students) using the Cognitive Walkthrough Method [40], which is an analytic inspection used to evaluate prototypes from the user’s perspective. The testers take the role of the user and “walk through” the process of using the product. The analysis uses storyboards and expert evaluation is based on the information obtained in the conceptualization phase (see Section 3.4.1). All 28 people involved in testing have either social or medical background, so none of them has strong IT-related skills, other than the usual ones.

The usability related to a specific aspect of the chatbot was measured by asking the subjects to rate it on a scale from 1 to 100. The charts display the average perceived usability on the vertical axis, where 0 means no usability and 100% means full ergonomy with respect to that specific aspect.

The information retrieval time was determined based on the logs of the chatbot and was measured as the total interaction time of the user with the chatbot divided by the number of information items searched and found by the users, based on their declaration.

The Users’ Perception of Human-like Chatbot

Figure 5 describes the impact which the interaction with the chatbot is having on the human perception, based on the rhythm of typing the automatic answers. Therefore as the chart reflects, if the answer came in less than 1 s, the perception of the conversation did not reach the desired objective, to be considered as human as possible. And also a typing response over 5 s, as long the conversations are practical, without having descriptive predefined answers. The vertical axis displays the percentage of users that considered the reaction of the chatbot as being the one of a human.

Modes of Interaction of the User with the Chatbot

The user can input information:

• by typing in the question;

• by choosing an option from a list of predefined ones;

• mixed—typing or/and choosing one of the predefined options provided by the chatbot.

The perceived interaction usability for each of the three scenarios is depicted in Figure 6. As expected, a mixed interaction is preferred to simple clicking and is way better than the old-school typing-in. The usability was graded on a scale from 1 to 100 by the users and Figure 6 displays the average grades for each possible type of interaction.

Depth of the Ontological Tree

The ontology is structured mainly as a tree (called ontological tree), as explained in Section 3.4.3. Definitely, there are cross-relationships between various nodes, which is the reason why the Figure 4 displays a graph, in which the ontological tree is depicted by gray arrows. However, the information is served to the user mainly in a tree-based manner.

Test and experiments aim at determining the optimal depth of the ontological tree, so that users feel comfortable of using it and find all the needed information in the shortest time. Therefore we asked the users to grade the usability of the platform and measured the time needed to reach the desired information, while varying the depth of the ontological tree from three to ten.

Figure 7 depicts the correlation between the depth of the ontology and the usability, respectively the correlation between the depth of the ontology and the average retrieval time. The X axis represents the depth of the ontology (from 3 to 10). The Y axes represent the usability of the chatbot as perceived by the user (image on the left), respectively the relative average time (in seconds) needed to retrieve the information (image on the right).

The usability ranges from 56% corresponding to a depth of 3, to 86% corresponding to a depth of 7. That means that the level 6–7 is the optimal depth to which an ontology is recommended to be developed. The average retrieval time behaves in the opposite way, which means that a too lower depth of the ontology determine high levels of retrieval, also a greater depth will lead to the same effect, due to the fact that information become more and more precise and specific, and also the quantity of knowledge is very large.

Optimal Number of Available Options

Figure 8 depicts the correlation between the number of available options from which a user can choose and the usability percentage. The X axis represents the depth of the ontology (from 3 to 8). The Y axes represent the usability of the chatbot as perceived by the user (image on the left), respectively the relative average time (in seconds) needed to retrieve the information (image on the right). The usability is having a higher percentage when the options to be chosen are fewer, which indicates that the accessibility to information reaches out to all of the four principles from a UX perspective: perceivable, operable, understandable, and robust [41].

The optimal number of available options should not exceed six, due to UX principles and the dimension of the screen utilized, because the visibility on the chatbot popup is limited, and this can lead to leakage of important information. The value of 6 options is a trade-off between the perceived usability (left chart, where usability is measured in percentages) and the information retrieval time (right chart, where the time is measured in seconds).

Number of Interconnections between Ontological Nodes

The interconnections between the ontological nodes are depicted in Figure 4 as green arrows. The optimal number of interconnections in this implementation is correlated to the number found by Yadav et al. in [42]. We started at a number of 32 nodes and the depth of the tree was 4. The correlated nodes from almost every leaf, increased the number of total backlinks to 32, meaning an average number of backlinks of 1.52 for each leaf.

It is important to define relations between nodes or topics, which may be needed together for a complete understanding by the user or which are complementary. It is hard to define key phrases to differentiate the specific information, from different persons, who in most cases don’t know what they are searching for, or often happens that the way they ask a question is unclear. To help and improve the conversation and the area of topics discussed, the chatbot offer hints in relevant and connected area of discussion, which may be helpful for the user to find the path to the specific information needed. This way, the chatbot will avoid having a circular conversation, which may lead to the same repetitive questions and answers. The diverse pool of questions will cover all the relevant information needed by user, from a specific topic.

3.4.5. Deployment into Production

As mentioned in Section 3, Ana was used during the application stage of the online admission process. It was trained initially on the questions in Table 1 and Table 2.

As shown in the software architecture depicted in Figure 2, the chatbot is able to integrate and collect information from third-party sources. For easier maintenance, we integrated the chatbot with:

• Admission platform (https://admissions.umfcluj.ro/frontend/auth/login, accessed on 15 January 2023);

• The database with specializations and number of available seats;

• The authentication details and granted rights of the admission committee.

To achieve openness and trust, by respecting the FAIR principles (findability, accessibility, interoperability and reusability), as well as GDPR regulation, imposed by the academic sector, the Ana chatbot obeys GDPR guidelines and ethical implications of handling sensitive data. On top of security measures such as authorization, authentication, and end-to-end encryption of communication channels, the following measures will be taken to preserve the candidate’s anonymity and are considered compliant with the privacy and security guidelines. Therefore the chatbot, but especially its integration with third-party data sources has been monitored against the top 10 OWASP security threats [43].

3.4.6. Maintenance

The chatbot is dynamic and is to be adapted to the context in which it operates and provides answers. Therefore its maintenance is crucial, more important than in most other cases of intelligent software systems [44]. Moreover, the testing phase was not exhaustive, therefore prolonged somehow into the production stage, hence the maintenance has also a second role, to fix possible inadvertence.

The maintenance of the chatbot requires a knowledge expert. The continuous development and maintenance of the ontology relies on:

• each stage of the admission process brings up new required questions and needs and leaves out caducous information;

• each question typed by the user and unanswered by the chatbot might be a new fact to be asserted to the ontology;

• other questions coming by other means (e.g., email, phone) are to be formalized and inserted into the ontological tree.

The feedback was requested at the end of each chat as a likable or dislikeable experience, as this is a classical way of collecting feedback from users online, as seen in Figure 9. Dislikes numbers decrease over time, while the number of likes increases at the same time. There is a little spike in dislikes in week 6 when the number of candidates was the highest, and human operators were not able to answer all the candidate’s questions received over the phone, email, and WhatsApp.

A significant amount of maintenance was allocated for adjusting the chatbot to the needs and questions of candidates during the admission sessions. The obsolescence of the information, such as admission deadlines, fees, places available for registration, etc., has been taken into account, due to the fact that it needs to be updated every year based on the admission methodology of the university, as well as in accordance with the decisions of the Ministry of Education. The number of available seats changes from one admission session to another, also there are questions that do not apply during the admission and are not relevant, but also there are questions that have the same answer throughout the time.

4. Results and Challenges Related to the Chatbot Lifecycle

4.1. Results of the Ana Chatbot

The ontology started with 32 nodes, out of which 7 first-level nodes and 21 leaves. The depth of the tree was 4. In total, there are 57 links to other nodes, averaging 1.84 for each node. The number of backlinks from leaves is 32, meaning an average of 1.52 backlinks from each leaf. The Ana chatbot was implemented in the 66 day out of 84 days of the international online admission session held by UMF Cluj between 4 May and 26 July 2022.

According to Table 3, in the 18 days, between 9 and 26 of July in which Ana worked alongside the human support staff, we had seen a total of 1342 chats being opened by candidates, on average with 75 chats per day, the most active day of the week being Monday with 105 opened chats on average, and the last active day of the week being Sunday with only 39 chats opened on average (See Figure 10).

The 75 average number of daily opened chats, or the total of 1342 open chats in the 18 day period corresponds to 40.27% of the number of international candidates (without Romanian citizenship) who submitted their online admission at UMF Cluj, out of which 78.51% (2616) were from Europe, 13.09% (436) from Africa, 5.58% (194) from Asia, 2.49% (83) from North America and only 0.09% (3) from South America, as seen in Figure 11, and more detailed in Table 4.

The usefulness of using Ana Chatbot as a bilingual (English and French) automated support assistant in a multicultural and international environment (See Figure 11) is clearly seen at UMF Cluj, as a large portion of international candidates from 76 countries used the chatbot as their primary support assistant, with a small ratio of “Dislikes” (See Figure 9) and a small number of redirected chats to human operators (See Figure 12).

In the analyzed 18 days in which Ana worked at UMF alongside human operators, the chatbot received a total of 5982 messages from candidates with a total of 16,459 responses out of which 15,843 (96.3%) were from Ana, as such it can be argued that Ana implementation proved successful in providing a 24/7 online admission support and assuring that the most basic and repetitive questions are addressed by the chatbot, and not by admission staff at UMF Cluj, which received only 809 messages from candidates (13.52%), representing the more complex questions that Ana couldn’t yet provide answers (See Table 3).

As depicted in Figure 12 and Figure 13 we can see that with the exception of Wednesday, responses from human operators to candidates inquires are less then 1, meaning that a large number of questions addressed by candidates to operators are not answered in the day they are addressed. This is due to the fact that UMF Cluj, like most universities, has only a Monday–Friday, 8 working hours schedule for answering questions through its admission staff, moreover as most candidates from the 2022 international admission were on different time zones than Eastern European Summer Time (EEST), there are also cases in which time zones between Romania and the candidates country are very different and thus provides a challenge for personal communication between international university candidates and the local admission staff.

In this instance, if candidates require the intervention of the academic staff on weekends or outside normal 9 AM–5 PM EEST working hours, they are told to send an email or reconnect between 9 AM–5 PM EEST from Monday to Friday. This lack of time synchronizations also represents one of the benefits of using Ana instead of relying heavily on human operators, as Ana not only works 24/7 but also adapts to previous more complex questions addressed to human operators, as in time its machine learning capabilities are developing alongside its Q&A database, which translates into a lesser reliance on human operators.

4.2. Sustainability of Ana Chatbot

The sustainability of the chatbot is given by continuous updates of the ontology according to the present reality of the admission, as things change over time. In some sense, the chatbot is alive and its ontology needs to be adjusted to the real life situation and that is achieved in the maintenance stage of the lifecycle.

From the total of 1342 chats opened in the last 18 days of the 2022 UMF International Admission Session, on average, as seen in Figure 14 each candidate generated 4.33 questions, out of which 3.83 (86.45%) were addressed to Ana and only 13.54% of the questions were addressed to human operators. On average we had 12.19 responses to 4.43 questions addressed in each chat or 2.75 responses for each question. Ana gave 11.74 from all of the total 12.19 responses per chat (96.30%). Only 3.7% out of the total answers sent to candidates were from operators.

For most of the questions addressed by candidates in an average chat, 86.45% were addressed to Ana, and only 13.54% of the questions were redirected to human operators. Out of the total questions addressed to human operators per chat, only 76.66% of them received a response (see Figure 14), as the admission staff at UMF responsible for chat responses usually work from 9 AM–5 PM (EEST) and only from Monday to Friday, in this sense chats opened by candidates that require human operators don’t receive a response in the weekends and/or outside business working hours. During the admission session, some users did not get the desired info directly from the chatbot. However, the number of unanswered questions decreased over time, as depicted in Figure 15.

The English version of the ontology contains 945 words structured in 45 nodes as shown in Section 3.4.3, which means an average of 21 words per message. As shown by Laufer et al. [45], adults typically type at about 40 words per minute when writing for enjoyment and 5 words per minute for in-depth essays or articles, with an average of 22.5 words per minute. This means that typing a message would require 0.93 min on average. As summarized in Table 3, the chatbot posted 15,843 messages, meaning 246.45 h saved in the 17 days between the 9th and the 26th of July, which account for 13.69 h (equivalent of 1.71 man-days) gained daily in favour of other tasks.

4.3. Challenges Related to the Chatbot Lifecycle

Developing and implementing Ana chatbot was provocative from the beginning. Some of the challenges that were raised were:

• Implementation issues due to a short deadline to finalize the chatbot;

• Formalisation of the admission concept was a challenge. In respect to the UX principle, the information must be very clear, easy to access, and easy to be read, in such a way that the result of the entire process is achieved.

• The multicultural identity of the chatbot raised brainstorming sessions for the team involved, due to the fact that there were a lot of differences between candidates countries regarding:

  • -. identity documents;

  • -. educational diplomas: Abitur, Bacalaureate, GMAT etc.;

  • -. language barrier which leaded to miss nominated words;

  • -. cultural habits.

• Developing Ana chatbot was not only for the purpose of researching the domain of the ontologies, it was a real-life project, implemented and used in 2022 for the admission process of the University of Medicine and Pharmacy “Iuliu Hatieganu” from Cluj-Napoca. Therefore, the main focus of the project was not on the research, but on facilitating communication way between the university staff and candidates all over the world. The aim of the ontology was to offer specific answers, in real-time, to candidates’ questions, covering time zone differences.

• The testing period was limited regarding the number and the type of persons involved in the process.

5. Learnt Lessons and Recommendations

Based on our real experience with UMF Cluj presented in this article, here are the recommended good practices:

• It is important that the chatbot has a human appearance in terms of name, icon (avatar), and behavior, meaning that, if the answers were delayed a couple of seconds, as written by humans, the user’s perception on the chat increased with 4.6%.

• The name of the bot has to be localized with respect to the geographical location of the users. Simple yet comprehensive names are preferred, based on world-wide ranks [33,46] and gender stereotypes [39].

• There should be a right mix between the number of predefined options or answers a user can choose from and the possibility to write open questions, freely. Too many or exclusive predefined options give the feeling of constriction; on the other hand, only free text makes the communication somehow exhausting for the user and difficult to formalize in terms of ontology.

• The maximum depth of the tree should be no more than six or seven (see Section “Depth of the Ontological Tree”, value correlated also with the research of Agarwal and Wadhwa [13].

• The number of predefined options within on reply should not exceed 6 (see Section “Optimal Number of Available Options”).

• It is important to define relations between related nodes or topics, which may be needed together for a complete understanding by the user or which are complementary (see Section “Number of Interconnections between Ontological Nodes”).

• For European countries, GDPR compliance is a must and the related conditions should be served at the very beginning of the chat before the first action is required from the user.

6. Conclusions

The article presents a chatbot used during the admission of foreign students at UMF Cluj University in July 2022.

The chatbot, called Ana has run 1342 times and forwarded the user to the operator 216 times, which means coverage of 83.9%. Due to the high amount of administrative work during the admission session, the knowledge expert succeeded to increased Ana’s ontology by barely 18.5%. For the next session, the knowledge is expected to increase by at least 120%. Due to business-related limitations, Ana was used only during the last 37.5% of the entire period. However, a whole admission session could be more conclusive and would provide deeper insight into what a chatbot is, how it works, and the way it improves the communication between the candidates and the administrative staff.

6.1. Answers to the Research Questions

Based on the experience with Ana chatbot, more specific research answers have been drawn.

• RQ1. 

  What is an appropriate structure of the ontology of the chatbot?

Firstly, the proper structure of the chatbot ontology covers the main information a candidate must know when applying to an university: from the faculties and the study domains, to the taxes applied and in the end to the documents needed to complete the enrolment process. Ana chatbot, can be scaled up to more universities, and to more countries, using different language pack and with a minimum effort the responses can be easily adapted due the fact that the core of the chatbot contains all these general aspects:

• faculties and the study domains

• taxes to be applied

• documents needed to complete the entire process

• RQ2. 

  How and to what extent a chatbot can contribute to the candidate support during the admission period?

Secondly, a chatbot can cover a huge amount of questions based on the candidate’s needs. It is an innovative support tool which handles properly repetitive questions came from different areas of the world, that for traditional admission process involved a considerable amount of money to cover all the expenses generated:

• for universities: salaries, indirect expenses, equipment usage;

• for candidates: the travel expenses, indirect expenses.

• RQ3. 

  What are good practices in desinging such a chatbot?

The good practices and the recommendations regarding the use of Ana chatbot are summarized in Section 5.

• RQ4. 

  What is the innovation and added value of such a chatbot to the admission process?

The innovation brought is described further in Section 6.2.

• RQ5. 

  What is the acceptance of such a chatbot and what is its (perceived) impact?

We live in an era where the technology is all over the domains. The chatbot fits perfect to the new generation, they are more accustom of using such tools. Based on our research, the digital native generation has shown good acceptance of Ana.

• RQ6. 

  What actions are needed for making the chatbot sustainable over more admission sessions?

Each admission session offers a huge amount of new information that can be converted into new nodes in the ontology. This leads to a better performing chatbot, that is more and more adapted to the needs of the candidates, and that leads to a higher response rate, that reduce to a minimum all the costs with the admission process. Obviously, the sustainability can be achieved by continuous maintenance of the chatbot and ontology and that is why the lifecycle has two extra steps Section 3, unlike the classical lifecycles: deployment into production and maintenance.

6.2. Novelty of the Ana Chatbot

In an era of digitized systems, Ana succeeded in sustaining the admission process for the University of Medicine and Pharmacy “Iuliu Hatieganu” from Cluj-Napoca, in July 2022. The implementation of the Ana chatbot brings novelty aspects into the field by:

• developing an ontology that pinpoints the multiculturalism aspects, being the first chatbot to document the acceptance and results of such technology with candidates from 76 countries;

• expanding the digitization activities in universities, especially in domains that are not directly related to the ICT;

• conceptualizing internal processes of a university in an innovative way;

• improving the ontological tree by collecting the data in a dynamic mode, and structuring the information in three different categories: public, personal and adaptive. In that way, all the candidates, staff, or third parties can co-create to improve the ontological tree of the chatbot;

• increasing productivity, helping users to obtain timely and efficient assistance or information. Therefore the time zone constriction it’s not anymore a limitation, the candidate will obtain the answer in real-time.

6.3. Future Developments of Ana Chatbot

Building an ontology for the admission process storm up a diversity of ideas to be implemented in the future. One of the most powerful skills nowadays is to know what to do with data: organize-store, use-analyze, share, reuse-maintain, archive-destroy, create-capture&collect [47];

A priority for Ana chatbot is the interoperability with other platforms used in the admission and enrolment process.

From the beginning of the conversation, we want the chatbot to have an open discussion with the prospects, covering the mentoring area in guiding and leading users to find out the specialization domains available at the university offers, and what fits them best. And in the next step the conversation to continue with all the procedural aspects of the enrolment.

Another future development will be multiplying the chatbot for all the faculties inside the university and improving its knowledge with specific information regarding the educational curriculum for each specialization, in that way the candidates may ask a specific question regarding the domain she wants to study.

Another future goal is to expand the ontological tree with the result from previous admission sessions and also to have the chatbot active all over the year for offering news, information about the university etc.

One of the goals of implementing the chatbot in a university was to increase the number of international students, therefore the accessibility aspects will be improved by integrating the Ana chatbot with the most popular online messaging services, such as Whats App and Facebook, in order to make the chatbot proactive and address questions to candidates in relation to their admission application. Therefore the multicultural level among students in university will increase.

An important aspect that needs to be considered is the constant improvement of the security of the data processed among the conversations, applying methodologies such as the ones proposed in the BIECO project (www.bieco.org, accessed on 15 January 2023) [48].

Generating more statistics, reports, and future prediction based on the previous admission session will help the university to improve its educational offer, to be more connected to the candidate’s needs all over the world, and to co-design and co-create with them the future of the university in the matter of creative learning in an ICT world.

Author Contributions

Project administration: S.C.M.; Methodology: O.M.; Software: L.A. and O.M.; Vizualization: L.A. and T.F.; Validation: S.C.M. and D.D.; Writing: L.A., T.F. and O.M. All authors have read and agreed to the published veriosn of the manuscript.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

The data presented in this study are available on request from the corresponding author. The data belongs to the University of Medicine and Pharmacy “Iuliu Hatieganu” from Cluj-Napoca, Romania (www.umfcluj.ro, accessed on 15 January 2023).

Acknowledgments

The research reported in this article has been done within HOLISUN (www.holisun.com, accessed on 15 January 2023) company, during the admission session at the University of Medicine and Pharmacy “Iuliu Hatieganu” from Cluj-Napoca, Romania (www.umfcluj.ro, accessed on 15 January 2023).

Conflicts of Interest

The authors declare no conflict of interest.

Sample Availability

Ana chatbot is available on https://admissions.umfcluj.ro/ accessed on 15 January 2023 during the admission sessions.

Abbreviations

The following abbreviations are used in this manuscript:

API	Application Programming Interface
AIML	Artificial Intelligence Markup Language
DINA	Dinus Intelligent Assistance
FAIR	Findability, Accessibility, Interoperability, Reusability
FAQ	Frequently Asked Questions
GDPR	General Data Protection Regulation
GMAT	Graduate Management Admission Test
ID	Identification
LSA	Latent Semantic Analysis
Q&A	Questions and Answers
UMF Cluj	University of Medicine and Pharmacy "Iuliu Hatieganu" from Cluj-Napoca
UDINUS	Universitas Dian Nuswantoro
UX	User Experience

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figures and Tables

Figure 1. University admission process.

Figure 4. The semantic diagram of the chatbot ontology.

Figure 5. The users’ perception of human-like chatbot.

Figure 6. Perceived Interaction Usability.

Figure 7. The depth of the ontological tree.

Figure 8. Optimal number of available options.

Figure 9. Like to dislike ratio over the whole admission period.

Figure 10. Average weekly opened chats.

Figure 11. The distribution of the candidates based on their continent of origin.

Figure 12. Messages sent by candidates to the bot and operators.

Figure 13. Messages sent by bot and operator to candidates.

Figure 14. The number of messages from the candidates, from the bot, from the operator and the number of chats to the operator and to the bot.

Figure 15. Number of messages without an answer from the chatbot or operator.

Table 3

Total daily chats results between 9 and 26 of July.

Days	Total			Candidates		Responses
				Message to		From
	Chats	Mess. from candid.	Total mess.	Bot	Operator	Bot	Operator
Monday*(3)	314	1543	5569	1347	196	3928	98
Tuesday*(2)	161	780	2823	649	131	1946	97
Wednesday*(2)	189	827	3141	708	119	2125	189
Thursday*(2)	172	877	3093	653	224	2082	134
Friday*(3)	241	939	3721	818	121	2698	84
Saturday*(3)	149	565	2312	562	3	1744	3
Sunday*(3)	116	451	1782	436	15	1320	11
Total	1342	5982	22,441	5173	809	15,843	616
Average	192	855	3206	739	116	2263	88
Avg. per day	75	332	1247	287	45	880	34

Table 4

International Candidates during the 2022 UMF Cluj International Admission.

No.	Country	%	No.	Country	%	No.	Country	%
1	France	61.40%	27	Cyprus	0.24%	53	Ukraine	0.06%
2	Germany	7.26%	28	Lebanon	0.21%	54	Georgia	0.06%
3	Morocco	6.66%	29	Pakistan	0.18%	55	Island	0.06%
4	Tunisia	3.87%	30	Austria	0.18%	56	Somalia	0.03%
5	Italy	1.74%	31	Iran	0.15%	57	Sudan	0.03%
6	Israel	1.65%	32	Luxembourg	0.15%	58	Benin	0.03%
7	U.A.E.	1.50%	33	Afghanistan	0.12%	59	Burkina Faso	0.03%
8	Switzerland	1.44%	34	Cameroon	0.12%	60	Cambodia	0.03%
9	Belgium	1.35%	35	Egypt	0.12%	61	Kenya	0.03%
10	Sweden	1.20%	36	Ghana	0.12%	62	Mauritania	0.03%
11	Guadeloupe	1.14%	37	Nigeria	0.12%	63	Niger	0.03%
12	Reunion	0.87%	38	Jordan	0.12%	64	Gabon	0.03%
13	Greece	0.78%	39	Kuwait	0.12%	64	Antigua & Barbuda	0.03%
14	Canada	0.69%	40	South Africa	0.09%	66	Haiti	0.03%
15	Saudi Arabia	0.66%	41	Dem. Rep. Congo	0.09%	67	East Timor	0.03%
16	United States	0.54%	42	Madagascar	0.09%	68	South Korea	0.03%
17	Qatar	0.51%	43	Brazil	0.09%	69	Indonesia	0.03%
18	United Kingdom	0.51%	44	Oman	0.09%	70	Malaysia	0.03%
19	Algeria	0.42%	45	Australia	0.09%	71	Poland	0.03%
20	Ireland	0.39%	46	Japan	0.09%	72	Hungary	0.03%
21	Spain	0.30%	47	Netherlands	0.09%	73	Albania	0.03%
22	Finland	0.30%	48	Senegal	0.06%	74	Croatia	0.03%
23	India	0.27%	49	Ethiopia	0.06%	75	Denmark	0.03%
24	Turkey	0.27%	50	Jamaica	0.06%	76	Monaco	0.03%
25	Norway	0.24%	51	Palestine	0.06%
26	Moldova	0.24%	52	Portugal	0.06%

References

1. Martínez-García, I.; Nielsen, T.; Alastor, E. Perceived stress and perceived lack of control of spanish education-degree university students: Differences dependent on degree year, basis for admission and gender. Psychol. Rep.; 2021; 125, pp. 1824-1851. [DOI: https://dx.doi.org/10.1177/00332941211006023] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/33823680]

2. Bakti, R.; Hartono, S. The Influence of Transformational Leadership and work Discipline on the Work Performance of Education Service Employees. Multicult. Educ.; 2022; 8, pp. 109-125.

3. Tuhuteru, H.; Siwalette, R. System Design and Implementation of Online Admission System at XYZ University. Inspir. J. Teknol. Inf. Dan Komun.; 2022; 12, pp. 105-117. [DOI: https://dx.doi.org/10.35585/inspir.v12i2.30]

4. Barus, S.P.; Surijati, E. Chatbot with Dialogflow for FAQ Services in Matana University Library. Int. J. Informatics Comput.; 2022; 3, pp. 62-70. [DOI: https://dx.doi.org/10.35842/ijicom.v3i2.43]

5. Chandra, Y.W.; Suyanto, S. Indonesian chatbot of university admission using a question answering system based on sequence-to-sequence model. Procedia Comput. Sci.; 2019; 157, pp. 367-374. [DOI: https://dx.doi.org/10.1016/j.procs.2019.08.179]

6. Mariacher, N.; Schlögl, S.; Monz, A. Investigating perceptions of social intelligence in simulated human-chatbot interactions. Progresses in Artificial Intelligence and Neural Systems; Springer: Cham, Switzerland, 2021; pp. 513-529.

7. Goot, M.J.; Hafkamp, L.; Dankfort, Z. Customer service chatbots: A qualitative interview study into the communication journey of customers. Proceedings of the International Workshop on Chatbot Research and Design; Online, 23–24 November 2020; pp. 190-204.

8. Patel, N.P.; Parikh, D.R.; Patel, D.A.; Patel, R.R. Ai and web-based human-like interactive university chatbot (unibot). Proceedings of the 2019 3rd International Conference on Electronics, Communication and Aerospace Technology (ICECA); Coimbatore, India, 12–14 June 2019; pp. 148-150.

9. Chocarro, R.; Cortiñas, M.; Marcos-Matás, G. Teachers’ attitudes towards chatbots in education: A technology acceptance model approach considering the effect of social language, bot proactiveness, and users’ characteristics. Educ. Stud.; 2021; pp. 1-19. [DOI: https://dx.doi.org/10.1080/03055698.2020.1850426]

10. Vlasiuk, R.; Petko, L. Alan Turing: A Founding Father of Computer Science, Artificial Intelligence and Modern Cognitive Science. Ph.D. Thesis; Lviv Polytechnic National University: Lviv, Ukraine, 2022.

11. Saygin, A.P.; Cicekli, I.; Akman, V. Turing test: 50 years later. Minds Mach.; 2020; 10, pp. 463-518. [DOI: https://dx.doi.org/10.1023/A:1011288000451]

12. Bassett, C. Apostasy in the temple of technology: ELIZA the more than mechanical therapist. Anti-Computing; Manchester University Press: Manchester, UK, 2022; pp. 168-185.

13. Agarwal, R.; Wadhwa, M. Review of state-of-the-art design techniques for chatbots. SN Comput. Sci.; 2020; 1, pp. 1-12. [DOI: https://dx.doi.org/10.1007/s42979-020-00255-3]

14. Xie, T.; Yang, X.; Lin, A.S.; Wu, F.; Hashimoto, K.; Qu, J.; Kang, Y.M.; Yin, W.; Wang, H.; Yavuz, S. Converse–A Tree-Based Modular Task-Oriented Dialogue System. arXiv; 2022; arXiv: 2203.12187

15. Park, D.M.; Jeong, S.S.; Seo, Y.S. Systematic Review on Chatbot Techniques and Applications. J. Inf. Process. Syst.; 2022; 18, pp. 26-47.

16. Tharammal, M.K.P.; Bashir, M.N.; Yusof, K.M.B.; Iqbal, S. ALICE Pattern Matching Based Chatbot for Natural Language Communication: System Development and Testing. iKSP J. Comput. Sci. Eng.; 2022; 2, pp. 34-42.

17. Thomas, L.; Kumar, M.; Prashanth, B.S.; Sneha, H.R. Seq2seq and Legacy techniques enabled Chatbot with Voice assistance. Proceedings of the 2022 IEEE 2nd Mysore Sub Section International Conference (MysuruCon); Mysuru, India, 16–17 October 2022; pp. 1-4.

18. Kaczorowska-Spychalska, D. Chatbots in marketing. Management; 2019; 23, pp. 251-270. [DOI: https://dx.doi.org/10.2478/manment-2019-0015]

19. Hughes, T.M.; Leafstedt, J.M. Communicating with Generation Z. The COVID-19 Impact on Higher Education Stakeholders and Institutional Services; Lexington Books: Lanham, MD, USA, 2022; Volume 93.

20. Barrett, M.; Branson, L.; Carter, S.; DeLeon, F.; Ellis, J.; Gundlach, C.; Lee, D. Using artificial intelligence to enhance educational opportunities and student services in higher education. Inquiry J. Va. Community Coll.; 2019; 22, 11.

21. Bačanin Džakula, N. Singibot-a student services chatbot. Sinteza 2020-International Scientific Conference on Information Technology and Data Related Research; Singidunum University: Belgrade, Serbia, 2020; pp. 318-323.

22. Hersi, A.H.; Hassan, M.M.; Hassan, A.A.; Mahdi, M.A.; Abdulle, A.W. An Intelligent Somali Language Chatbot Serving as an Online Admission Help Desk; SORER: Mogadishu, Somalia, 2021.

23. Nazir, A.; Khan, M.Y.; Ahmed, T.; Jami, S.I.; Wasi, S. A novel approach for ontology-driven information retrieving chatbot for fashion brands. Int. J. Adv. Comput. Sci. Appl. IJACSA; 2019; 10, [DOI: https://dx.doi.org/10.14569/IJACSA.2019.0100972]

24. Vegesna, A.; Jain, P.; Porwal, D. Ontology based chatbot (for e-commerce website). Int. J. Comput. Appl.; 2020; 179, pp. 51-55. [DOI: https://dx.doi.org/10.5120/ijca2018916215]

25. Ranoliya, B.R.; Raghuwanshi, N.; Singh, S. Chatbot for university related faqs. Proceedings of the 2017 International Conference on Advances in Computing, Communications and Informatics (ICACCI); Udupi, India, 13–16 September 2017; pp. 1525-1530.

26. Santoso, H.A.; Winarsih, N.A.S.; Mulyanto, E.; Wilujeng Saraswati, G.; Enggar Sukmana, S.; Rustad, S.; Syaifur Rohman, M.; Nugraha, A.; Firdausillah, F. Dinus intelligent assistance (dina) chatbot for university admission services. Proceedings of the 2018 International Seminar on Application for Technology of Information and Communication; Semarang, Indonesia, 21–22 September 2018; pp. 417-423.

27. Colace, F.; DeSanto, M.; Lombardi, M.; Pascale, F.; Pietrosanto, A.; Lemma, S. Chatbot for e-learning: A case of study. Int. J. Mech. Eng. Robot. Res.; 2018; 7, pp. 528-533. [DOI: https://dx.doi.org/10.18178/ijmerr.7.5.528-533]

28. Dimitriadis, G. Evolution in education: Chatbots. Homo Virtualis; 2020; 3, pp. 47-54. [DOI: https://dx.doi.org/10.12681/homvir.23456]

29. McCarty, T.V.; Light, J.C. Supporting peer interactions for students with complex communication needs in inclusive settings: Paraeducator roles. Perspect. Asha Spec. Interest Groups; 2022; 7, pp. 229-244. [DOI: https://dx.doi.org/10.1044/2021_PERSP-21-00141]

30. Sattar, A.; Surin, E.S.M.; Ahmad, M.N.; Ahmad, M.; Mahmood, A.K. Comparative analysis of methodologies for domain ontology development: A systematic review. Int. J. Adv. Comput. Sci. Appl.; 2020; 11, [DOI: https://dx.doi.org/10.14569/IJACSA.2020.0110515]

31. Abdelghany, A.; Darwish, N.R.; Hefni, H.A. An agile methodology for ontology development. Int. J. Intell. Eng. Syst.; 2019; 12, pp. 170-181. [DOI: https://dx.doi.org/10.22266/ijies2019.0430.17]

32. Kramer, M. Best practices in systems development lifecycle: An analyses based on the waterfall model. Rev. Bus. Financ. Stud.; 2018; 9, pp. 77-84.

33. Sidhu, D.M.; Deschamps, K.; Bourdage, J.S.; Pexman, P.M. Does the name say it all? Investigating phoneme-personality sound symbolism in first names. J. Exp. Psychol. Gen.; 2019; 148, 1595. [DOI: https://dx.doi.org/10.1037/xge0000662] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/31368761]

34. Feine, J.; Gnewuch, U.; Morana, S.; Maedche, A. Gender bias in chatbot design. Proceedings of the International Workshop on Chatbot Research and Design; Amsterdam, The Netherlands, 19–20 November 2019; pp. 79-93.

35. McDonnell, M.; Baxter, D. Chatbots and gender stereotyping. Interact. Comput.; 2019; 31, pp. 116-121. [DOI: https://dx.doi.org/10.1093/iwc/iwz007]

36. Singer, G.; Anuar, R.; Ben-Gal, I. A weighted information-gain measure for ordinal classification trees. Expert Syst. Appl.; 2020; 152, 113375. [DOI: https://dx.doi.org/10.1016/j.eswa.2020.113375]

37. Zebari, R.; Abdulazeez, A.; Zeebaree, D.; Zebari, D.; Saeed, J. A comprehensive review of dimensionality reduction techniques for feature selection and feature extraction. J. Appl. Sci. Technol. Trends; 2020; 1, pp. 56-70. [DOI: https://dx.doi.org/10.38094/jastt1224]

38. Wu, W.; Hou, J.; Zhang, Z.; Li, F.; Zhang, R.; Gao, L.; Ni, H.; Zhang, T.; Long, H.; Lei, M. et al. Information entropy-based strategy for the quantitative evaluation of extensive hyperspectral images to better unveil spatial heterogeneity in mass spectrometry imaging. Anal. Chem.; 2022; 94, pp. 10355-10366. [DOI: https://dx.doi.org/10.1021/acs.analchem.2c00370]

39. Toader, D.C.; Boca, G.; Toader, R.; Măcelaru, M.; Toader, C.; Ighian, D.; Rădulescu, A.T. The effect of social presence and chatbot errors on trust. Sustainability; 2020; 12, 256. [DOI: https://dx.doi.org/10.3390/su12010256]

40. Weninger, M.; Grünbacher, P.; Gander, E.; Schörgenhumer, A. Evaluating an interactive memory analysis tool: Findings from a cognitive walkthrough and a user study. Proc. ACM Hum.-Comput. Interact.; 2020; 4, pp. 1-37. [DOI: https://dx.doi.org/10.1145/3394977]

41. Díaz, E.; Arenas, J.J.; Moquillaza, A.; Paz, F. A systematic literature review about quantitative metrics to evaluate the usability of e-commerce web sites. Proceedings of the International Conference on Intelligent Human Systems Integration; Modena, Italy, 19–21 February 2019; pp. 332-338.

42. Kaushal, V.; Yadav, R. Exploring B2B Chatbots Adoption Experiences: Lessons for Successful Implementation in Businesses. Res. Sq.; 2022; preprint [DOI: https://dx.doi.org/10.21203/rs.3.rs-1776145/v1]

43. Lala, S.K.; Kumar, A.; Subbulakshmi, T. Secure web development using owasp guidelines. Proceedings of the 2021 5th International Conference on Intelligent Computing and Control Systems (ICICCS); Madurai, India, 6–8 May 2021; pp. 323-332.

44. Sarferaz, S. Lifecycle Management. Compendium on Enterprise Resource Planning; Springer: Cham, Switzerland, 2022; pp. 559-570.

45. Batia, L.; Rozovski-Roitblat, B. Incidental vocabulary acquisition: The effects of task type, word occurrence and their combination. Lang. Teach. Res.; 2011; 2011 15, pp. 391-411.

46. Baxter, D.; McDonnell, M.; McLoughlin, R. Impact of chatbot gender on user’s stereotypical perception and satisfaction. Proceedings of the 32nd International BCS Human Computer Interaction Conference; Belfast, UK, 4–6 July 2018; pp. 1-5.

47. Rahul, K.; Banyal, R.K. Data lifecycle management in big data analytics. Procedia Comput. Sci.; 2020; 173, pp. 364-371. [DOI: https://dx.doi.org/10.1016/j.procs.2020.06.042]

48. Matei, O.; Erdei, R.; Delinschi, D.; Andreica, L. Data based message validation as a security cornerstone in loose coupling software architecture. Computational Intelligence in Security for Information Systems Conference; Springer: Berlin/Heidelberg, Germany, 2021; pp. 214-223.