Sentiments analysis of customer satisfaction in public services using K-nearest neighbors algorithm and natural language processing approach

Abstract

Customer satisfaction is very important for public service providers, customer satisfaction can be delivered with a survey application or writing criticism that can be used to evaluate and improve service. Unfortunately, there are only a few customers who are willing to give an assessment. The survey application cannot represent the overall feeling of the customer, so it is necessary to analyze the content of the conversation between the customer and the service personnel to determine the level of customer satisfaction. In small amounts, it can be done manually, but in large quantities it is more effective to use the system. A solution is needed in the form of a system that converts voice conversations into text and analyzes customer satisfaction to obtain information for evaluation and improvement of services. This research uses K-nearest neighbors (KNN) and term frequency-inverse document frequency (TF-IDF) algorithm with natural language processing (NLP) approach to classify conversations into 2 classes, "satisfied" and " dissatisfied ". The results of this study received 74.00% accuracy, 76.00% precision and 73.08% recall. In conversations with the label "satisfied" shows customers satisfied with the service and fulfillment of customer desires, while in conversations with the label "not satisfied" customers are less satisfied with the waiting time.

Headnote

Keywords:

K-nearest neighbors

Natural language processing

Public service

Sentiment analysis

Speech recognition system

ABSTRACT

Customer satisfaction is very important for public service providers, customer satisfaction can be delivered with a survey application or writing criticism that can be used to evaluate and improve service. Unfortunately, there are only a few customers who are willing to give an assessment. The survey application cannot represent the overall feeling of the customer, so it is necessary to analyze the content of the conversation between the customer and the service personnel to determine the level of customer satisfaction. In small amounts, it can be done manually, but in large quantities it is more effective to use the system. A solution is needed in the form of a system that converts voice conversations into text and analyzes customer satisfaction to obtain information for evaluation and improvement of services. This research uses K-nearest neighbors (KNN) and term frequency-inverse document frequency (TF-IDF) algorithm with natural language processing (NLP) approach to classify conversations into 2 classes, "satisfied" and " dissatisfied ". The results of this study received 74.00% accuracy, 76.00% precision and 73.08% recall. In conversations with the label "satisfied" shows customers satisfied with the service and fulfillment of customer desires, while in conversations with the label "not satisfied" customers are less satisfied with the waiting time.

(ProQuest: ... denotes formulae omitted.)

1.INTRODUCTION

Public satisfaction is a determining factor in the quality of public services, so every organization that provides public services is expected to be able to provide satisfaction to customers [1]. To find out customer satisfaction, the right measuring instrument is needed [2]. Customer satisfaction is usually measured by a survey application or critical writing that can be used to evaluate and improve service. Unfortunately, there are only a few customers who are willing to give an assessment. Low conditions of public services are characterized by public services that are not transparent, discriminatory, complicated, and corrupt. The application of public service policy innovations becomes a necessity in realizing quality public services [3].

The survey application cannot represent the overall feeling of the customer, it is necessary to analyze the contents of the conversation between the customer and the service personnel to determine the level of customer satisfaction and absorb and understand customer desires [4]. Another purpose is to prevent discriminatory or corrupt practices in the delivery of services, so that they can deliver services that are fair, democratic and friendly [5]. Analysis of service conversations in small amounts can be done manually, but in it is more effective to use the system in large quantities.

Researchers have developed a product called "Kata Kita" which serves to support public services. "Kata Kita" has the feature of translating conversations into text and displaying them on the screen to help the deaf understand service personnel and sentiment analysis features to assess customer satisfaction from the conversation. Sentiment Analysis is used to find out the opinion about a topic which is as positive, negative, and neutral sentiments [6]. The purpose of this study is to examine the effectiveness of "Kata Kita" to conduct sentiment analysis on public service conversations in one of the hospitals in Yogyakarta, Indonesia.

Conversations are recorded and converted to text using Speech Recognition System technology. Speech recognition system is the process of converting speech signals into word sequences using an algorithm implemented as a computer program [7]. The text of the conversation is processed using machine learning to be labeled automatically without the need to wait for input from customers, so that the process of evaluating and improving services can run effectively.

Previous research by Lutfi and Permatasari [8] analyzed the marketplace to find out positive or negative user sentiment with an approach utilizing support vector machine with an accuracy of 93.42%. In a study conducted by Akhmad Deviyanto and Wahyudi [9] presented a sentiment analysis of opinion data on Twitter user sentiment analysis on the topic of local elections in the Jakarta of Indonesia in 2017 with an accuracy of 67.2%. In another study by Norman Kendal [10] showed the use of natural language processing (NLP) can improve the prediction of fashion product categories or sub categories of product titles.

The purpose of this study is to analyze the sentiments of public service conversation in one of the hospitals in Yogyakarta, Indonesia using the uses K-nearest neighbors (KNN) algorithm by weighting term frequency-inverse document frequency (TF-IDF), and classify into two classes "satisfied" and "dissatisfied". Then find out attributes which are the focused of "satisfied" and "dissatisfied" assessments, so that service providers can properly evaluate and improve the system [11]. Machine learning is an efficient way to extract knowledge from large amounts of data [12]. The problem in analyzing public service conversations is the large number of non-standard words that are difficult to understand by the system. The NLP approach is needed to improve the language of the conversation so that it is more easily understood by the system. Without NLP, machine learning cannot make meaningful progress [13].

2.RESEARCH METHOD

The flow of research sentiment analysis of conversations in public services using the KNN and TF-IDF algorithm by using the NLP approach is explained in Figure 1. The initial step of the study was the collection of recorded conversations of front desk services at one of the hospitals in Yogyakarta, Indonesia on July 13, 2020. Conversation recordings were converted into text using the "Kata Kita" application with the speech recognition system from Google speech-to-text [14]. The next step, text was stored in a MySQL database so that it is easily managed [15]. The workflow of the speech to text system in the "Kata Kita" application is explained in Figure 2.

The next step, conversation text was then purchased manually by three respondents, consisting of customers, public relations and medical students. The dataset of 250 conversations was labeled in two classes, "satisfied" and "dissatisfied". The last step, dataset is subjected to a preprocessing process using the NLP approach. Preprocessing data is the process of cleaning and preparing data for analysis [16]. The purpose of preprocessing is to correct the language in the conversation, because in the conversation found many uses of non-standard words and local languages. In the case of text classification, many preprocessing techniques can be used [17]. The workflow of the preprocessing stage with the NLP approach is explained in Figure 3.

NLP is a breakthrough in transcending language barriers [16]. NLP is a branch of Artificial Intelligence that focuses on processing natural language effectively and accurately as humans do [18]. The NLP features used in this study are as follows :

- Lowercase folding

Lowercase folding is changing all tokens to lowercase letters [19]. Conversation "Selamat pagi kak ada yang bisa dibantu " will be changed to "selamat pagi kak ada yang bisa dibantu ", alphabet "S" in the form of capital is changed into"s" in small form.

- Word normalizer

The word normalizer feature is used to correct words in sentences, so that a good and correct sentence is produced according to grammar rules [20]. Conversation "selamat pagi kak ada yang bisa dibantu" then after the Word Normalizer process becomes "salam kakak ada yang bisa dibantu ". Term "selamat pagi" changed into "salam" and term "kak" changed into "kakak".

- Stemming

The stemming feature functions is to reduce words to basic words by eliminating the affixes that exist in those words [21]. For example, sentences "salam kakak ada yang bisa dibantu" changed into "salam kakak ada yang bisa bantu", term "dibantu" changed to basic words "bantu".

- Stopword removal

The stopword removal feature works to eliminate words that often appear in natural language but have very little meaning [22]. For example sentences "salam kakak ada yang bisa bantu" changed into "salam kakak bisa bantu ", term "ada" and "yang" removed.

Term frequency is a method for finding the weight of a document by looking for the number of occurrences of terms in the document. The more often the term appears, it will affect the amount of weight and the suitability value of the document. Inverse Document Frequency is a method for calculating the distribution of terms in a document [23]. The TF-IDF method allows documents to be classified into two classes (positive and negative) [24]. TF-IDF calculation using (1).

... (1)

W_x,y is the weight of the term (t_y) of document (d_x). While tf_x,y is the number of occurrences of term (t_y) in document (d_x). N is the number of documents in the database and df_x is the number of documents containing the term (t_y), there is at least one word, term (t_y).

3.RESULTS AND ANALYSIS

3.1.Service conversation text

Conversation text used is obtained from the activity of changing conversations into text. This service conversation was taken in July 2020 at a hospital in Yogyakarta, Indonesia. Examples of conversations are shown in Table 1. Based on Table 1, Customer-1 was redeeming drugs at the hospital pharmacy and is satisfied with the service he received. Customer-2 was paying medical expenses at the cashier, but he had been transferred to another cashier for using Badan Penyelenggara Jaminan Sosial Ketenagakerjaan (BPJS) without saying an apology, so the customer felt dissatisfied. Customer-3 was talking to doctors with a mixture of Javanese words about the process of recovering hands, and feeling satisfied with the services provided. Customer-4 was dissatisfied when he was in a queue position, and he must leave the hospital temporarily, while the queue order will return from the beginning if the customer did not return when he was called. Customer-5 was in a child's poly for immunizing his child and he was feeling satisfied because the doctor can calm the customer. Information from this conversation can be processed to be a future improvement.

3.2.Conversation analysis

The data used 250 conversations, which were labeled by 3 correspondents and resulted in 125 "satisfied" conversations and 125 "dissatisfied" conversations. The conversation is made into a dataset with a composition of 200 training data and 50 testing data, then preprocessing using the NLP approach. At the preprocessing stage the word normalizer, stemming and stopword removal features are applied for each conversation.

Text normalization is a challenge to find the same words with different word variations [25]. Word normalizer can handle variations of writing words which have the same meaning so that they are counted as a single term in the calculation of the TF-IDF algorithm. Examples of variations in writing words which have the same meaning are shown in Table 2. In Table 2 shows variations of words which have the same meaning. Term "pak", "bu", "dok", "kak", "dik", "mas" and "mbak" used to refer the other person, so that it can be replaced with term "sodara". Term "selamat sore", "selamat siang"" and "assalamualaikum" used to give greetings so that it can be replaced with term "salam".

The next stage is the process of stemming, to eliminate the prefixes, inserts and word suffixes so that it becomes the basic form. Stemming is an important technique in NLP for efficient and effective information retrieval [26]. An example of stemming application is shown in Table 3. In Table 3 terms "diperiksa", "diperiksakan" dan "memeriksa" omitted prefixes, inserts and endings will be the same basic term is "periksa". Terms "mengantar" and "diantar" after going through the stemming process, it becomes the same basic term is "antaf". Stemming makes a word into its basic form and becomes the same term.

The next step is the stopword removal process to remove the stopword from the conversation. The stopword list which is used is made by itself, refers to the context of words which are often used in hospital services and numbers. Number have no effect on sentiment analysis, and removing them can reduce noise and increase efficiency [27]. An example of a stopword list used is shown in Table 4. Can be seen in Table 4, the term "yang" is one of the most frequently stoppedword lists, which is 204 times, and the term "di" which appears 101 times. Stopwords can be removed because they usually appear in large numbers and do not have meaning as a single term. The overall results of the preprocessing process can be seen in Table 5. Table 5 shows conversations which have passed preprocessing using the NLP approach with the word normalize, stemming and stopword removal features. One indicator of satisfied customers is to say "terima kasih" at the end of the conversation which indicates that customers feel helped and are satisfied with the services provided.

In conversation number 2, there is a mistake of the customer because of the wrong choice of the cashier, but customers disappointed cause is that no words of forgiveness make customers feel comfortable. Apology deals with the relationship between individuals involved in the wrong situation [28]. The choice and word usage in public services affect the level of customer satisfaction, although service personnel must be strict with the rules, with the right choice of words can maintain customer feelings.

3.3.KNN calculation using TF-IDF

This study uses the KNN algorithm with k=3 and the term weighting uses TF-IDF. TF-IDF reflects the importance of a word in a text document [29]. The TF-IDF calculation is applied to 200 training data and 50 testing data after preprocessing using the NLP approach. The next step is calculating the number of matches labeling results from the prediction of 50 testing data compared with manual labeling by correspondents using confusion matrix. The confusion matrix divides the results of the predictions into four categories, they are; 1) true positive (TP) shows the amount of data with positive classes and true predictive results, 2) true negative (TN) shows the amount of data with negative classes and true predictive results, 3) false positive (FP) shows the number of data with positive classes and false, and 4) false negative prediction results (FN) shows the number of data with negative class and wrong prediction results. Prediction results are shown in Table 6.

The results of the confusion matrix are performed calculations to get the value of accuracy, precision and recall. Accuracy is the proportion of the total number of correct predictions compared to the total number of samples, accuracy is calculated using (2) [30]. Precision is the proportion of positive samples which are correctly classified against the positive total number of samples, precision is calculated using (3). A recall is a positive sample that is classified correctly to the total number of positive samples, recall is calculated using (4) [31].

... (2)

... (3)

... (4)

In (2-4) are applied to the test data testing results and the results are as shown in Table 7.

The measurement results in Table 7 show that labeling using KNN and TF-IDF with the NLP approach produces an accuracy 74.00%, precision 76.00% and recall 73.08%. Overall the test results showing positive results show machine learning has good performance. Previous research [9] using KNN and TF-IDF algorithms in classifying sentiments on Twitter social media obtained an accuracy of 67.2%, compared to this study obtained a higher accuracy of 74.00%. Higher accuracy values indicate that the use of NLP for preprocessing provides an increase in labeling accuracy.

3.4.Term occurrence frequency

Terms which has a high frequency of occurrence in conversation can describe market acceptance of service quality. The terms often used in conversations are indicated by wordcoud created with the "Wordart" application in Figure 4 and Figure 5. In conversations with the label "satisfied", the term which has the highest frequency of occurrence is the term "akan" in 143 times, term "ingin" in 98 times, term "apakah" in 78 times, term "baik" in 54 times, term "terima" in 52 times and term "kasih" in 51 times. Term "apakah" and "ingin" refer to customer questions and desires, while terms "baik", "terima"" and "kasih"" refers to customer satisfaction because questions or desires can be realized.

In conversations with the label "satisfied", the term that has the highest frequency of occurrence is the term "tidak-bisa" in 138 times, term "lama"" in 105 times, term "apa"" in 75 times, term "BPJS"" in 70 times and term "kira"" in 55 times. Term "apa"" and "tidak-bisa"" shows customers do not get what is expected. Term "lama"" and "kira"" shows that the customer is not comfortable waiting. Term "BPJS" shows the use of BPJS for payment of service fees.

4.CONCLUSION

Sentiment analysis on public service conversations can provide deeper information about customer ratings on services provided. Utilization of the speech recognition system technology can convert voice conversations into text, thus helping the process of analyzing customer satisfaction. Classification using the KNN and TF-IDF algorithm with optimization using the NLP approach at the preprocessing stage can result in better labeling of the conversation data. Testing with confusion matrix produces a value 74.00% of accuracy, 76.00% of precision and 73.08% of recall.

This research found that terms are dominant in conversation "satisfied" is term "akan", "ingin", "apakah" , "baik", "terima" and "kasih" which shows customer satisfaction on the service received and customer desires that can be realized. Whereas the dominant terms in the conversation "dissatisfied" is term "tidak-bisa", "lama", "apa", "BPJS' and "kira" shows customer disappointment at the service received. This research is a series of studies on the product "Kata Kita" to be applied in public services that function to record conversations in the form of text, labeling auto learning based on machine learning and digging information. Information extracted from service conversations can be used as material for evaluating and developing service quality. Further research can be carried out on other features of "Kata Kita" products or with a variety of datasets from different public service sources.

ACKNOWLEDGEMENTS

Researchers would like to thank the Magister of Informatics, Universitas AMIKOM Yogyakarta for providing support and guidance to this research.

Sidebar

Article Info

Article history:

Received Jul 22, 2020

Revised Aug 7, 2020

Accepted Sep 5, 2020

Corresponding Author:

Elik Hari Muktafin

Magister of Informatics

Universitas AMIKOM Yogyakarta

Ring Road Utara St., Ngringin, Condongcatur, District. Depok, Sleman Regency

Special Region of Yogyakarta 55281, Indonesia

Email: [email protected]

BIOGRAPHIES OF AUTHORS

Elik Hari Muktafln, bom in Sragen, Jawa Tengah, Indonesia, 2 September 1992. He holds a Bachelor of Informatics (S.Kom) from STMIK AMIKOM Yogyakarta in 2015. Currently the author is carrying out studies in the field of Magister of Informatics, Universitas AMIKOM Yogyakarta.

Email address: elik.muktafmfiîstudents.amikom.ac.id / elixropasfiîgmail.com

Pramono, born in Boyolali, Jawa Tengah, Indonesia 23 Juli 1986, he Holds a Bachelor of Informatics (S.Kom) From STMIK DUTA BANGSA Surakarta in 2012. Currently the author is carrying out studies in the field of Magister of Informatics, Universitas AMIKOM Yogyakarta.

Email address: [email protected] or [email protected]

Kusrini is an associate professor from Universitas AMIKOM Yogyakarta Indonesia. She finished her doctoral program from Universitas Gadjah Mada Yogyakarta Indonesia in 2010. She is interested in exploring many things about machine learning and other artificial intelligence field. She also loves in doing research about decision support system and database. She is member of the IEEE and IEEE Systems, Man, and Cybernetics Society

References

REFERENCES

[1] Barata, Atep, and Adya, "Dasar-dasar pelayanan prima," Jakarta: PT. Elexmedia Computindo, 2003.

[2] I. Ograjenšek and I. Gal, "The concept and assessment of customer satisfaction," Modern Analysis of Customer Surveys: With Applications Using R, pp. 107-127, 2011.

[3] D. Albury, "Fostering Innovation in Public Services," Public Money Manag., vol. 25, pp. 51-56, Feb. 2005.

[4] T. M. Hidayat, "Interest of Public Administration, Accelerated Improvement of Public Service Quality, (in Bahasa: Bunga Rampai Administrasi Publik, Akselerasi Peningkatan Kualitas Pelayanan Publik)," Bunga Rampai Administrasi Publik. pp. 51-63, 2016.

[5] M. Cui, T. Wang, Z. Pan, and L. Ni, "Definition of People with Impediments and Universality Evaluation of Public Service in Airport Travel Scenarios," International Conference on Human-Computer Interaction, pp. 623-639, 2020.

[6] Y. Wang, K. Kim, B. Lee, and H. Y. Youn, "Word clustering based on POS feature for efficient twitter sentiment analysis," Human-centric Comput. Inf. Sci., vol. 8, no. 1, p. 1-17, 2018.

[7] N. Shaikh and R. Deshmukh, "Speech Recognition System-A Review," IOSR J. Comput. Eng., vol. 18, pp. 1-9, 2016.

[8] A. A. Lutfi, A. E. Permanasari, and S. Fauziati, "Sentiment Analysis in the Sales Review of Indonesian Marketplace by Utilizing Support Vector Machine," J. Inf. Syst. Eng. Bus. Intell., vol. 4, no. 1, pp. 57-64, 2018.

[9] A. Deviyanto and M. D. R. Wahyudi, "Application of Sentiment Analysis to Twitter Users Using the K-Nearest Neighbor Method (in Bahasa: Penerapan Analisis Sentimen Pada Pengguna Twitter Menggunakan Metode K-Nearest Neighbor)," JISKA (Jurnal Inform. Sunan Kalijaga), vol. 3, no. 1, pp. 1-13, 2018.

[10] K. Norman, Z. Li, Y. T. Oh, G. Golwala, S. Sundaram, and J. Allebach, "Application of natural language processing to an online fashion marketplace," IS TInt. Symp. Electron. Imaging Sci. Technol., pp. 1-5, 2018.

[11] N. Kim and S. Hong, "Text mining for the evaluation of public services: the case of a public bike-sharing system," Serv. Bus., vol. 14, pp. 315-331, 2020.

[12] B. Cheekati and S. Padala, "Evaluating the Progressive Performance of Machine Learning Techniques on Ecommerce Data," Smart Computing and Informatics, pp.95-107, 2018.

[13] M. T. Khan, M. Durrani, A. Ali, I. Inayat, S. Khalid, and K. H. Khan, "Sentiment analysis and the complex natural language," Complex Adapt. Syst. Model., vol. 4, no. 1,2016.

[14] B. Iancu, "Evaluating Google Speech-to-Text API's Performance for Romanian e-Learning Resources," Inform. Econ., vol. 23, no. 1/2019, pp. 17-25, 2019.

[15] A. Solichin, "MySql 5: From Beginner to Advanced (in Bahasa: Dari Pemula Hingga Mahir)," Univ. Budi Luhur, Jakarta, Jan. 2010.

[16] M. Mhatre, D. Phondekar, P. Kadam, A. Chawathe, and K. Ghag, "Dimensionality reduction for sentiment analysis using pre-processing techniques," 2017 International Conference on Computing Methodologies and Communication (ICCMC), pp. 16-21, 2017.

[17] Z. Jianqiang and G. Xiaolin, "Comparison Research on Text Pre-processing Methods on Twitter Sentiment Analysis," IEEE Access, vol. 5, pp. 2870-2879, 2017.

[18] G. Chowdhary, "Natural Language Processing," Annual Review of Information Science and Technology, vol. 37, no. 1, pp. 51-89 2020.

[19] L. A. Mullen, K. Benoit, O. Keyes, D. Selivanov, and J. Arnold, "Fast, Consistent Tokenization of Natural Language Text," J. Open Source Softw., vol. 3, no. 23, p. 655, 2018, doi: 10.21105/joss.00655.

[20] A. N. Rohman, E. Utami, and S. Raharjo, "Deteksi Kondisi Emosi pada Media Sosial Menggunakan Pendekatan Leksikon dan Natural Language Processing," Eksplora Inform., vol. 9, no. 1, pp. 70-76, 2019.

[21] B. P. Pande and H. S. Dhami, "Application of Natural Language Processing Tools m Stemming," Int. J. Comput. Appl., vol. 27, no. 6, pp. 14-19, 2011.

[22] A. Prakash and U. Kumar, "Authentication Protocols and Techniques: A Survey," International Journal of Computer Sciences and Engineering, vol. 6, no. 6, pp. 1014-1020, 2019.

[23] I. Riadi, S. Sunardi, and P. Widiandana, "Mobile Forensics for Cyberbullying Detection using Term Frequency - Inverse Document Frequency (TF-IDF)," J. Ilm. Tek. Elektro Komput. dan Inform., vol. 5, pp. 68-76, 2019.

[24] R. Melita, V. Amrizal, H. Suseno, and T. Dirjam, "Application of Term Frequency Inverse Document Frequency (TfIdf) and Cosine Similarity Methods in Information Retrieval Systems to Find Web-Based Hadiths (Case Study: Hadits Sahih Bukhari-Muslim), (in Bahasa: Penerapan Metode Term Frequency Inverse Document Frequency (TfIdf) dan Cosine Similarity pada Sistem Temu Kembali Informasi untuk Mengetahui Syarah Hadits Berbasis Web (Studi Kasus: Hadits Shahih Bukhari-Muslim))," J. Tek. Inform., vol. 11, no. 2, pp. 149-164, 2018.

[25] V. Jahjah, R. Khoury, and L. Lamontagne, "Word Normalization Using Phonetic Signatures," Conference: Canadian Conference on Artificial Intelligence, 2016.

[26] J. Asian, H. E. Williams, and S. M. M. Tahaghoghi, "Stemming Indonesian," Conf. Res. Pract. Inf. Technol. Ser., vol. 38, no. January, pp. 307-314, 2005.

[27] M. Khader, et al., "The Effects of Natural Language Processing on Big Data Analysis: Sentiment Analysis Case Study," 2018 International Arab Conference on Information Technology (ACIT), pp. 1-7, 2018.

[28] E. Marthanty and I. Lestari, "The Culture of Apology at Work in Japanese Dramas: A Sociolinguistic Overview , (in Bahasa: Budaya Permintaan Maaf Di Tempat Kerja Dalam Drama Jepang: Tinjauan Sosiolinguistik)," Jurnal Bahasa, Sastra dan Budaya Jepang, vol. 8, no. 2, pp. 82-1002019.

[29] A. Tripathy, A. Agrawal, and S. K. Rath, "Classification of sentiment reviews using n-gram machine learning approach," Expert Syst. Appl., vol. 57, pp. 117-126, 2016.

[30] A. K. Santra and C. J. Christy, "Genetic Algorithm and Confusion Matrix for Document Clustering 1," International Journal of Computer Science, vol. 9, no. 1, pp. 322-328, 2012.

[31] A. Tharwat, "Classification Assessment Methods," Appl. Comput. Informatics, no. January, pp. 1-13, 2018.