A Survey of Recommendation Systems: Recommendation Models, Techniques, and Application Fields

Abstract

This paper reviews the research trends that link the advanced technical aspects of recommendation systems that are used in various service areas and the business aspects of these services. First, for a reliable analysis of recommendation models for recommendation systems, data mining technology, and related research by application service, more than 135 top-ranking articles and top-tier conferences published in Google Scholar between 2010 and 2021 were collected and reviewed. Based on this, studies on recommendation system models and the technology used in recommendation systems were systematized, and research trends by year were analyzed. In addition, the application service fields where recommendation systems were used were classified, and research on the recommendation system model and recommendation technique used in each field was analyzed. Furthermore, vast amounts of application service-related data used by recommendation systems were collected from 2010 to 2021 without taking the journal ranking into consideration and reviewed along with various recommendation system studies, as well as applied service field industry data. As a result of this study, it was found that the flow and quantitative growth of various detailed studies of recommendation systems interact with the business growth of the actual applied service field. While providing a comprehensive summary of recommendation systems, this study provides insight to many researchers interested in recommendation systems through the analysis of its various technologies and trends in the service field to which recommendation systems are applied.

1. Introduction

The development and spread of the Internet and smart devices have contributed significantly to an increase in the traffic of web, app, and SNS platforms. In addition, these platforms are increasing their collection of various information data that can identify users’ preferences. In particular, the active use of users’ SNS platforms makes it possible to acquire a wide range of data, such as information about followers related to the user, tweet data, and information uploaded by the user. In addition, the development of wearable sensors linked to smart devices facilitates the collection of various user-related exercise and bio-related medical data. Therefore, the Internet and smart devices have become environments in which various data about users can be collected. Not only explicit data directly provided by the user, such as likes and ratings, which were mainly used in the existing recommendation system, but also the user’s click data (click stream) and implicit visit information data representing the user’s behavior patterns, such as records, can be utilized in recommendation systems. Recently, by utilizing implicit data in a recommendation system, research on cognitive-based recommendation systems that analyze users’ personality or behavior to identify their preferences is increasing. This method offers the advantage of rapidly reflecting changes in users’ preferences in recommendation systems [1].

In order to utilize a user’s explicit data and implicit data for item recommendation, it is necessary to understand the user’s taste through various data mining techniques. Therefore, research on techniques to continuously refine the provision of insights for recommendations based on the past history of the item selected by the user, feedback on recommendation results, and correlation evaluation between users, etc., is ongoing [2,3,4,5,6,7]. Therefore, research to improve the performance of the recommendation system itself is being actively conducted and expanded. Based on the results of analyzed data, it is necessary to filter the various item information provided by the service to recommend items customized to users’ tastes or needs. The recommendation system model corresponds to this filtering method. In the field of research related to the recommendation system model, the limitations of the existing models were continuously supplemented and developed [8,9,10,11,12]. In other words, research on sophisticated recommendation systems has rapidly increased over the past decade, including data mining techniques and recommendation models, to broadly understand users’ tastes and provide more reasonable recommendations to users by using vast data collected in a multi-domain environment.

On the other hand, although recommendation systems already have a history of more than 29 years, the reason why it has been receiving particular attention recently is the video content streaming field, exemplified by Netflix, which is rapidly expanding its market. This service field recommends video content reflecting the user’s taste by analyzing a vast amount of image content, information data, user activity data, and user data similar to that of the user. The use of this recommendation system in the streaming field has increased user satisfaction. Therefore, Netflix is creating a consumption phenomenon that is replacing or exceeding the consumption of existing theater and TV-based media services. In addition, SNS platform-based services can collect various data, such as evaluation data and social data, and various recommendations are possible. In addition, SNS data are utilized in other service fields to provide services suitable for users’ tastes. This has not only helped to improve the performance of recommendations but has also led to many business changes in service fields, such as travel and e-commerce. In the service area that users directly experience, interest in recommendation systems is drawn from not only professional researchers, but also from general users themselves. The service areas in which recommendation systems are used have expanded, and studies have been performed to precisely match the characteristics of the service area. Therefore, the discovery of service areas to which recommendation systems a reapplied and the interest in representative services leading to it can be considered to have stemmed from an increasing trend in core research related to the recommendation system.

There are two approaches to data collection in this study. First, examining the research and trend of the recommendation model and recommendation technique. Second, for research on recommendation service area, high-reliability studies among various studies related to recommendation systems were selectively collected and analyzed. To this end, Google Scholar was used as the main search engine, and all the papers were collected by utilizing keywords such as Recommender System, Recommendation System, Content-Based Filtering, Collaborative Filtering, and Hybrid Recommender System since 2010. Furthermore, this study is limited to papers from journal databases such as ELSEVIER, IEEE, ACM, JSTOR, and Springer, and again limited to the top 100 journals. During the collection process, conference papers, Master’s doctoral papers, textbooks, unpublished papers, non-English papers, and news articles were excluded from the survey for this study. However, in the case of conference papers, papers from top-tier conferences such as NIPS, AAAI, ICML, WWW, KDD, IJCAI, etc. were included in the analysis target after collection, and, finally, research was performed based on about 100 analysis target papers. Through this, the research that links the theory of the recommendation system with practical services is structured with a focus on reliable papers. Figure 1 visually explains the brief process.

Next, for the trend analysis of the application field of the recommendation service, Google Scholar was used as the main search engine to collect all the papers that were searched for service application field and service recommendation with keywords since 2010. Subsequently, the number of published papers by year was arranged and the scale or value trend of representative services by service field was also indicated during the period, so that the connection between research and application could be reviewed together. Figure 2 visually explains the brief process.

Unlike the previous section, which included data from top-tier journals, the research for this section was conducted to provide insight based on the entire data field in order to analyze the flow of overall research and service fields. Through these two research approaches, we intend to increase the reliability of the major recommendation system-related contents and provide a stepping stone for setting the direction for future research by grasping the flow of a large amount of research.

2. Literature Surveys

Recommendation systems are a useful technology that can alleviate the problem of overload of information provided to users. It predicts the grade of items to be recommended to the user, creates a list of recommendation rankings for each user, and makes it possible to recommend items related to the user [13]. Several platform services actively recommend personalized items that meet the needs of users by introducing a recommendation system. In order to improve the performance of these recommendations, studies on various recommendation filtering models and data mining techniques are being conducted [14].

In this study, data mining techniques are analyzed for the recommendation model used in recommendation systems and its utility through research papers since 2010, and the overall flow of related research is analyzed. Figure 3 is a table that summarizes the recommendation model used in recommendation systems through research papers since 2010. Figure 4 is a figure that summarizes the overall flow of analysis of the recommendation model and recommendation technique. In addition, it is possible to understand the overall flow of the recommendation system field by classifying the main service application fields in which the recommendation system is used, and presenting the research in the related field and the main service and related research trends.

2.1. Recommendation Models

Since the first Collaborative Filtering model proposed in the 1990s was developed, recommendation systems have been actively studied, applied, and expanded in all fields of academia and industry until recently [15]. Recommendation systems are information filtering systems providing a personalized item recommendation to a user in a service environment that can hold or collect various data. Information filtering, which is mainly used in recommendation systems, is tailored to the user’s preferences or suggested only items judged to be useful to the user [16]. Iyengar et al. [17] stated that users prefer to have many options and, at the same time, when the difficulty of selection increases, the satisfaction of selection decreases. That is, in order to increase a user’s satisfaction with the service of the recommendation system, it is necessary to recommend various items to the user through the recommendation model to broaden the range of item selection for the user. At the same time, by analyzing the implicit and explicit data of the user and the data of a group of users similar to the user, the selection overload is reduced by drawing a list of items that match the habits and characteristics of the user. Figure 5 summarizes the general processes of the recommendation system models, such as Collaborative Filtering, Content-Based Filtering, and Hybrid System, at a glance.

2.1.1. Content-Based Filtering

In 1992, starting with the study of Loeb et al. [18], various models for filtering information emerged. Content-Based Filtering is a method for recommending items with attributes similar to those that users like, and recommends them based on the information of the items [19]. That is, it is a method of recommending similar items based on information on items selected by the user in the past. The Content-Based Filtering model is the most basic model within the overall recommendation system model and was mainly used in early recommendation systems.

However, as a result of the study by Salter et al. [20], the Content-Based filtering model recommends only data related to items closely related to items previously evaluated by the user, so the system is known for its limitation in not being able to recommend new items. That is, there is a disadvantage in that it does not provide the user with an opportunity to access various contents. Because of these limitations, this model has been mainly used in services that recommend items or text data items that are easy to recommend based on item information and user profile information. It was mainly used in various application areas, such as recommendations according to the properties of music [21], recommendations according to the properties of movies [20], e-commerce recommendations [22], and educational material recommendations [23]. The Content-Based Filtering Model uses text mining technology to identify user preferences, Semantic Analysis [24], TF-IDF (Term-Frequency Inverse Document Frequency [25], Neural Network [26], Naive Bayes, and SVM [19]. Since 2012, as research on the Hybrid Recommendation Model has increased, the use of Content-Based Filtering alone has gradually decreased, and its application area has also begun to shrink. Figure 6 is a visual summary of the item recommendation principle of the Content-Based Filtering Model.

2.1.2. Collaborative Filtering

Collaborative Filtering is an information filtering model that first appeared in the 1990s, and became a stepping stone for the subsequent research on recommender systems [27,28]. Collaborative Filtering is a model that constructs a user’s preference database using the user’s evaluation data to predict items that fit the user’s taste, and then uses it for recommendation [14]. This model can be classified into Memory-Based Collaborative Filtering and Model-Based Collaborative Filtering [29]. Memory-Based Collaborative Filtering can be further divided into User-Based Collaborative Filtering and Item-Based Collaborative Filtering.

User-Based Collaborative Filtering is a model that compares similarities between users by comparing the evaluation data for each user of the same item, and then creates and recommends a list of top N items that fit the taste based on the rating of each item of a similar user group. Item-Based Collaborative Filtering predicts an item by using the similarity between the item and the item selected by the user by creating a rating matrix of the user and the item. In other words, Memory-Based Collaborative Filtering uses technologies such as Pearson Correlation, Vector Cosine Correlation, and KNN to create similar groups (neighborhood groups) among users and recommend items to users within the same group [30,31].

As Memory-Based Collaborative Filtering was used in E-Commerce Services such as Amazon, the Collaborative Filtering recommendation model prevented customer churn and increased sales [32]. However, if the model does not include enough data, three problems can occur: sparsity, cold start, and gray sheep. First, the sparsity problem is a problem that occurs when there are not enough data available for recommendation [33]. Similarly, the cold start problem occurs when there are no evaluation data, that is, the first rater due to the influx of new users at the beginning of the service [34]. Finally, gray sheep is a problem in which recommendation difficulties arise when the set of users whose evaluation data are similar to that of the individual user is too small [35]. To solve this problem, Model-Based Collaborative Filtering, which estimates or learns a model for prediction using the data evaluated by the user, has been studied [32]. For Model-Based Collaborative Filtering, techniques such as Clustering, SVD, and PCA were mainly used.

The Collaborative Filtering model was adopted and used as a recommendation model more frequently than Content-Based Filtering. However, despite the development of collaborative filtering, the scalability problem and the sparsity problem [36] are not solved, so there is a limitation in that the accuracy of the recommendation is lowered. In order to overcome this limitation, a Hybrid System filtering model combined with Content-Based Filtering has recently been used a significant amount. Since 2010, the study related to Collaborative Filtering has mainly aimed at improving the performance of Collaborative Filtering, including a study aimed at developing a similarity calculation technique [37], a study aiming to expand the model by using customer feedback [38], a study aiming to obtain more user preference data using tagging [39], a study using clustering [40,41,42], etc. Figure 7 is a visual summary of the item recommendation principle of the Collaborative Filtering Model.

2.1.3. Hybrid System

Both filtering models feature limitations because the Content-Based Filtering model relies on metadata about the user’s item, and Collaborative Filtering relies on the user’s item rating data. A Hybrid recommendation model was proposed to solve the limitations of both recommendation filtering models and to improve the recommendation performance [43].

The Hybrid recommendation model is divided into seven types: Weighted Hybridization, Switching Hybridization, Cascade Hybridization, Mixed Hybridization, Feature-Combination, Feature-Augmentation, and Meta-level, according to the method that combines filtering techniques in Burks’ study [44]. Table 1 is a table summarizing the seven methods of the Hybrid recommendation model introduced in Burks’ study [44].

Since the Hybrid Recommendation model is mainly designed to solve the sparsity problem, the main goal of most studies dealing with the Hybrid recommendation model is to compensate for the lack of rating data by integrating the information of the Content-Based Filtering and Collaborative Filtering models. Hybrid Recommendation model-related studies include the following: a study on supplementing taste data by storing user evaluation data in a matrix using a Bayesian Probabilistic Matrix Factorization Framework, and learning side information data with insufficient user preference information using an auto-encoder [45]; a study using an automatic encoder that learns the nonlinear activity of users and items and removes stacked noise, which integrates information [46]; and a study on constructing a recommendation system model by introducing various items of side information in addition to the explicit rating accorded to items, such as the social connection between users and item information data [47]. These studies each attempted to alleviate the sparsity problem. Figure 8 is a visual summary of the item recommendation principle of the Hybrid Recommendation model.

2.1.4. Qualitative Evaluation Metrics of Recommendation Systems

In order to evaluate the performance of the Recommendation System model, it is necessary to quantify the characteristics of an excellent recommendation system [48]. In this study, the measurement criteria for evaluating the quality of models commonly used in recommendation system research were analyzed.

The simplest way to evaluate the performance of a recommendation system is RMSE (Root Mean Squared Error). RMSE is an index commonly used to evaluate prediction accuracy and is obtained by taking the square root of the mean squared error (MSE) calculated by dividing the sum of the squares of the difference between the actual grade and the predicted grade by the total number of grades that have been predicted [49].

Common qualitative evaluation indicators of recommendation systems include Precision [50,51], Recall [50,51], Accuracy [48], F-Measure [51], ROC Curve [49], AUC (Area Under the Curve) [52]. The confusion matrix is used to calculate the value of the qualitative evaluation index of this recommendation model. Table 2 is a confusion matrix that visualizes the predicted value and the actual value to measure the performance of recommendation systems. This matrix enables quantitative measurement by classifying whether the user’s preferred item is an item recommended by the recommendation system. Each row means an item that reflects the user’s preference, and each column indicates whether the recommendation model has recommended the corresponding item. The True Positives (TP) of the confusion matrix indicate the number of items that matched the user’s preference when the recommendation system recommends the item. True Negatives (TN) represent the number of items preferred by the user that are not recommended by the recommender system. False positives (FP) represent the number of cases in which the system recommends items that the user does not prefer. False Negatives (FN) represent the number of cases in which the system does not provide recommendations for items that users do not prefer. Here, the ratio of actual positives among all the positives is called TPR (True Positive Rate), and the ratio of false positives among all the negatives is called FPR (False Positive Rate) [53]. TPR is the same as the recall value.

A good performance of the recommendation system means that it recommends an item suited to the user’s taste. Furthermore, not recommending items that are not suited to the user’s taste is also part of a good recommendation system model. In general, Accuracy was used as an indicator to evaluate the performance of the recommendation model. Accuracy means the ratio of successful recommendations to all the recommended items. Accuracy can intuitively evaluate model performance, but it is difficult to accurately judge model performance when the information data on user preference items are unbalanced [54]. Therefore, to evaluate performance in more detail, Precision and Recall indicators are used. Precision is derived by calculating the proportion of items that match the user’s taste based on the item recommended by the model to the user. Recall is derived by calculating the ratio of items to be recommended by the recommendation model to the user based on the item selected by the actual user. Precision and Recall are trade-offs. Therefore, the F-measure value is sometimes derived when it is necessary to check the result of the integration of two indicators [55]. The F-measure is the calculated harmonic average of Precision and Recall. Precision, Recall, and F-measure possess values between 0 and 1, and the higher the value, the better the performance of the recommendation model.

The ROC curve is a graph showing the relationship between FPR and TPR. This method is mainly used to visually explain the ratio of the performance results of Precision and Recall. Since the ROC curve is a graph, it is difficult to obtain a quantitative value. To compensate for this problem, the AUC index is mainly used. AUC is the area below the ROC curve, and the accuracy of the recommendation model can be measured by calculating the area of AUC. As the AUC value is closer to 1, it can be evaluated that the performance of the model is excellent. In general, if the AUC value is 0.8 or higher, the model is judged to offer high accuracy [52]. Table 3 summarizes the formulas and definitions of general qualitative evaluation indicators that evaluate the performance of the recommendation system.

2.1.5. Research Trend of Recommendation Models

As shown in Figure 9 and Table 4, the point of time after 2010 surveyed in this paper is Web 3.0, which has been created since 2010 following Web 2.0 [56]. It is an ‘intelligent web’ in which the information data analyzed through semantic web and data mining are combined with machine learning and artificial intelligence. This evolved from Web 2.0, in which the spread of social network services such as Twitter and blogs connected people and created and shared various data based on these connections. By analyzing a variety of highly increased data, it is now possible to recommend items that take into account the user’s taste. Data collected from the web started to be used in the recommendation system in earnest after 2012 [57], and it has become a major cornerstone of the steady advance of the recommendation system. Figure 9 visualizes the development of the Web, and Table 4 compares the characteristics of each element according to the development of Web 1.0, Web 2.0, and Web 3.0.

Figure 10 visualizes the trend change of the papers that studied the three recommendation models among the papers collected according to the survey criteria set out in this paper, according to year.

Content-Based Filtering is the simplest recommendation model. In the early 2000s, there were many studies using it to present recommendations to users, but due to its disadvantage of recommending only biased items, the number of studies using this model alone has gradually decreased since 2010. However, it is still being studied and utilized continuously in the fields of books and news, which are application fields centered on text information.

Collaborative Filtering is the most frequently used and studied recommendation model, accounting for 41.6% of all papers. This is because the recommendation filtering model used in Amazon’s product recommendation system is highly effective and has been recognized as offering high industrial use value. Although there are limitations to Collaborative Filtering, studies have continued to overcome them. As shown in Figure 10, research on the Collaborative Filtering recommendation system model has decreased since 2014, and the number of Hybrid recommendation model studies has increased more than that of Collaborative Filtering, but it can be seen that research continues to be actively conducted.

2.2. Recommendation Techniques

Data mining is a technique for deriving useful information by discovering correlations and patterns between data based on data analysis in large datasets [58]. It analyzes the information of the item, makes it possible to recommend items similar to the item to the user, and creates a similar user group among users to identify the client/visitor click stream data matching the user group. It can also recommend customized browsing options to meet the needs of specific users [59]. Various data mining analysis techniques are used for such recommendations. Figure 11 is a visual summary of the techniques mainly used in the recommendation system to be described in this section. Furthermore, Figure 12 shows a typical data mining process.

2.2.1. Text Mining

Text mining is a technique for discovering useful text information by extracting text-related information from data. With the recent development of natural language processing technology, semantically important information has been extracted from the corresponding text [60]. When natural language processing is used in some text analysis processes, there is a tendency to analyze texts based on the frequency of words, so there is a limit to understanding semantics [61]. For this reason, in order to accurately grasp the meaning of the text, the ontology [62], which defines the common vocabulary of items and organizes the meaning by constructing the conceptual schema of the text domain, started to be used.

This text mining is used to recommend similar items by performing semantic analysis of item information in the Content-Based Filtering recommendation model [19]. In addition, the Collaborative Filtering recommendation model evaluates the semantic knowledge of information data between users, enabling item recommendation with similarity [62]. Figure 13 is a visual summary of a typical text mining process.

On the other hand, in order for a user to search for an item on the Internet, it is important to show an accurate semantic analysis result of the text, but a mechanism for identifying the context is also required. Therefore, in order to make communication between humans and computers useful, text mining technology is being advanced to context awareness [63]. As a text mining technique, fuzzy linguistic modeling (FLM) introduces fuzzy logic to natural language processing and analyzes the meaning of language using fuzzy subsets [64]. If FLM is used in the recommendation system, it is possible to identify the multilingual context of the item. In particular, when user preference is not clear or user preference data are insufficient, it offers the advantage of additionally acquiring insufficient preference data through a preference relationship after analyzing text data between items using FLM [65].

The text mining technique mainly used in the Content-Based Filtering recommendation model is Term Frequency–Inverse Document Frequency (TF-IDF), which gives weight according to the number of repetitions of a specific text. This technique expresses the components of the text of the document as vectors, and then the importance of a term is identified by calculating the relative frequency of a word in a specific document using the weight function TF-IDF [66]. Text mining techniques are mainly used in health care, education, tourism, and academic services.

2.2.2. KNN (K-Nearest Neighbor)

K-Nearest Neighbor (KNN) is an algorithm that classifies K-nearest neighbors of test tuple and train tuple to classify a dataset. KNN classifies datasets based on the closest distance by comparing the similarity between each item of data through distance-based weighting [67]. Euclidean distance, cosine similarity, and Pearson correlation are mainly used as measures to compare similarities. When the KNN algorithm is used in a recommendation system, the user’s search pattern can be classified and the user’s future preference can be predicted. After analyzing the patterns of user behavior data, such as users’ web server logs and clickstream data, it can be used to classify items similar to users’ tastes, and then use the results to recommend suitable items.

On the other hand, Jannach et al. [68] published a study result indicating that the performance of the recommendation model studied using the KNN algorithm was poor. Since KNN must select an appropriate value for K, which governs the performance of the model, there are inefficiencies that require repeated experiments and the problem of being biased by K [68]. In addition, the performance of KNN is degraded in analyzing data with a large input size. Therefore, when the size of the input data is large, dimensionality reduction that transforms the data into a meaningful expression should be used. The main goal of dimensionality reduction is to reduce data without loss of information, and principal component analysis (PCA) and linear discriminant analysis (LDA) are mainly used for this purpose.

2.2.3. Clustering

Clustering is an algorithm that identifies finite categories or clusters to describe data, and is widely used in recommendation systems because of its low redundancy and ambiguity [69]. There are many different types of clustering techniques used in the recommendation system, but K-means clustering is mainly used. K-means clustering is an algorithm that clusters around the mean after setting the number of K clusters. After calculating the similarity between all the data in the recommendation system, it is assigned to the nearest cluster and the calculation is repeated in the order of calculating the cluster center [70]. However, if the number of clusters is small, K-means clustering is vulnerable to the scalability problem, in which the calculation speed decreases when the number of users and items increases while the recommendation system is servicing [71].

On the other hand, Gong [72] performed user clustering and item clustering using a similar inter-user clustering technique based on user ratings for items and conducted a study to recommend personalized recommendation items. As a result of studying the method of finding a cluster group similar to the recommended target user based on the similarity between items in this study, the problems of scalability and scarcity, which are the disadvantages of the existing Collaborative Filtering approach, were solved.

Clustering was mainly used in the Collaborative Filtering recommendation model and was generally studied in the recommendation system in the tourism, education, and e-commerce fields.

2.2.4. Matrix Factorization

Matrix Factorization recommendation system became widely known through the Netflix Prize, and especially solves the problem of scarcity in Collaborative Filtering [73]. Matrix Factorization is a method to characterize items and user data after inferring elements from user evaluation data for items and storing them as vectors [74]. The main purpose of this technique is to find the dimension of the latent factor that expresses the user information and the user’s preference by storing the user’s evaluation data in the Rating Matrix [75]. In particular, it offers good scalability and flexibility as it can analyze users’ tastes by using data such as mouse movement and search pattern, as well as explicit data that the user directly evaluates for a specific item [74].

Singular value decomposition (SVD) is a method that provides predictive data to customers by analyzing various item data to transform the user and selected items into a space of the same latent factor [76]. The algorithm offers the advantage of increasing the efficiency of the recommendation system by reducing computational overload. In the existing recommendation system, it was difficult to identify user preferences with tag and like data in the SNS field because it was possible for multiple users to make recommendations based on numerical evaluation of common items [77]. However, Matrix Factorization is widely used because it makes it possible to provide recommendations using information collected in various ways in the SNS field [4,78,79,80].

2.2.5. Neural Network

In recent years, the use of neural networks has expanded in various fields, such as speech recognition, image recognition, photo search, and language translation. On the other hand, although the introduction and use of neural networks in the recommendation system field is relatively small compared to other fields, many studies are being conducted as one of the main areas of interest in recommendation-system-related research. It has been widely used in research to obtain additional data in situations where it is difficult to understand user preferences with historical data [81,82]. In addition, He et al. [83] modeled noisy implicit feedback data using a deep neural network (DNN) to improve the performance of a recommendation system; deep learning offers the potential to improve the performance of recommendation systems. In other words, in developing a recommendation system, neural networks are used for modeling research to additionally secure and supplement data to solve the problem of sparsity and cold start in Collaborative Filtering, or for the purpose of improving the performance of the recommendation system itself.

2.2.6. Research Trends of Recommendation System Techniques

Figure 14 and Figure 15 are visualizations that can be used to analyze the research trends of techniques used in the recommendation system field. Figure 14 visualizes the number of recommended techniques used in the papers analyzed according to the survey collection criteria set in this paper. Furthermore, Figure 15 visualizes the utilization of each recommended technique according to the flow during the year.

Text Mining is a technology necessary to analyze the characteristics of the items selected by the user in Collaborative Filtering models, such as the Item-Based Collaborative Filtering model of the Content-Based Filtering Recommendation model and the Hybrid Recommendation model. This technique can be used in various recommendation system models and is continuously required in areas such as medical data (health care field), academics, and tourism, that contain a significant amount of text information. From Figure 14, it can be seen that text mining is a technique that is actively used in recommendation system research. Furthermore, through Figure 15, it can be confirmed that Text Mining is continuously used in research related to recommendation systems.

On the other hand, KNN’s performance was the least utilized in the recommendation system research collected in this paper after 2010 due to the inefficient search process for K values, the bias problem of K, and the problem that it cannot be used when the data size is large [68].

Clustering is a technique mainly used to identify user groups similar to users in the Collaborative Filtering model. When clustering is used for Content-Based Filtering and in the Hybrid Recommendation model, it is mainly used when clustering and analyzing the similarity of location-based data in the recommendation system field of the travel field. Clustering is useful when analyzing similar groups or similar items when the evaluation data is not a number. However, currently, in various apps and web services, the frequency of using clustering as a recommendation technique is decreasing because users evaluate items using ’likes’, ’star ratings’, or numerical data. However, from Figure 15, it can be seen that clustering is used consistently by year.

MF is a technology that searches for potential factors that express the user’s preference for the items provided by the service, and it is possible to analyze not only numerical data for specific items, but also data collected through external situation data, mouse streams, etc. The calculation time can also be reduced because it is calculated by decomposing the matrix consisting of user and item evaluations. Therefore, the number of studies using MF than KNN has increased because the existing KNN problems have been supplemented. However, as the number of businesses using large-scale servers increases, the need for a new recommendation system utilization technology to build a model that can calculate more than MF technology in order to derive results faster and more accurately than before increases in turn.

The development of smartphones, wearable devices, and SNS has enabled the collection of various data related to users. In other words, even if the user does not input evaluation data, body data that is naturally generated while the user is active, posts, keywords, and images uploaded by the user to SNS are all data that can be utilized in the recommendation system. Neural Network technology began to be used to analyze these various types of data in recommendation system research. In particular, Neural Network technology specializing in image analysis and image prediction can more accurately and quickly analyze an image uploaded by a user or an item purchased by a user. Clustering technology can also be used to analyze a map of a user’s favorite travel destination and recommend a user’s favorite destination among nearby places, but a neural network can be used to recommend a user’s preferred travel destination by learning various features of the user’s preferred travel destination. Therefore, the Neural Network can analyze the characteristics of various items that the researcher has not thought of before; in Figure 14 and Figure 15, it can be seen that this technology was the most commonly used technology in the 10 years of recommendation system research.

2.3. Application Fields

The recommendation system has been expanded and used in various service fields. In this study, we intend to analyze how the recommendation models and technologies for various recommendation systems described above are studied and utilized according to the characteristics and purpose of the actual service field. Based on the papers collected for analysis, the service fields in which the recommendation system was used were classified into seven main categories: Streaming Service, Social Network Service, Tourism Service, E-Commerce Service, Healthcare Service, Education Service, Academic Information Service. The seven main categories are divided based on the list of services that use a recommendation system with increasing users or increasing business value, and the list of services that appear frequently when ’Recommendation System’ of the Google Scholar search engine is searched as a keyword. Figure 16 is a visual summary of the list of services mainly used in the recommendation system to be described in this section.

2.3.1. Streaming Service

In the past, video content, such as movies, were mainly consumed by users through TV or movie theaters. However, recently, a significant amount of video content is consumed through streaming platforms such as Netflix and YouTube. Audio content is also changing from downloading and consuming files to a user’s local device to consuming content through streaming platforms such as Spotify. Streaming services related to media content have been developed along with the recommendation system because it is necessary to reduce users’ worries about choosing a vast amount of content and to provide content that is tailored to each user. In general, in the streaming service field, user preference data is collected centering on the user’s media content service usage history data, and after mapping user preference with all the content owned by the streaming service, recommendations are generated in the order of the content most similar to the user’s preference. [36,84].

Until the early 2000s, in the streaming service field, the Content-Based Filtering model was widely used in recommendation systems. However, in the Web 2.0 era, due to the inefficiency of providing only some content to users in a streaming service that features large amounts of rich content data, ’research on the usefulness of the Collaborative Filtering method that recommends items in consideration of the similarity between users’ was mainly conducted [85,86]. On the other hand, the biggest feature of streaming services is that information is categorized by genre, artist, and cast of items [87]. Due to these characteristics, studies using text mining techniques were mainly used in the study of recommending streaming services. In the study by Odić et al. [88], Text Mining was performed on the properties of users’ video viewing item list in order to recommend content desired by the user. In Text Mining, ontology was used to not only find movie items desired by users, but also to recommended items considering users’ situational information.

Recently, the number of streaming service users has exponentially increased. Therefore, to ensure smooth service, it is necessary to reduce the overload of calculating user preference analysis. In order to solve this problem, studies have been conducted to improve various recommendation techniques and recommendation filtering models for streaming services. Barragáns-Martínez et al. [36] used the Singular Value Decomposition (SVD) algorithm to reduce computational overload when calculating user preferences to alleviate major problems such as limitations in scalability and data sparsity. Through this, a Hybrid Recommendation model that recommends TV programs with higher user preference was proposed. As a result of evaluating the performance of the proposed model, 0.78 was derived from MAE, and as a result of additional user testing, it was claimed that positive feedback was received. A study by Walek et al. [9] proposed a Hybrid recommendation model that recommends movies that fit user preferences using the SVD algorithm and Fuzzy Logic, based on user’s preferred movie genres and movie ratings. As a result, promising values for Precision (81%), Recall (83%), and F-measure (82%) were achieved.

Since the music streaming service field exerts a greater influence on user preference compared to other content, high predictive power is required for user preferences [89]. To this end, studies on a Content-Based Filtering recommendation model considering user data and audio content data [80,90] and on a Hybrid recommendation model study that combines a Collaborative Filtering model that considers evaluation data of other users similar to the user [86,90,91,92,93] were performed. On the other hand, in the field of music streaming service, the study of music content recommendation was mainly conducted in consideration of audio characteristics such as genre, melody, and rhythm that reflect the user’s taste [94]. Wang et al. [89] analyzed the item properties of music using Text Mining, extracted audio signal features such as the rhythm and melody of music data, and recommended items. This study proposed a Hybrid recommendation model that recommends music that users prefer using a Neural Network. As a result of comparing the RMSE values of the Hybrid model proposed in this paper and the general Content-Based Filtering model, the RMSE of the Hybrid model was 0.255 and the result of the Content-Based Filtering RMSE 0.270 was derived. Therefore, it could be confirmed that the Hybrid model proposed in this paper showed better recommendation performance. McFee et al. [87] proposed a study to analyze audio similarity through content analysis of data samples preferred by similar user groups obtained through Collaborative Filtering. Through the similarity analysis results for each audio signal of the optimized audio content, a Hybrid recommendation model that enables the recommendation of various lesser-known music was proposed. In other words, this study proved the expandability of the recommendation system for music streaming service by solving the weakness of Content-Based Filtering, the narrow range of user preference items.

Table 5 summarizes the research related to the recommendation system used in the streaming service field among the papers collected according to the research criteria of this study.

2.3.2. Social Network Service

Online social network services (SNS) such as Facebook, Instagram, Twitter, and LinkedIn are huge digital-based social exchanges where users can not only lifelog their daily life, hobbies, interests, etc., but also provide a field of interaction with other users [97]. The vast increase in the use of SNS was also accompanied by a vast increase in user-related data.

It is possible to collect content information that users register with posts through SNS. In addition, user evaluation data can be collected; as well as rating data, these include various types of feedback data, such as likes and comments. The collected data are not only used for recommendations within SNS, but are also open to utilization in recommendation systems for other businesses. In other words, various data collected through SNS f are closely related with the advancement of recommendation systems.

Since SNS is connected to various users who are not friends, the data of other users who are similar to the user can also be used for analysis to produce recommendations. Therefore, it is easy to use Collaborative Filtering and Hybrid recommendation models [98]. Figure 17 shows the structure of the concept and principle of the Hybrid recommendation model mainly used in the SNS recommendation system.

Collaborative filtering is mainly used to recommend friends or items based on the behavioral patterns of other users who possess a friend list similar to the user on SNS [3]. However, the vulnerabilities of Collaborative Filtering [99] such as cold start, sparsity, and gray sheep need to be addressed. For this purpose, research on SNS recommendation system using Model-Based Collaborative Filtering model [4,78] has been conducted. A number of studies have been conducted to supplement the weaknesses of Collaborative Filtering by using various techniques along with the Recommendation Filtering model. For example, Kim et al. [78] proposed TWILITE, a Twitter follower recommendation system, and a tweet message recommendation system suitable for users’ preferences. This study designed a follower recommendation system using Matrix Factorization with a Collaborative Filtering model. In addition, the weakness of the Collaborative Filtering model was addressed by recommending a tweet message suitable for user preference using a probabilistic model based on LDA (Latent Dirichlet allocation). As a result of empirically evaluating and comparing the performance of TWILITE’s recommendation algorithm with other algorithms, the proposed model showed higher values than other algorithms in terms of Recall, Precision, and average hit-rank index, demonstrating its excellent performance. Similarly, Yang et al. [4] also studied the Collaborative Filtering model and Matrix Factorization Technique for the SNS recommendation system. However, the difference with the study by Kim et al. [78] is that users with the highest trust relationship with the target user is defined as users with a similar taste through the proposed Trust model, thereby overcoming the weakness of the model. In order to evaluate the performance of the proposed model, this paper compared the results of applying the proposed model to four large data sets and the results of applying other recommendation algorithms, PMF, SoRec, RSTE, and SocialMF, to the same data set. The experimental results were derived as the performance indicators, Precision, Recall, F1-score, and NDCG (Normalized Discounted Cumulative Gain). As a result, all of the performance indicators of the algorithm proposed in this paper were excellent. This model showed superior performance compared to other algorithms, especially when an item was recommended to cold-start users.

In the field of SNS, a number of studies on Hybrid recommendation models have been conducted. Amato et al. [5] proposed a Hybrid recommendation model that can provide recommendations based on user interaction and generated multimedia content on multiple SNSs. Through Text Mining, we analyze user preference data, text comments that can relate user emotions, behavioral data such as past logs performed by users, and user evaluation data for various contents. As a result of the analysis, similar items were clustered to provide user-centered recommendations for social networks. In the study by Capdevila et al. [6], the user’s preference was also analyzed using the user’s geographic location and the full text data of the SNS using the Text Mining technique. A personalized item was recommended to the user according to the analysis result. Furthermore, in this study, a Hybrid recommendation model that can recommend even a location to a user was proposed.

As a result of evaluating the performance of the Hybrid model proposed in this paper through the AUC value, in the recommendation consisting of a user and item pair, 0.6566, in the non-paired recommendation, it was 0.6044, which showed better recommendation performance than the existing CB and CF models. Table 6 summarizes the research related to the recommendation system used in the social network service field among the papers collected according to the research criteria of this study.

2.3.3. Tourism Service

As the demand for travel has increased, recommendation systems have begun to be used in the tourism service field to recommend tourist destinations, route recommendations, and transportation methods. As the travel-related recommendation system uses situational data, such as review data and location data, user location, time, and weather, collected through SNS, research on recommendation systems using SNS has increased in the tourism service field.

SNS stores the user’s check-in data and the location of the post uploaded by the user, and tourism services can use it as a dataset for recommending tourist attractions and routes [103,104]. The travel recommendation system analyzes these SNS data and provides travel information suited to the user’s taste, thereby increasing user satisfaction and increasing loyalty in tourism, encouraging users to return to tourist destinations after their trip. Kesorn et al. [103] proposed a Personalized Tourism Information Service (PTIS) framework that recommends tourist destinations customized to users based on their Facebook check-in data analysis. PTIS utilizes a Hybrid recommendation model to recommend tourist attractions to users, as well as users’ Facebook friend check-in data. The model continuously updates the user’s profile by utilizing data that can identify preferences that users generate while using the service, such as history and evaluation data. Therefore, as the number of PTIS users increases, it is possible to improve the recommendation performance through massive and varied data. In the recommendation model using popular tourist destination data, Facebook close friend data, and appropriate visit time data proposed in this paper, Precision yielded an average of 87.75%. In addition, Sun et al. [104] performed spatial clustering based on the geo-tagged data of photos uploaded to Flickr, a photo-sharing SNS, to identify important tourist destinations; the rankings were then sorted in ascending order, and routes were recommended to users by considering both the popularity of the tourist destinations and the road length data through machine learning. An empirical evaluation was performed to evaluate the performance of the model proposed in this paper. The tourism route recommended by the proposed model was less than twice as short as the shortest route considering only the length of the road. Through this, it was shown that optimal tourist routes can be recommended to users by considering the image taken on the road, the point of interest (POI), and the length of the road. That is, the study made it possible to recommend an optimal travel route that reflects user preference. Figure 18 visualizes the recommendation principle in this recommendation system to provide overall popular tourist destinations and optimal routes.

On the other hand, there are studies that utilize not only the evaluation data of tourists but also the current situation data of users in the recommendation system model for recommendation in the tourism service field [2,105,106]. Representatively, Abbasi-Moud et al. [2] proposed a context-aware tourism recommendation system by deriving context data from users’ review data on tourist destinations. This study derived situation data including time, location, and weather data, from tourist destinations using Text Mining and performed text sentiment analysis and semantic clustering of review data to determine user preferences. As a result of comparing the proposed Hybrid model with similar systems proposed in other studies, the proposed model showed the highest values in Precision, Recall, and F-Measure indicators. In particular, the Precision value increased compared to previous studies because, first, by analyzing sentiment, the user’s disliked words were filtered from the default setting list and, second, a tourist destination adjusted to the current situation was recommended by considering several contextual factors. Thus, it is possible to recommend a preferred tourist destination according to the user’s situation.

Smirnov et al. [105] proposed the TAIST (Tourist Assistant Infomobility System), a recommendation system using a collaborative filtering model. In this study, the user’s preference was identified using the evaluation data of tourists, and a list of tourist destinations was created and then recommended. Next, based on the ontology-based situation data for recommended tourist destinations, the optimal travel route considering the accessibility of tourists and recommendations for public transportation were provided. As a result of user evaluations by 1000 tourists accessing information on cultural heritage, the proposed system has proven its superiority through its ability to recommend cultural heritage sites in a timely manner and support tourists during travel.

Table 7 summarizes the research related to the recommendation system used in the tourism service field among the papers collected according to the research criteria of this study.

2.3.4. E-Commerce Service

In the past, items such as clothes, food, and books were mainly consumed by users through offline stores. However, in recent years, with the development of digital platforms such as the web and applications, the form of consuming items has changed through e-commerce platforms such as Amazon, eBay, and Alibaba. E-commerce offers consumers many items and various options in the online environment and provides sellers with an easy way to sell. In particular, consumers were unable to go outside due to the lockdown measures due to COVID-19, and as a result, they were unable to use offline stores. Consequently, consumption using digital platforms increased exponentially. The categories of items sold on digital platforms also began to diversify [112].

The E-commerce service collects data related to various users for business expansion and actively utilizes these data in a recommendation system. The service predicts user preference by analyzing auxiliary user information, such as gender and age group. Based on users’ preferences, it is used for item recommendation [113]. In addition, research is underway to collect shared opinions on reviews or items that reflect the subjective opinions of users through the Virtual Community (VC) provided by each commerce and utilize the recommendation system for it [114]. Currently, tracking data is generated by tracing the actions of the mouse and keyboard when the user uses the service, and this data is used in the recommendation system. Mouse and keyboard data can analyze user preferences by tracking user interactions with browsers and apps to determine purchase intentions [115].

Therefore, by using the data collected from the VC and the recommendation system, it recommends the item that reflects the user’s preference. Furthermore, it is possible to recommend an item to a user using the taste information of a group of other users with similar preferences to the user [116].

The most significant characteristic of the e-commerce service is that consumers usually show a pattern of spending that complements the items they have previously preferred or have already purchased. Therefore, finding items similar to items previously purchased by the user is helpful in recommending items suitable for the user [117]. For this reason, the Collaborative Filtering recommendation model and the Hybrid recommendation model are mainly used in the service [72,118].

In a study using a Collaborative Filtering recommendation model, Gong [72] derived the most similar neighboring users among users through user clustering technology using the users’ purchased item data. As a result, a group of neighboring users similar to the user and a collaborative filtering system that used this selected item clustering technique to recommend appropriate items to users were suggested. In an experiment comparing the proposed CF with the existing CF, the number of neighbors had a significant effect on the recommendation quality. As a result of calculating the MAE by increasing the number of neighbors from 20 to 50, the proposed algorithm showed a lower MAE value as the number of neighbors increased, and the gap to the existing CF model widened, demonstrating its excellent performance. Hwangbo et al. [118] extracted customer preferences by using preference data from customers who purchased offline items and click data (clickstream) for the same online items as offline items. Because the study adds offline data, it can prevent cold start, improving recommendation performance. As a result of an experiment comparing the proposed recommendation system, K-RecSys, with the recommendation system used in shopping mall sites, in the case of the existing system, 5.8% of all clicks occurred, in the case of K-RecSys, 9.9% of all clicks occurred from recommendations. In addition, the purchase rate through K-RecSys’ recommendation was 12.3%, while in the existing system it was only 8.9%, proving that the proposed system recommends items that are suited to users’ preferences.

However, as the vulnerability of Collaborative Filtering began to be mentioned [119], the E-commerce service field began to utilize user assistance information to compensate for this problem. In particular, a number of Hybrid recommendation models were used that recommended using all of the review, comment, and user preference analysis data collected from VC. A study by Zhang et al. [113] proposed a ’Hybrid Probabilistic Matrix Factorization’ model-based recommendation system that models user preferences from user auxiliary information and items through a Neural Network, and distinguishes keywords from text information about items. This approach simultaneously considers users’ unique preferences and emotional tendencies to predict users’ ratings of items. DUPIA is a model that is proposed through the RMSE value, and performance evaluation and comparison results for MovieLens (ML), AIV data sets of PMF, CTR, CDL, ConvMF, ConvMF models; DUPIA achieved the best performance and showed improvement of 1.41% in ML100K, 1.16% in ML1M, 1.35% in ML10M, and 11.87% in AIV compared with the second-best model. In particular, DUPIA performs very well on the sparse data set AIV, with a performance improvement of about 12%, indicating that the proposed approach can be very effective in cold start situations. This is because user groups with different user preferences and item characteristics can be separated more precisely. Guo et al. [120] proposed a system to recommend products suitable for consumers by fusing neural networks and multi-source information to determine the weight of recommended items and analyzing the requirements of mobile e-commerce consumers. As a result of comparing the proposed algorithm with the existing traditional algorithm, the accuracy of the existing algorithm was 80.01% and that of the proposed algorithm was 91.23%, using multi-source data on consumer behavior to more accurately analyze consumer needs. In addition, the coverage rate continuously increased until it reached or approached a peak when user information was continuously reinforced. This can be a strong advantage, especially when the number of consumers skyrockets due to advertising, promotions, and other activities.

Research related to the recommendation system in the E-commerce field is increasing rapidly every year. In addition, recently, research on advanced recommendations related to the interface of the digital platform has been conducted. As the number of items advertised in E-commerce increases, the user’s level of interest gradually decreases. Therefore, research related to arranging an effective interface to increase user interest is being conducted. In Sulikowski et al.’s study [121], the Evaluation of a Recommending Interface (PERI) framework was implemented and similar performances were shown, regardless of the vertical and horizontal directions of the interface, for a fixed period of time. However, in terms of purchase commitments, the recommendation performance of the vertical layout increased by more than double. In addition, by providing a vertical arrangement and a visual effect of slowly blinking, the user’s interest in the item was further elicited.

Figure 19 visualizes the recommendation principle of the system that developed the existing Item-Based Collaborative Filtering recommendation model by combining offline customer preference data and online customer preference data. Table 8 summarizes the research related to the recommendation system used in the e-commerce service field among the papers collected according to the research criteria of this study.

2.3.5. Healthcare Service

As interest in health increases, the number of users who use smart wearable devices has started to increase as the technology has become compatible with smartphones, and their convenience of use has also increased [123]. Such wearable devices can efficiently monitor the user’s bio-state [123]. Smart Watch, a representative wearable device, regularly measures the user’s body data [124], helps users who do not possess specialized medical knowledge to prevent diseases, and enables self-diagnosis. These wearable devices collect a vast amount of user biometric data to help with disease-related research or appropriate diagnosis through specific body situations [125] and, furthermore, it has been helpful in research that recommends treatment [126]. Health-related recommendation system studies analyzed the relationship between patients’ symptom patterns and diseases to provide users with insight into better treatment options [127,128]. In this study, the recommendation system used in the healthcare service field was subdivided into the health recommendation system field and the e-health field, which support professional treatment according to the purpose of the system application.

In the field of health recommendation systems that help users with professional treatment, the main goal is to provide suitable treatment methods according to the symptoms of various types of diseases and the stages of each disease. To this end, the health recommendation system analyzes the patient’s information and the characteristics of the disease, offers an accurate diagnosis of the disease to the patient, and recommends an appropriate treatment according to the diagnosed disease. To this end, the Content-Based Filtering model is frequently used because it is necessary to analyze the patient’s information and the characteristics of the patient’s disease. Duan et al.’s [129] research is representative as a recommendation system study using the Content-Based Filtering model. In this study, Text Mining was conducted by creating a tree structure of prefixes of the item dataset to recommend treatment plans. Therefore, a study was conducted to recommend a patient care management plan that provides clinical decision support, nursing education, and impression quality management by composing an item ranking list. A study by Chen et al. [130] divided a patient’s disease into stages, and according to each stage, a disease diagnosis and treatment recommendation system (DDTRS, Disease Diagnosis and Treatment Recommendation System) that recommends an accurate disease diagnosis and treatment plan was proposed. The DDTRS clustered large historical disease diagnosis and treatment data sets extracted from patients’ examination reports. Furthermore, by applying the association analysis algorithm, the similarity of the content information of the disease, diagnosis and treatment data similar to the user was searched, and an accurate disease diagnosis and effective treatment plan were recommended. Five factors, effectiveness, chronergy, non-harmful side-effects, economy, and patient satisfaction, were utilized as indictors for the evaluation of the quality of treatment recommendations provided by DDTRS. In terms of indicators, evaluation was performed through physician feedback. The effectiveness index was the highest, with an average value of 4.33, compared to the other four indexes. The economic index showed the lowest ranking, at 3.24. Since this generally means that a high-quality treatment recommendation has a high therapeutic effect, a large chronological effect, and few side effects, the experimental results demonstrated that the proposed system provides high-quality treatment recommendations.

Figure 20 shows the workflow of the disease–symptom clustering analysis module and the disease diagnosis and treatment recommendation module, the two core modules constituting the DDTRS proposed by Chen et al. [130]. In the field of professional medical treatment, it contains the main process of analyzing the symptoms of a patient’s disease through historical treatment datasets and recommending an appropriate treatment.

In addition, the field of health recommendations can suggest similar treatment modalities to other patients with similar symptoms to the patient. The Collaborative Filtering model is frequently used in studies of health recommendation systems that recommend other patient groups with symptoms similar to those of the patient. The study by Thong et al. [131] derived relational data after calculating the similarity between patients using the fuzzy clustering technique. Based on the relational data derived in this way, a medical diagnosis recommendation system that classifies patients into groups according to disease characteristics was recommended. In this study, after finding a patient group with similar disease characteristics to the diagnosed patient, the Collaborative Filtering recommendation model was used to diagnose the patient by checking the medical diagnosis record of the corresponding patient group. As a result of evaluating and comparing the performance of the proposed algorithm HIFCF with the IFCF, DAVIS, HASAN, DE, Samuel, SZMIDT algorithms, and MAE values, the average MAE value of HIFCF was 0.395, which was lower than that of the other algorithms in all cases. The average MAE value of the second lowest DE algorithm was 0.481, suggesting a gap. Therefore, the accuracy of the proposed recommendation algorithm was verified.

In the field of specialized treatment, in addition to diagnosing a patient’s disease and providing treatment based on professional medical knowledge, studies have recently been conducted on recommending an appropriate diet in response to patient information such as disease, age, gender and weight, individual nutritional expectations, and food preference [132,133,134]. The study by Iwendi et al. [133] proposed deep learning solutions that use various Neural Network techniques, such as Logistic Regression and Recurrent Neural Network, to share information among patients with the same disease and recommend which kind of diet they should follow. A performance evaluation was performed on various Neural Network techniques, such as MLP, GRU, LSTM, RNN, Naive Bayes, and Logistic Regression. As a result, in the case of the class in which the LSTM was allowed, the precision was 98%; and in both the recall and F-Measure, the value was 99%. For the disallowed class, the values were: Precision, 89%; Recall, 73%; and F-Measure, 80%. Overall, the model showed an excellent performance. In particular, it showed an excellent performance for the disallowed class. Other models performed well for the permitted class but did not perform well for the disallowed class. On the other hand, LSTM outperformed all the other models in all the mentioned indicators and a performance evaluation was performed on various neural network techniques, such as MLP, GRU, LSTM, RNN, Naive Bayes, and Logistic Regression. As a result, in the case of the class in which the LSTM was allowed, the precision was 98%, and in both Recall and F-Measure, the value was 99%. For the disallowed class, the values were: Precision, 89%; Recall, 73%; and F-Measure, 80%. Overall, the model showed an excellent performance. In particular, it showed an excellent performance for the disallowed class. The other models performed well for the permitted class, but did not perform well for the disallowed class. On the other hand, LSTM outperformed all the other models in all the mentioned indicators and produced good results for both allowed and disallowed classes.

On the other hand, in the field of e-health services, health data customized to the user are used to help the user independently control their health data [135]. In the field of e-health services, we provide a variety of content containing the health data needed by users without having to go through a specialist, and we focus on enabling users to access personalized health information as well as everyday health information at any time and anywhere. Therefore, in this field, the recommendation system based on the Content-Based model that provides personalized health contents based on the user’s health data has mainly been used [136,137,138]. Wiesner et al. [137] provided individualized context-aware health information to users by integrating a Content-Based recommendation model with personal health data from the Personal Health Record System, and it used semantic network and ontology techniques to propose an approach that allows non-expert general users to easily manage their health data. A study by Sanchez Bocanegra et al. [136] suggested a recommendation system utilizing Text Mining ontology techniques. High-quality content is selected from health video content posted on YouTube; it is then connected to a health education website and provides consumers with the health information they need. To evaluate the recommendation performance of the proposed system, Precision and NDCG indicators were used for recommendation links to video content on general medicine, diabetes, and hypertension. As a result, the precision values were 0.5–0.87 for the over-the-counter video and 0.68–0.89 for the diabetes video, and the NDCG values were 0.5–0.88 for the over-the-counter video and 0.72–0.90 for the diabetes video. These values show that the recommendation performance was generally higher when recommending over-the-counter drugs and diabetes images, compared with the Precision and NDCG values of 0.39–0.62 and 0.46–0.65 for the hypertension video, respectively. In addition, through the proposed system, we demonstrated the feasibility of providing an educational health website as an additional explanation for a given health video with a semantic-based recommendation algorithm. Table 9 summarizes the research related to the recommendation system used in the health care service field among the papers collected according to the research criteria of this study.

2.3.6. Education Service

From the traditional form of education in the classroom or lecture hall, a new education trend, called Smart Learning, has formed through e-learning, in which learning is conducted through an online environment [140]. Smart education has started to be gradually used in education due to the increase in the spread of various smart devices and the development of wireless networks. Smart education can access vast digital resources and seamlessly provide personalized learning tailored to the needs, goals, talents, and interests of learners without time and space constraints. In addition, the educational form has improved by reflecting the learning trend of the digital age [141,142]. Therefore, the field of education services using the recommendation system provides learning resources in consideration of the learning style and knowledge level of learners, thus providing an effective and efficient learning experience. In other words, personalized learning content can be provided to learners.

In the study of recommending learning content suitable for learners with a focus on the similarity between learners and learning objects, the Content-Based Filtering recommendation model was mainly used after analyzing the learner’s profile information and learning object information [143,144,145]. A study by Shu et al. [144] utilized a Content-Based Filtering recommendation model that learns learning resource text data using Neural Network technology and provides learning materials at an appropriate level to learners by combining them with learners’ preferences. To evaluate the performance of the proposed algorithm, CBCNN, it was compared with existing traditional recommendation algorithms and state-of-the-art methods. As a result, CBCNN proved the accuracy of its recommendation performance by showing the lowest values compared to the other algorithms for the MAE and RMSE indicators: 2.6032 and 3.3841, respectively. Furthermore, it is generally known that the kNN-based recommendation method is more advantageous than other methods in terms of objective function, and the proposed algorithm showed the best performance in Precision, Recall, and F-Measure values among the non-kNN models. Experimental results have proven that CBCNN can alleviate the cold start problem and provide recommendations for appropriate learning resources. Similarly, a study by Chen et al. [145] also proposed a mathematical framework to which the Markov Decision Process was applied based on information about learners and learning resources and an Adaptive Learning System that introduced Plain Vanilla Systems.

For this purpose, studies have mainly been conducted using the Collaborative Filtering model, which recommends appropriate learning content by calculating similarities between learning activities or learners [7,146,147,148]. In addition, by combining the knowledge-based model and the ontology technique with the existing Collaborative Filtering model, it was possible to alleviate the problem of data sparseness based on the semantic similarity between learners and to generate more suitable recommendations [7,148].

Dwivedi et al. [147] proposed a Collaborative Filtering model-based recommendation framework that forms a learner group by reflecting the individual preferences of e-learning learners and provided the most appropriate learning content to learners based on their learning style and knowledge level. In the experimental results on the training data set to evaluate the performance of the proposed system ‘eL_GRS’, with Precision from 0.6580 to 0.7243, Recall from 0.7316 to 0.9011, F-Measure from 0.7316 to 07993, ‘eL_GRS ‘performed better than the classical group recommendation in terms of Recall and F-Measure for both random and similar groups. In particular, the recommendation performance for similar groups was more effective. In addition, in terms of the MAE of the proposed system, it showed better performance than the existing system for all the sample data. Through this, the high-quality resource recommendation ability of the proposed recommendation system for the learner group was proven. Tarus et al. [7] proposed an online learning resource recommendation system based on a Knowledge-Based Collaborative Filtering model by considering the learner’s sequential learning patterns to derive similarities between users based on the knowledge of the learner and learning resources by using the ontology technique and the Sequential Pattern Mining(SPM) algorithm. Since the proposed hybrid algorithm combines CF, ontology, and SPM, an experiment to compare the performance with the CF algorithm and the ‘CF+ontology’ algorithm was performed. All three algorithms showed optimal MAE values when the number of neighbors was 20. At this time, the proposed hybrid algorithm showed MAE values of 0.66, and ‘CF+ontology’ and CF showed MAE values of 0.69 and 0.76, respectively. Furthermore, the proposed hybrid algorithm always showed better results than the other two algorithms for the Precision and Recall indicators. In the learner’s satisfaction evaluation of the system, it was confirmed that 94% of the learners were satisfied with the recommendations of the proposed system, and the satisfaction levels for ‘CF+ontology’ and CF algorithm were 76% and 62%, respectively. Through this, we proved that the proposed algorithm outperforms the other two algorithms in terms of prediction accuracy to provide high-quality recommendations suitable for learners.

In the education service field, several studies were conducted on the smart education system in the e-learning environment, which covered not only a system that recommends learning content suitable for learners, but also a system that recommended college preparation topics and course selections so that students could experience a more substantial education [149,150,151]. Because it is important to consider not only the data on the item to be selected, but also the data of other users of the item, the Hybrid recommendation model, which combines two or more models, was mainly used in the generation of recommendations for curriculum selection. Esteban et al. [151] proposed a hybrid multi-criteria lecture recommendation system to be able to select a suitable lecture by combining a Collaborative Filtering model that considers student information, including student ratings and grades for university courses, and a Content-Based Filtering model that considers lecture information, including professors and lecture contents. To evaluate the performance of the proposed model, comparative experiments were performed using the existing CB and CF models and various criteria, including multi-criteria, lecture evaluation, and professorship. As a result, the RMSE value of the proposed hybrid approach was the lowest at 0.971, and the second-lowest value was 1.123, showing the best performance through the difference in results. Furthermore, it was proven that recommendation using multiple criteria is more effective than recommendation using single criteria.

Figure 21 visualizes the recommendation principle of a Hybrid recommendation system model that recommends learning materials suitable for users based on existing learning materials, learner preferences, and learning material data used by learners. Table 10 summarizes the research related to the recommendation system used in the education service field among the papers collected according to the research criteria of this study.

2.3.7. Academic Information Service

Due to the exponentially increasing amount of academic information, researchers in academia need to devote significant time and effort to finding academic information in the field related to their research. In the field of academic information services, research on recommendation systems has been conducted to provide information and technologies that can be helpful to scholars when conducting research.

A representative service to which the recommendation system in the academic information field is applied is the Digital Library, an information collection system that enables users to quickly and easily search for and utilize various digital materials around the world. In particular, the University Digital Libraries (UDL), a service that supports university learning, education, and research, also actively uses the recommendation system [152,153]. The recommendation system was not only used to help the process of accessing such academic information, but also research was conducted to support the researcher’s research-related academic data, thesis writing, and submission process [154,155]. In other words, the main purpose of recommendation system research in the academic information field is to recommend and provide academic information suitable for various users, including scientific communities, research institutions, and development practitioners, as well as to support the research itself. On the other hand, most recommended content in the academic information field are academic research materials composed mainly of texts. Therefore, a number of studies on the recommendation of academic information using the Content-Based Filtering recommendation model and the Hybrid recommendation model using Text Mining techniques have been conducted [156,157,158].

Achakulvisut et al. [158] proposed a digital library system based on the content of academic research using a Content-Based Filtering recommendation model so that researchers can quickly and accurately find academic materials in the scientific field. As a result of testing the performance of the proposed Digital Library system, it was shown that the proposed algorithm greatly outperforms the recommendations of other general keyword-based algorithms. Serrano-Guerrero et al. [152] performed text analysis through fuzzy logic by collecting information written by other users stored in Google Wave to efficiently search digital academic materials. A UDL integrated recommendation system was proposed using a Content-Based Filtering recommendation model through text similarity. In order to evaluate the performance of the proposed UDL integrated recommendation system, a recommendation result analysis was performed considering user opinions. The recommended user’s Precision, Recall, and F-Measure indicators were 0.7881, 0.8609, and 0.8197, respectively, while the recommended resource index values were 0.8674, 0.8734, and 0.8693, respectively. Through this, it can be seen that the system offers excellent recommendation performance and facilitates the dissemination of user and resource information in multidisciplinary resource-related tasks stored in UDL. Tejeda-Lorente et al. [153] used a Hybrid recommendation model after performing fuzzy text analysis, which is mainly used in general UDL recommendation systems. In particular, their study increased the accuracy of recommendations by adding a quality item to the data and made it easier for users to access related research resources. The proposed system provides a recommendation considering the quality of the resource. As a result of the performance evaluation of the recommendation, the average values of the F-Measure index and the MAE value were 0.6765 and 0.7565, respectively. As a result of the performance evaluation of the recommendation, which did not consider the quality of the resource, the performance of the proposed recommendation system was proven by comparing the average F-Measure and MAE values of 0.5939 and 0.7823.

A system that provides recommendations to cite data suitable for research when writing a thesis is called a citation recommendation system. These systems are created based on the user’s profile and the list of citations already included in the article the user is writing. Therefore, the Content-Based Filtering recommendation model was mainly used for citation recommendation. He et al. [154] proposed a study on a Content-Based Filtering citation recommendation system using a context-aware Text Mining technique. Unlike other citation recommendation systems, their study proposed a method of providing a citation list ranking suitable for the thesis by recognizing the context of the thesis that the user is currently writing without relying on the existing citation list. To investigate the performance of the proposed approach, CRM, extensive empirical evaluation was performed using Recall, Co-cited Probability, and NDCG indicators. When CRM was compared to seven other citation recommendation approaches, CRM performed best for all three metrics. This means that the proposed system is effective at recommending references by considering all the past citation contexts of authors.

Furthermore, a study was conducted to recommend a list of journals or conferences suitable for research. In a representative case, Wang et al. [155] proposed a system that recommends a list of journals or conferences suitable for content based on the abstract of the thesis. As a result of testing, the proposed system achieved an accuracy of 61.37% and proved that the most suitable journal or conference for research can be suggested in an average of about 5 seconds. Figure 22 shows an overview of the citation recommendation generation process reviewed in the study by Färber et al. [159] of the recommendation systems used in the academic information service field. Table 11 summarizes the research related to the recommendation system used in the academic information service field among the papers collected according to the research criteria of this study.

2.4. Recommendation System Research Trend

In this chapter, using the large amount of research data that was collected by keywords related to the application service field using the recommendation system, and without limiting the journal ranking in Google Scholar from 2010–2021, the market size, or the value of major services for each application field by collecting various data about the system, we derive a comprehensive trend in the development of recommendation systems.

2.4.1. Streaming Service and Research Trend

Figure 23 shows the trends of research and major services in the field of streaming services between 2010 and 2021. To this end, the number of published papers by year of all the papers searched for by Google Scholar with the keyword ’Streaming Service Recommendation’ from 2010 to 2021 was summarized. Furthermore, the trends in international subscribers of Netflix [160] and Disney Plus [161], the representative services of the OTT industry from 2010 to 2020, is also shown. Netflix, which started in 1997, has steadily increased its number of users [162,163] and, along with this, research in the field of streaming services has also steadily increased. However, in 2012, Netflix achieved tangible success in expanding its service globally but lost sales. In 2013, copyright issues such as the illegal copying of Netflix’s original series arose, causing difficulties for the business [164]. It may also feature a certain partial correlation with the declining interest in streaming service-related studies in 2012 and 2013. Nevertheless, Netflix has further expanded its service worldwide, making it the streaming service with the most subscribers among the current OTT services. After 2020, Netflix’s subscribers increased more rapidly due to social isolation as a consequence of COVID-19, along with changes in media consumption [165]. The success of Netflix served as an opportunity to expand the OTT service industry, contributing to the increase in OTT services, including Disney Plus and Amazon Prime.

In the case of Disney, which distributes media content through movie theaters, there was a financial loss in related sectors due to a reduction in film production as a consequence of COVID-19 [166]. However, Disney rapidly increased its subscribers and occupied the market through the Disney Plus service in early 2020 [167].

As shown in Figure 23, the study of the recommendation systems used by streaming service, which has increased rapidly since 2015, can be seen as featuring a correlation with the development of streaming services and the increase in their users. Furthermore, in the context of this advanced technology, COVID-19, which began to spread in early 2020, is believed to have led to the rapid growth of streaming services.

2.4.2. Social Network Service and Research Trends

Figure 24 shows the trends in social network service research and major services between 2010 and 2021. For this purpose, the number of published papers by year of all the papers searched by Google Scholar with the keyword ‘Social Network Service Recommendation’ from 2010 to 2021 was summarized. In addition, the trend in international users of Facebook [168], Instagram [169], and Twitter [170], which are representative services of the social network industry from 2010 to 2020, is also shown.

Twitter was an early SNS service that started in 2006, but since 2010, Facebook has overwhelmingly dominated SNS services. Facebook possesses a wider public range than Twitter, allows users to check the public timeline of people around the world, and offers the advantage that all users can become friends, so the number of users has steadily increased [171]. Instagram first started service as an IOS app in 2010; it then released the Android version in 2012, after which its number of users increased rapidly [172], Subsequently, in April of the same year, Facebook took over Instagram and performed technological developments, such as updating user feed posts and location tag functions [173]. Since 2014, Instagram has surpassed Twitter’s number of users [174].

In 2012 and 2013, when the number and interest of existing social network service users rapidly increased, research on recommendation systems based on social network services showed an explosive increase. Research related to Social Network Service recommendation systems, which had stalled in 2014, began to reactivate from 2016. In 2016, Instagram further updated its post advertisement promotion function for business accounts, and the number of users increased. Although the link between the increase in the number of studies and the improvement of the Instagram function is unclear, it can be interpreted that a linkage exists. Furthermore the data collected from Instagram and Facebook began to be used in various fields, and from 2017, recommendation research in various fields, combined with SNS data, increased in a much higher volume than in the field of recommendation system research used only in the SNS field [175,176,177].

2.4.3. Tourism Service and Research Trend

Figure 25 shows the trends of tourism service research and major services between 2010 and 2021. To this end, the number of published papers by year of all the papers searched by Google Scholar with the keyword ‘Tourism Service Recommendation’ from 2010 to 2021 was summarized. Furthermore the trend in international users of Airbnb [178], a representative service of the tourism industry from 2010 to 2020, is also shown.

The accommodation sharing service Airbnb added a social connection function by linking with Facebook in 2011 [179]. The sharing function between users was activated and a differentiated business model was established, which served as an opportunity to increase the number of users in a short period of time. From Figure 25, it can be seen that during the period from 2014 to 2019, when the number of Airbnb users increased rapidly, interest in tourism service recommendation research increased slightly at the beginning, and then exploded in 2019.

In 2020, the number of Airbnb users plummeted due to a decrease in travelers as a result of the COVID-19 pandemic. Despite the sharp drop in the number of users, recommendation research in the tourism service field only slightly declined; it has remained high since 2019. It is thought that this may be related to users’ desire for tourism-related services after the COVID-19 pandemic. Meanwhile, new research aimed at circumventing the COVID-19 situation, such as virtual travel, has also begun [180,181,182,183,184].

2.4.4. E-Commerce Service and Research Trends

Figure 26 and Figure 27 show the trends in e-commerce service field research and major services from 2010 to 2021 together. For this purpose, all the papers searched by Google Scholar with the keyword ’E-Commerce Service Recommendation’ from 2010 to 2021 were summarized in the number of published papers by year. Furthermore, the annual sales of Amazon [185] and Alibaba [186], which are representative services of the e-commerce industry from 2010 to 2020, are shown together.

Amazon is an e-commerce service that has been using a recommendation system for more than 20 years, since 1998. Various recommendation system studies have been conducted on Amazon’s recommendation system model, starting with the study by Linden et al. [187,188]. Although Amazon’s annual sales have been steadily increasing, Amazon’s annual sales started to increase even more significantly from 2017 to 2020. From 2013 to 2015, a large amount of research on recommendation systems in the field of e-commerce service was conducted, so we can infer its relevance to the advancement of the service.

The recommendation system also played an important role in the growth of Taobao, an online e-commerce service operated by Alibaba [189]. The dataset collected from Taobao, which provides various shopping records, user data, and item datasets, has supported research into recommendation systems [190]. Taobao is a C2C-type e-commerce service, gradually expanding its business to a platform including purchase, distribution, and pay. Accordingly, annual sales have increased rapidly since 2016, and it has established itself as the largest e-commerce service in China. Due to the rapid growth of the e-commerce market, research on recommendation systems is also on the rise.

2.4.5. Healthcare Service and Research Trends

Figure 28 shows the trends of research and major services in the healthcare service field between 2010 and 2021. To this end, the number of published papers by year of all the papers searched by Google Scholar with the keyword ‘Healthcare Service Recommendation’ from 2010 to 2021 was summarized. In addition, the number of users of Apple Watch [191], the representative of the wearable device industry most closely related to healthcare, from 2014 to 2020, and the number of users of Fitbit [192] worldwide from 2012 to 2019 are shown together.

Apple Watch, which first appeared in 2014, is a new type of watch for health management and tracking that records heart rate, blood oxygen, and movement. The number of users of Apple Watch has steadily increased, surpassing the number of users of Fitbit, which was the first smart watch in 2017, and is still overwhelmingly leading the market. On the other hand, when Fitbit first released Smart Watch, but in 2014, an allergic reaction occurred in some of Fitbit’s product lines, so they recalled the entire product and released a new product that was tested by a dermatologist [193]. In general, before 2018, research on recommendation systems in the health care service field was sluggish compared to research on recommendation systems in other fields, but research on recommendation systems related to healthcare has increased significantly since 2018. This seems to have had a significant impact on the performance of related research as Apple started supporting research in various healthcare fields, such as research on irregular heart rate detection algorithms [194], from 2017. Meanwhile, in 2017, as Apple Watch shipments began to surpass the export of Swiss watches, smart watches began to lead the market share of existing watches [195,196]. research on recommendation systems related to healthcare, which has been rapidly increasing since 2018, and the market growth in the healthcare field are influencing each other, creating a synergistic effect.

2.4.6. Education Service and Research Trends

Figure 29 shows the trends in research and major services in the field of education service between 2010 and 2021. For this purpose, the number of published papers by year of all the papers searched for with the keyword ‘Education Service Recommendation’ in Google Scholar from 2010 to 2021 was summarized. Furthermore, the market value [197] of e-learning services from 2010 to 2020 is also shown.

E-learning is a classroom environment in which students learn using electronic devices, unlike the traditional face-to-face classroom environment. E-learning was initially a form of learning supported only by computers. However, from 2010, due to the development and distribution of various smart devices, such as tablet PCs and smartphones, smart recommendations for learner-tailored learning gradually began to expand in the field of e-learning [198]. In addition, as conferencing apps such as Zoom’s beta service and Google Meeting (2017) began to appear in 2012, an environment that enables real-time communication with educators began to be provided. Therefore, it can be seen that e-learning gradually solves the problems of existing learning with the advancement of technology, and studies related to learning recommendation systems have shown an overall increase since 2010. In addition, as the demand increases, research on educational service recommendation systems using various learning content data and learning data also increases.

In particular, when it became impossible to go to school after the COVID-19 outbreak in 2019, schools provided students with learning using various Conferencing App and LMS (Learning Management System). Therefore, research on recommendation systems in the field of education services has also grown in interest since 2019.

2.4.7. Summary of Research Trends

Figure 30 visualizes the ratio of the total papers collected on seven service fields in this study from 2010 to 2021. Figure 31 visualizes the trend of the ratio of the number of papers by year. Through these figures, it can be seen that a significant amount of research on recommendation systems was conducted from 2010 to 2021 in the following order: social network services, tourism, healthcare, e-commerce, and education. Furthermore, from Figure 31, it can be seen that tourism, healthcare, and education have recently taken up a high proportion, and research is expanding. From the data, it can be seen that the interest in the field reflects the rapid increase in people’s interest in particular lifestyles, and that the field of education enables effective alternatives to offline education.

3. Conclusions

The development and spread of the Internet, smart devices, and SNS have led to the expansion of various web and application services. Therefore, it is necessary to develop a variety of recommendation systems that can help users to efficiently receive item information and make decisions amid the rapidly increased amount of item information due to the expansion of these services.

Hence, recommendation systems for several application fields that leverage real-time data acquired by wearable devices and click stream drive better outcomes in many cases. For example, the recommended results, such as the diagnosis and treatment method provided by a healthcare field recommendation system has a lower affinity than results based on the clinical data. However it offers considerable value in the form of supportive information, which can provide timely advice for consultation services and immediate measures, since real-time data ensures a more relevant result by reflecting the current status of patients.

In terms of technology, the recommendation system is largely divided into a data mining part that performs analysis based on data collected about items and users, and a recommendation filtering model area. Each technology and model has been researched and developed to be more customized to the service field applying the recommendation system.

In this study, after collecting research on recommendation systems from 2010 to 2020, the trend in recommendation system models, the various technologies used in recommendation systems, and the business fields where these recommendation systems are utilized was analyzed. First, in the case of the recommendation system model, the Content-Based Filtering recommendation model, which was one of the earliest models to have been used, has gradually been used alone. Furthermore, although research related to Collaborative Filtering gained traction from 2014, it was found that research on Hybrid systems that can complement the strengths and weaknesses of the Content-Based Filtering recommendation model and the Collaborative Filtering recommendation model increased significantly. However, there are cases where the use of Content-Based Filtering and Collaborative Filtering is more appropriate, depending on the service application field. Therefore, research should be conducted in connection with the research according to the field of service to be applied.

As for the filtering model of the recommendation system, studies on techniques such as Text Mining, KNN, Clustering, Matrix Factorization, and Neural Network were analyzed. Over a long period, Text Mining technology for analyzing text information and Clustering technology for analyzing user or location data of a similar group for recommendation have been extensively studied. However, recently, interest in the high possibility of applying the Neural Network technology to a recommendation system has increased, and modeling studies to additionally secure or supplement data are increasing. Therefore, studies to aiming improve the performance of recommendation systems themselves are being actively conducted and expanded.

Meanwhile, recommendation systems are closely related to a service area in which various recommendations are used. Based on papers published by representative companies that provide real services, such as Netflix, Amazon, and Yahoo, there are cases where the research on recommendation systems is actively used in real services. In this study, representative service areas where the recommendation system was used from 2010 to 2021 were defined. Furthermore, through an assessment of the statistics that confirm the value of businesses, such as the number of total studies in each field, the number of users with the highest valuation in each service field, and annual revenue, the study of the recommendation system suggested that it can work with interconnectivity on the growth of actual business.

In this study, not only the technical analysis of the recommendation system research, but also examples of practical service applications and the interconnectivity between recommendation system-related research and the business of the application service were discussed from a macro perspective. This was aimed at providing a broad understanding to researchers interested in recommendation systems. In the future, based on this study, we plan to expand our research to the research and development of recommendation systems suitable for the characteristics of business by application service field [94,199].

Author Contributions

Conceptualization, H.K., S.L., Y.P., A.C.; methodology, H.K., S.L., Y.P., A.C.; validation, H.K.; formal analysis, S.L., Y.P.; investigation, H.K., S.L., A.C.; writing—original draft preparation, H.K., S.L., Y.P., A.C.; writing—review and editing, H.K., S.L.; visualization, H.K., S.L., Y.P., A.C.; supervision, H.K.; project administration, S.L., A.C.; funding acquisition, H.K.; All authors have read and agreed to the published version of the manuscript.

Funding

This research was supported by the MISP (Ministry of Science, ICT and Future Planning), Korea, under the National Program for Excellence in SW (2016-0-00022) supervised by the IITP (Institute for Information and Communications Technology Promotion) (2016-0-00022). The work was further supported by a research grant from Seoul Women’s University (2021-0144).

Data Availability Statement

Publicly available datasets were analyzed in this study. This data can be found here: https://support.apple.com/en-us/HT206983 (accessed on 27 April 2021), https://backlinko.com/netflix-users (accessed on 25 February 2021), https://www.statista.com/statistics/1095372/disney-plus-number-of-subscribers-us/ (accessed on 19 August 2021), https://web.archive.org/web/20120109175608/http://www.bbc.co.uk/news/technology-16467432 (accessed on 9 January 2012), https://web.archive.org/web/20141127211314/http://www.prnewswire.com/news-releases/netflix-launches-in-sweden-denmark-norway-and-finland-174749581.html (accessed on 27 November 2014), https://variety.com/2013/digital/news/pirates-swarm-netflixs-house-of-cards-1200333171/ (accessed on 4 April 2013), https://www.latimes.com/entertainment-arts/business/story/2020-12-09/everything-hollywood-lost-during-the-pandemic (accessed on 9 December 2020), https://www.insiderintelligence.com/insights/ott-vod-video-streaming-services/ (accessed on 21 July 2021), https://www.statista.com/chart/10047/facebooks-monthly-active-users/ (accessed on 4 February 2021), https://www.connectivasystems.com/instagram-stats-2020/ (accessed on 1 April 2020), https://www.statista.com/statistics/282087/number-of-monthly-active-twitter-users/ (accessed on 31 December 2021), https://onezero.medium.com/the-inside-story-of-how-facebook-acquired-instagram-318f244f1283 (accessed on 5 August 2020), https://www.businessinsider.com.au/chart-of-the-day-instagram-is-now-bigger-than-twitter-2014-12 (accessed on 11 De-cember 2014), https://lb-aps-frontend.statista.com/chart/20386/guests-staying-at-airbnb-appartments-on-new-years-eve/ (accessed on 2 January 2020), https://techcrunch.com/2011/05/09/airbnb-social-connections/ (accessed on 10 May 2011), https://www.statista.com/statistics/273963/quarterly-revenue-of-amazoncom/ (accessed on 19 July 2021), https://www.statista.com/statistics/225614/net-revenue-of-alibaba/ (accessed on 15 September 2021), https://www.statista.com/statistics/1221051/apple-watch-users-worldwide/ (accessed on 17 March 2021), https://www.statista.com/chart/13083/worldwide-fitbit-shipments/ (accessed on 4 August 2020), https://www.huffpost.com/entry/fitbit-force-recall_n_4832771 (accessed on 21 February 2014), https://www.mobihealthnews.com/news/apple-study-finds-watch-can-detect-more-types-irregular-heartbeats (accessed on 29 September 2021), https://www.statista.com/statistics/526005/global-apple-watch-shipments-forecast/ (accessed on 31 December 2021), https://www.businessinsider.com/apple-watch-sells-more-than-swiss-watches-charts-2018-2 (accessed on 13 February 2018), and https://elearningfeeds.com/elearning-statistics-and-trends-2020/ (accessed on 14 December 2020).

Conflicts of Interest

The authors declare no conflict of interest.

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Figures and Tables

Figure 1. Process of selecting various recommendation system-related studies that were carried out for research on recommendation model, recommendation technique, trend, and recommendation service area.

Figure 2. Process of analyzing recommended service area research and business size of representative service for each service field.

Figure 3. Overview of recommendation models.

Figure 4. Overall flow of recommendation models and recommendation techniques.

Figure 5. Process of general recommendation system models.

Figure 6. Recommendation principle of Contents-Based Filtering Model.

Figure 7. Recommendation principle of Collaborative Filtering Model.

Figure 8. Recommendation principle of Hybrid recommendation model.

Figure 9. Web development.

Figure 10. Trend by recommendation model of recommendation system research of papers collected according to this study standard.

Figure 11. Technology mainly used in recommendation system.

Figure 12. Typical data mining process.

Figure 13. Typical text mining process.

Figure 14. Frequency (number) of papers using recommendation techniques during the period investigated in this paper (2010–2020).

Figure 15. Trend in recommendation technique papers by year (by period) during the period investigated in this paper (2010–2020).

Figure 16. List of services that mainly use the recommendation system analyzed in this study.

Figure 17. Hybrid recommendation overview of overall social network service.

Figure 18. Structural diagram of a tourism recommendation system for providing popular landmarks and optimal routes.

Figure 19. Product recommendation system that extends item-based CF through online–offline data combination.

Figure 20. Workflow of disease diagnosis and treatment recommendation system.

Figure 21. Structure of the educational content recommendation system.

Figure 22. Citation list recommendation system.

Figure 23. Trends in streaming service field research from 2010 to 2021 and the number of users of Netflix and Disney, the representative streaming service types.

Figure 24. Social network service field research trend from 2010 to 2021 and the number of users of Facebook, Twitter, and Instagram, which are representative social network service types.

Figure 25. Tourism service field research trends from 2010 to 2021 and the number of users of Airbnb, a representative type of tourism service.

Figure 26. E-Commerce service field research trends from 2010 to 2021 and the number of users of Amazon, a representative service type of e-commerce service.

Figure 27. E-commerce service field research trends from 2010 to 2021 and the number of users of Alibaba, a type of e-commerce service.

Figure 28. Trends in healthcare service field research from 2010 to 2021 and the number of users of Apple Watch and Fitbit, representative healthcare-related wearable devices.

Figure 29. Education service field research trends and e-learning service market value from 2010 to 2020.

Figure 30. Distribution of seven recommendation system fields from 2010 to 2021.

Figure 31. Recommendation system research in seven fields from 2010 to 2021.

Table 1

Seven types and concepts according to the method combining the filtering technique of the Hybrid system.

Hybrid Method	Description
WeightedHybridization	A method in which the weight is gradually adjusted according to the degree to which the user’s evaluation of an item coincides with the evaluation predicted by the recommendation system.
SwitchingHybridization	A method of changing the recommendation model used depending on the situation.
CascadedHybridization	After using one of the recommendation system models to create a candidate set with a similar taste to the user, the method combinesthe previously used recommendation system model with another model to sort the candidate set in the order of items most suited to the user’s taste.
MixedHybridization	When many recommendations are made at the same time, Content-Based Filtering can recommend items based on the description of the items without user evaluation, but there is a start-up problem in that it cannot recommend new items with insufficient information. In order to solve this problem, the Mixed Hybridization method recommends items to the user by integrating the user’s past history data that is collected when the recommendation system service is started.
Feature-Combination	A collaborative filtering model is used for featured data and example data for items, and a Content-Based Filtering model is used for augmented data.
Feature-Augmentation	A Hybrid method in which one Recommendation System Model is used to classify an item’s preference score or item, and the generated information is integrated into the next Recommendation System Model.
Meta-Level	A method of using the entire model of one recommendation system as the input data in the model of another recommendation system. Since the user’s taste is compressed and expressed using Meta-Level, it is easier to operate the Collaborative Mechanism than when raw rating data are used as single-input data.

Table 2

Confusion Matrix of Recommendation Systems.

Preference	Recommended	Not Recommended
User-preferred item	True Positives (TP)	True Negatives (TN)
User-non-preferred item	False Positives (FP)	False Negatives (FN)

Table 3

Confusion matrix of recommendation systems.

Evaluation Metrics	Equation	Definition
Precision	Precision = TP/TP + FP	The ratio of items that match the user’s taste among all items recommended to the user.
Recall	Recall = TP/TP + FN	The ratio of items that match the user’s taste among all items recommended to the user.
Accuracy	Accuracy = TP + TN/TP + FN + FP + TN	The ratio of successful referrals to total referrals.
F-Measure	F-Measure = 2 × (Precision × Recall)/(Precision + Recall)	Harmonic mean value of Precision and Recall.
ROC curve	Ratio of TP Rate (= TP/TP + FN) and FP Rate (= FP/FP + TN)	A graph showing the relationship between FPR and TPR. A visual description of the ratio of the performance results of Precision and Recall.
AUC	Area under the ROC curve	AUC measures the probability that a random relevant item is ranked higher than a random irrelevant item

Table 4

Characteristics of each element according to the development of Web 1.0, Web 2.0, and Web 3.0.

	Web 1.0	Web 2.0	Web 3.0
Communication	Broadcast	Interactive	Engaged/Invested
Information	Static/Read-only	Dynamic	Portable/Personal
Focus	Organization	Community	Individual
Personal	Home Pages	Blogs/ SNS	Life Streams
Interaction	Web Forms	Web Applications	Smart Applications
Search	Directories	Keywords/Tags	Context/Relevance
Metrics	Page Views	Cost Per Click	User Engagement
Research	Britannica Online	Wikipedia	The Semantic Web

Table 5

Recommendation system study of streaming service field.

Streaming Service	RS Model	RSTechniques	Literature Sources
Video	CF	Clustering	[95]
		MatrixFactorization	[85]
	HybridSystem	Text Mining	[8,91]
		MatrixFactorization	[9,36,88]
Music	CB	Text Mining	[90]
		Neural Network	[81]
	CF	KNN	[96]
	HybridSystem	MatrixFactorization	[87,92]
		Neural Network	[89]

Table 6

Recommendation system study of social network service field.

SNSService	RSModel	RSTechniques	Literature Sources
SNS Followersor ItemRecommendation	CF	Text Mining	[100]
		KNN	[3]
		Matrix Factorization	[4,78]
	HybridSystem	Text Mining	[6]
		Clustering	[5]
InformationRecommendationUsing SNS Data	CB	Text Mining	[18,98]
		Matrix Factorization	[101]
		Neural Network	[11]
	CF	Text Mining	[97]
		Matrix Factorization	[10,79]
		Neural Network	[10]
	HybridSystem	Text Mining	[12]
		Clustering	[102]

Table 7

Research on the recommendation system in tourism service field.

Tourism Service	RS Model	RSTechniques	Literature Sources
Tourist Attractionsor TouristInformationRecommendation	CB	Clustering	[107]
	CF	Text Mining	[108]
		Clustering	[109]
		Matrix Factorization	[109]
	HybridSystem	Text Mining	[103,110,111]
		Clustering	[2]
Tourist Routeor TransportationRecommendation	CB	Clustering	[103,106]
	CF	Text Mining	[105]

Table 8

E-commerce service field recommendation system study.

E-CommerceService	RSModel	RSTechniques	Literature Sources
Web	CF	Text Mining	[122]
		Clustering	[72]
			[118]
	HybridSystem	Neural Network	[113]
Mobile	HybridSystem	Neural Network	[120]

Table 9

Healthcare service field recommendation system study.

Healthcare Service	RSModel	RSTechniques	Literature Sources
MedicalTreatment or Diet Recommendation	CB	Text Mining	[129,139]
		Clustering	[130]
		Neural Network	[134]
	CF	Clustering	[131]
	HybridSystem	Neural Network	[133]
Health Information RecommendationUsing E-Health	CB	Text Mining	[136,137]
		Neural Network	[138]

Table 10

Study of the education service field recommendation system.

EducationService	RSModel	RS Techniques	Literature Sources
E-Learningand Customized LearningRecommendation	CB	Text Mining	[145]
		Neural Network	[144]]
	CF	Text Mining	[147]
	HybridSystem	Text Mining	[7,148]
		Clustering	[143,146]
Education CourseRecommendation	HybridSystem	Text Mining	[149,151]

Table 11

Recommendation system study of academic information service field.

Academic Information Service	RSModel	RSTechniques	Literature Sources
RecommendationIn Digital Library	CB	Text Mining	[158]
	HybridSystem	Text Mining	[152,153]
CitationRecommendation	CB	TextMining	[154,155]

References

1. Beheshti, A.; Yakhchi, S.; Mousaeirad, S.; Ghafari, S.M.; Goluguri, S.R.; Edrisi, M.A. Towards Cognitive Recommender Systems. Algorithms; 2020; 13, 176. [DOI: https://dx.doi.org/10.3390/a13080176]

2. Abbasi-Moud, Z.; Vahdat-Nejad, H.; Sadri, J. Tourism Recommendation System Based on Semantic Clustering and Sentiment Analysis. Expert Syst. Appl.; 2021; 167, 114324. [DOI: https://dx.doi.org/10.1016/j.eswa.2020.114324]

3. Liu, F.; Lee, H.J. Use of Social Network Information to Enhance Collaborative Filtering Performance. Expert Syst. Appl.; 2010; 37, pp. 4772-4778. [DOI: https://dx.doi.org/10.1016/j.eswa.2009.12.061]

4. Yang, B.; Lei, Y.; Liu, J.; Li, W. Social Collaborative Filtering by Trust. IEEE Trans. Pattern Anal. Mach. Intell.; 2017; 39, pp. 1633-1647. [DOI: https://dx.doi.org/10.1109/TPAMI.2016.2605085] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/27608451]

5. Amato, F.; Moscato, V.; Picariello, A.; Piccialli, F. SOS: A Multimedia Recommender System for Online Social Networks. Future Gener. Comput. Syst.; 2019; 93, pp. 914-923. [DOI: https://dx.doi.org/10.1016/j.future.2017.04.028]

6. Capdevila, J.; Arias, M.; Arratia, A. GeoSRS: A Hybrid Social Recommender System for Geolocated Data. Inf. Syst.; 2016; 57, pp. 111-128. [DOI: https://dx.doi.org/10.1016/j.is.2015.10.003]

7. Tarus, J.K.; Niu, Z.; Yousif, A. A Hybrid Knowledge-Based Recommender System for e-Learning Based on Ontology and Sequential Pattern Mining. Future Gener. Comput. Syst.; 2017; 72, pp. 37-48. [DOI: https://dx.doi.org/10.1016/j.future.2017.02.049]

8. Choi, S.-M.; Ko, S.-K.; Han, Y.-S. A Movie Recommendation Algorithm Based on Genre Correlations. Expert Syst. Appl.; 2012; 39, pp. 8079-8085. [DOI: https://dx.doi.org/10.1016/j.eswa.2012.01.132]

9. Walek, B.; Fojtik, V. A Hybrid Recommender System for Recommending Relevant Movies Using an Expert System. Expert Syst. Appl.; 2020; 158, 113452. [DOI: https://dx.doi.org/10.1016/j.eswa.2020.113452]

10. Pan, Y.; He, F.; Yu, H. Learning Social Representations with Deep Autoencoder for Recommender System. World Wide Web; 2020; 23, pp. 2259-2279. [DOI: https://dx.doi.org/10.1007/s11280-020-00793-z]

11. Aivazoglou, M.; Roussos, A.O.; Margaris, D.; Vassilakis, C.; Ioannidis, S.; Polakis, J.; Spiliotopoulos, D. A Fine-Grained Social Network Recommender System. Soc. Netw. Anal. Min.; 2020; 10, 8. [DOI: https://dx.doi.org/10.1007/s13278-019-0621-7]

12. García-Sánchez, F.; Colomo-Palacios, R.; Valencia-García, R. A Social-Semantic Recommender System for Advertisements. Inf. Process. Manag.; 2020; 57, 102153. [DOI: https://dx.doi.org/10.1016/j.ipm.2019.102153]

13. Bollen, D.; Knijnenburg, B.P.; Willemsen, M.C.; Graus, M. Understanding Choice Overload in Recommender Systems. Proceedings of the Fourth ACM Conference on Recommender Systems; Barcelona, Spain, 26–30 September 2010; pp. 63-70.

14. Im, I.; Hars, A. Does a One-Size Recommendation System Fit All? The Effectiveness of Collaborative Filtering Based Recommendation Systems across Different Domains and Search Modes. ACM Trans. Inf. Syst.; 2007; 26, 4. [DOI: https://dx.doi.org/10.1145/1292591.1292595]

15. Wu, M.-L.; Chang, C.-H.; Liu, R.-Z. Integrating Content-Based Filtering with Collaborative Filtering Using Co-Clustering with Augmented Matrices. Expert Syst. Appl.; 2014; 41, pp. 2754-2761. [DOI: https://dx.doi.org/10.1016/j.eswa.2013.10.008]

16. Van Meteren, R.; Van Someren, M. Using content-based filtering for recommendation. Proceedings of the Machine Learning in the New Information Age MLnet/ECML2000 Workshop; Barcelona, Spain, 30 May 2000; pp. 47-56.

17. Iyengar, S.S.; Lepper, M.R. When Choice Is Demotivating: Can One Desire Too Much of a Good Thing?. J. Personal. Soc. Psychol.; 2000; 79, pp. 995-1006. [DOI: https://dx.doi.org/10.1037/0022-3514.79.6.995]

18. Loeb, S.; Terry, D. Information Filtering. Commun. ACM; 1992; 35, pp. 26-28. [DOI: https://dx.doi.org/10.1145/138859.138860]

19. Cantador, I.; Fernández, M.; Vallet, D.; Castells, P.; Picault, J.; Ribière, M. A Multi-Purpose Ontology-Based Approach for Personalised Content Filtering and Retrieval. Advances in Semantic Media Adaptation and Personalization; Wallace, M.; Angelides, M.C.; Mylonas, P. Studies in Computational Intelligence Springer: Berlin/Heidelberg, Germany, 2008; Volume 93, pp. 25-51. ISBN 978-3-540-76359-8

20. Salter, J.; Antonopoulos, N. CinemaScreen Recommender Agent: Combining Collaborative and Content-Based Filtering. IEEE Intell. Syst.; 2006; 21, pp. 35-41. [DOI: https://dx.doi.org/10.1109/MIS.2006.4]

21. Iwahama, K.; Hijikata, Y.; Nishida, S. Content-Based Filtering System for Music Data. Proceedings of the 2004 International Symposium on Applications and the Internet Workshops. 2004 Workshops; Tokyo, Japan, 26–30 January 2004; 2004; pp. 480-487.

22. Weihong, H.; Yi, C. An E-Commerce Recommender System Based on Content-Based Filtering. Wuhan Univ. J. Nat. Sci.; 2006; 11, pp. 1091-1096. [DOI: https://dx.doi.org/10.1007/BF02829216]

23. Ghauth, K.I.; Abdullah, N.A. Learning Materials Recommendation Using Good Learners’ Ratings and Content-Based Filtering. Educ. Technol. Res. Dev.; 2010; 58, pp. 711-727. [DOI: https://dx.doi.org/10.1007/s11423-010-9155-4]

24. Di Noia, T.; Mirizzi, R.; Ostuni, V.; Romito, D.; Zanker, M. Linked open data to support content-based recommender systems. Proceedings of the 8th International Conference on Semantic Systems; Graz, Austria, 5–7 September 2012; pp. 1-8.

25. Kompan, M.; Bieliková, M. Content-Based News Recommendation. E-Commerce and Web Technologies; Buccafurri, F.; Semeraro, G. Lecture Notes in Business Information Processing Springer: Berlin/Heidelberg, Germany, 2010; Volume 61, pp. 61-72. ISBN 978-3-642-15207-8

26. Son, J.; Kim, S.B. Content-Based Filtering for Recommendation Systems Using Multiattribute Networks. Expert Syst. Appl.; 2017; 89, pp. 404-412. [DOI: https://dx.doi.org/10.1016/j.eswa.2017.08.008]

27. Goldberg, D.; Nichols, D.; Oki, B.M.; Terry, D. Using Collaborative Filtering to Weave an Information Tapestry. Commun. ACM; 1992; 35, pp. 61-70. [DOI: https://dx.doi.org/10.1145/138859.138867]

28. Resnick, P.; Iacovou, N.; Suchak, M.; Bergstrom, P.; Riedl, J. GroupLens: An Open Architecture for Collaborative Filtering of Netnews. Proceedings of the 1994 ACM Conference on Computer Supported Cooperative Work; Chapel Hill, NC, USA, 22–26 October 1994; pp. 175-186.

29. Park, S.-H.; Han, S.P. Empirical Analysis of the Impact of Product Diversity on Long-Term Performance of Recommender Systems. Proceedings of the 14th Annual International Conference on Electronic Commerce—ICEC ’12; Singapore, 7–8 August 2012; pp. 280-281.

30. Koren, Y.; Bell, R. Advances in collaborative filtering. Recommender Systems Handbook; Springer: Boston, MA, USA, 2015; pp. 77-118.

31. Nilashi, M.; Ibrahim, O.; Bagherifard, K. A Recommender System Based on Collaborative Filtering Using Ontology and Dimensionality Reduction Techniques. Expert Syst. Appl.; 2018; 92, pp. 507-520. [DOI: https://dx.doi.org/10.1016/j.eswa.2017.09.058]

32. Ansari, A.; Essegaier, S.; Kohli, R. Internet Recommendation Systems. J. Mark. Res.; 2000; 37, pp. 363-375. [DOI: https://dx.doi.org/10.1509/jmkr.37.3.363.18779]

33. Papagelis, M.; Plexousakis, D.; Kutsuras, T. Alleviating the Sparsity Problem of Collaborative Filtering Using Trust Inferences. Trust Management; Herrmann, P.; Issarny, V.; Shiu, S. Lecture Notes in Computer Science Springer: Berlin/Heidelberg, Germany, 2005; Volume 3477, pp. 224-239. ISBN 978-3-540-26042-4

34. Wei, J.; He, J.; Chen, K.; Zhou, Y.; Tang, Z. Collaborative Filtering and Deep Learning Based Recommendation System for Cold Start Items. Expert Syst. Appl.; 2017; 69, pp. 29-39. [DOI: https://dx.doi.org/10.1016/j.eswa.2016.09.040]

35. Gras, B.; Brun, A.; Boyer, A. Identifying Grey Sheep Users in Collaborative Filtering: A Distribution-Based Technique. Proceedings of the 2016 Conference on User Modeling Adaptation and Personalization; Halifax, NS, Canada, 13–16 July 2016; pp. 17-26.

36. Barragáns-Martínez, A.B.; Costa-Montenegro, E.; Burguillo, J.C.; Rey-López, M.; Mikic-Fonte, F.A.; Peleteiro, A. A Hybrid Content-Based and Item-Based Collaborative Filtering Approach to Recommend TV Programs Enhanced with Singular Value Decomposition. Inf. Sci.; 2010; 180, pp. 4290-4311. [DOI: https://dx.doi.org/10.1016/j.ins.2010.07.024]

37. Liu, H.; Hu, Z.; Mian, A.; Tian, H.; Zhu, X. A New User Similarity Model to Improve the Accuracy of Collaborative Filtering. Knowl.-Based Syst.; 2014; 56, pp. 156-166. [DOI: https://dx.doi.org/10.1016/j.knosys.2013.11.006]

38. Shi, Y.; Larson, M.; Hanjalic, A. Collaborative Filtering beyond the User-Item Matrix: A Survey of the State of the Art and Future Challenges. ACM Comput. Surv.; 2014; 47, pp. 1-45. [DOI: https://dx.doi.org/10.1145/2556270]

39. Kim, H.-N.; Ji, A.-T.; Ha, I.; Jo, G.-S. Collaborative Filtering Based on Collaborative Tagging for Enhancing the Quality of Recommendation. Electron. Commer. Res. Appl.; 2010; 9, pp. 73-83. [DOI: https://dx.doi.org/10.1016/j.elerap.2009.08.004]

40. Bobadilla, J.; Hernando, A.; Ortega, F.; Gutiérrez, A. Collaborative Filtering Based on Significances. Inf. Sci.; 2012; 185, pp. 1-17. [DOI: https://dx.doi.org/10.1016/j.ins.2011.09.014]

41. Koohi, H.; Kiani, K. User Based Collaborative Filtering Using Fuzzy C-Means. Measurement; 2016; 91, pp. 134-139. [DOI: https://dx.doi.org/10.1016/j.measurement.2016.05.058]

42. Choi, K.; Suh, Y. A New Similarity Function for Selecting Neighbors for Each Target Item in Collaborative Filtering. Knowl.-Based Syst.; 2013; 37, pp. 146-153. [DOI: https://dx.doi.org/10.1016/j.knosys.2012.07.019]

43. Basilico, J.; Hofmann, T. Unifying Collaborative and Content-Based Filtering. Proceedings of the Twenty-First International Conference on Machine Learning—ICML ’04; Banff, AB, Canada, 4–8 July 2004; 2004; 9.

44. Burke, R. Hybrid recommender systems: Survey and experiments. User Model. User-Adapt. Interact.; 2002; 12, pp. 331-370. [DOI: https://dx.doi.org/10.1023/A:1021240730564]

45. Kim, Y.M.; Choi, S. Scalable Variational Bayesian Matrix Factorization with Side Information. Proceedings of the Seventeenth International Conference on Artificial Intelligence and Statistics; Reykjavik, Iceland, 22–25 April 2014; pp. 493-502.

46. Strub, F.; Gaudel, R.; Mary, J. Hybrid Recommender System Based on Autoencoders. Proceedings of the 1st Workshop on Deep Learning for Recommender Systems; Boston, MA, USA, 15 September 2016; pp. 11-16.

47. Zhao, H.; Yao, Q.; Song, Y.; Kwok, J.T.; Lee, D.L. Side Information Fusion for Recommender Systems over Heterogeneous Information Network. ACM Trans. Knowl. Discov. Data; 2021; 15, pp. 1-32. [DOI: https://dx.doi.org/10.1145/3441446]

48. Fayyaz, Z.; Ebrahimian, M.; Nawara, D.; Ibrahim, A.; Kashef, R. Recommendation Systems: Algorithms, Challenges, Metrics, and Business Opportunities. Appl. Sci.; 2020; 10, 7748. [DOI: https://dx.doi.org/10.3390/app10217748]

49. Bag, S.; Kumar, S.; Awasthi, A.; Tiwari, M.K. A Noise Correction-Based Approach to Support a Recommender System in a Highly Sparse Rating Environment. Decis. Support Syst.; 2019; 118, pp. 46-57. [DOI: https://dx.doi.org/10.1016/j.dss.2019.01.001]

50. Khelloufi, A.; Ning, H.; Dhelim, S.; Qiu, T.; Ma, J.; Huang, R.; Atzori, L. A Social-Relationships-Based Service Recommendation System for SIoT Devices. IEEE Internet Things J.; 2021; 8, pp. 1859-1870. [DOI: https://dx.doi.org/10.1109/JIOT.2020.3016659]

51. Liu, Y.; Cheng, J.; Yan, C.; Wu, X.; Chen, F. Research on the Matthews Correlation Coefficients Metrics of Personalized Recommendation Algorithm Evaluation. Int. J. Hybrid Inf. Technol.; 2015; 8, pp. 163-172. [DOI: https://dx.doi.org/10.14257/ijhit.2015.8.1.14]

52. Krichene, W.; Rendle, S. On Sampled Metrics for Item Recommendation. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining; Virtual Event, CA, USA, 23 August 2020; pp. 1748-1757.

53. Hsu, F.-M.; Lin, Y.-T.; Ho, T.-K. Design and Implementation of an Intelligent Recommendation System for Tourist Attractions: The Integration of EBM Model, Bayesian Network and Google Maps. Expert Syst. Appl.; 2012; 39, pp. 3257-3264. [DOI: https://dx.doi.org/10.1016/j.eswa.2011.09.013]

54. Parra, D.; Sahebi, S. Recommender Systems: Sources of Knowledge and Evaluation Metrics. Advanced Techniques in Web Intelligence-2; Velásquez, J.D.; Palade, V.; Jain, L.C. Studies in Computational Intelligence Springer: Berlin/Heidelberg, Germany, 2013; Volume 452, pp. 149-175. ISBN 978-3-642-33325-5

55. Kawasaki, M.; Hasuike, T. A Recommendation System by Collaborative Filtering Including Information and Characteristics on Users and Items. Proceedings of the 2017 IEEE Symposium Series on Computational Intelligence; Honolulu, HI, USA, 27 November–1 December 2017; pp. 1-8.

56. Hendler, J. Web 3.0 emerging. Computer; 2009; 42, pp. 111-113. [DOI: https://dx.doi.org/10.1109/MC.2009.30]

57. Di Noia, T.; Mirizzi, R.; Ostuni, V.C.; Romito, D. Exploiting the Web of Data in Model-Based Recommender Systems. Proceedings of the Sixth ACM Conference on Recommender Systems—RecSys ’12; New York, NY, USA, 9–13 September 2012; 253.

58. Lika, B.; Kolomvatsos, K.; Hadjiefthymiades, S. Facing the Cold Start Problem in Recommender Systems. Expert Syst. Appl.; 2014; 41, pp. 2065-2073. [DOI: https://dx.doi.org/10.1016/j.eswa.2013.09.005]

59. Gündüz, S.; Özsu, M.T. A web page prediction model based on click-stream tree representation of user behavior. Proceedings of the 9th ACM International Conference on Knowledge Discovery and Data Mining (KDD); Washington, DC, USA, 24–27 August 2003; pp. 535-540.

60. Gómez-Pérez, A.; Corcho, O. Ontology languages for the Semantic Web. IEEE Intell. Syst.; 2002; 17, pp. 54-60. [DOI: https://dx.doi.org/10.1109/5254.988453]

61. Yu, Eunji; Kim, Jung-Cheol; Lee, Choon Yeul; Kim, Namgyu Using Ontologies for Semantic Text Mining. J. Inf. Syst.; 2012; 21, pp. 137-161. [DOI: https://dx.doi.org/10.5859/KAIS.2012.21.3.137]

62. Noy, N.F.; McGuinness, D.L. Ontology Development 101: A Guide to Creating Your First Ontology; Stanford University: Stanford, CA, USA, 2001.

63. Abowd, G.D.; Dey, A.K.; Brown, P.J.; Davies, N.; Smith, M.; Steggles, P. Towards a Better Understanding of Context and Context-Awareness. Handheld and Ubiquitous Computing; Gellersen, H.-W. Lecture Notes in Computer Science Springer: Berlin/Heidelberg, Germany, 1999; Volume 1707, pp. 304-307. ISBN 978-3-540-66550-2

64. Zadeh, L.A. The Concept of a Linguistic Variable and Its Application to Approximate Reasoning—I. Inf. Sci.; 1975; 8, pp. 199-249. [DOI: https://dx.doi.org/10.1016/0020-0255(75)90036-5]

65. Porcel, C.; Herrera-Viedma, E. Dealing with Incomplete Information in a Fuzzy Linguistic Recommender System to Disseminate Information in University Digital Libraries. Knowl.-Based Syst.; 2010; 23, pp. 32-39. [DOI: https://dx.doi.org/10.1016/j.knosys.2009.07.007]

66. Trstenjak, B.; Mikac, S.; Donko, D. KNN with TF-IDF Based Framework for Text Categorization. Procedia Eng.; 2014; 69, pp. 1356-1364. [DOI: https://dx.doi.org/10.1016/j.proeng.2014.03.129]

67. Guo, G.; Wang, H.; Bell, D.; Bi, Y.; Greer, K. KNN Model-Based Approach in Classification. On The Move to Meaningful Internet Systems 2003: CoopIS, DOA, and ODBASE.; Meersman, R.; Tari, Z.; Schmidt, D.C. Lecture Notes in Computer Science Springer: Berlin/Heidelberg, Germany, 2003; Volume 2888, pp. 986-996. ISBN 978-3-540-20498-5

68. Jannach, D.; Lerche, L.; Kamehkhosh, I.; Jugovac, M. What Recommenders Recommend: An Analysis of Recommendation Biases and Possible Countermeasures. User Model User-Adap. Inter.; 2015; 25, pp. 427-491. [DOI: https://dx.doi.org/10.1007/s11257-015-9165-3]

69. Shepitsen, A.; Gemmell, J.; Mobasher, B.; Burke, R. Personalized Recommendation in Social Tagging Systems Using Hierarchical Clustering. Proceedings of the 2008 ACM Conference on Recommender Systems—RecSys ’08; Lausanne, Switzerland, 23–25 October 2008; 259.

70. Bilge, A.; Polat, H. A Comparison of Clustering-Based Privacy-Preserving Collaborative Filtering Schemes. Appl. Soft Comput.; 2013; 13, pp. 2478-2489. [DOI: https://dx.doi.org/10.1016/j.asoc.2012.11.046]

71. Gan, G.; Ma, C.; Wu, J. Data Clustering: Theory, Algorithms, and Applications; Society for Industrial and Applied Mathematics: Philadelphia, VA, USA, 2007.

72. Gong, S. A Collaborative Filtering Recommendation Algorithm Based on User Clustering and Item Clustering. J. Softw.; 2010; 5, pp. 745-752. [DOI: https://dx.doi.org/10.4304/jsw.5.7.745-752]

73. Mnih, A.; Salakhutdinov, R.R. Probabilistic matrix factorization. Advances in Neural Information Processing Systems; IEEE: Piscatawy, NJ, USA, 2008; pp. 1257-1264.

74. Koren, Y.; Bell, R.; Volinsky, C. Matrix Factorization Techniques for Recommender Systems. Computer; 2009; 42, pp. 30-37. [DOI: https://dx.doi.org/10.1109/MC.2009.263]

75. Kim, D.; Park, C.; Oh, J.; Lee, S.; Yu, H. Convolutional Matrix Factorization for Document Context-Aware Recommendation. Proceedings of the 10th ACM Conference on Recommender Systems; Boston, MA, USA, 15–19 September 2016; pp. 233-240.

76. Koren, Y. Factorization meets the neighborhood: A multifaceted collaborative filtering model. Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; New York, NY, USA, 24–27 August 2008; pp. 426-434.

77. Ma, H.; Yang, H.; Lyu, M.R.; King, I. SoRec: Social Recommendation Using Probabilistic Matrix Factorization. Proceedings of the 17th ACM Conference on Information and Knowledge Mining—CIKM ’08; Napa Valley, CA, USA, 26–30 October 2008; 931.

78. Kim, Y.; Shim, K. TWILITE: A Recommendation System for Twitter Using a Probabilistic Model Based on Latent Dirichlet Allocation. Inf. Syst.; 2014; 42, pp. 59-77. [DOI: https://dx.doi.org/10.1016/j.is.2013.11.003]

79. Zhang, Y. GroRec: A Group-Centric Intelligent Recommender System Integrating Social, Mobile and Big Data Technologies. IEEE Trans. Serv. Comput.; 2016; 9, pp. 786-795. [DOI: https://dx.doi.org/10.1109/TSC.2016.2592520]

80. Seo, Y.-S.; Huh, J.-H. GUI-Based Software Modularization through Module Clustering in Edge Computing Based IoT Environments. J. Ambient Intell. Human. Comput.; 2019; pp. 1-15. Available online: https://link.springer.com/article/10.1007%2Fs12652-019-01455-3 (accessed on 6 September 2019).

81. Van den Oord, A.; Dieleman, S.; Schrauwen, B. Deep Content-Based Music Recommendation. Proceedings of the 26th International Conference on Neural Information Processing Systems; Red Hook, NY, USA, 15–19 December 2013; pp. 2643-2651.

82. Wang, H.; Wang, N.; Yeung, D.-Y. Collaborative Deep Learning for Recommender Systems. Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; Sydney, Australia, 10–13 August 2015; pp. 1235-1244.

83. He, X.; Liao, L.; Zhang, H.; Nie, L.; Hu, X.; Chua, T.-S. Neural Collaborative Filtering. Proceedings of the 26th International Conference on World Wide Web; Perth, Australia, 3–7 April 2017; International World Wide Web Conferences Steering Committee: pp. 173-182.

84. Gomez-Uribe, C.A.; Hunt, N. The Netflix Recommender System: Algorithms, Business Value, and Innovation. ACM Trans. Manage. Inf. Syst.; 2016; 6, pp. 1-19. [DOI: https://dx.doi.org/10.1145/2843948]

85. Koren, Y. Factor in the Neighbors: Scalable and Accurate Collaborative Filtering. ACM Trans. Knowl. Discov. Data; 2010; 4, pp. 1-24. [DOI: https://dx.doi.org/10.1145/1644873.1644874]

86. Bell, R.M.; Koren, Y. Lessons from the Netflix Prize Challenge. SIGKDD Explor. Newsl.; 2007; 9, pp. 75-79. [DOI: https://dx.doi.org/10.1145/1345448.1345465]

87. McFee, B.; Barrington, L.; Lanckriet, G. Learning Content Similarity for Music Recommendation. IEEE Trans. Audio Speech Lang. Process.; 2012; 20, pp. 2207-2218. [DOI: https://dx.doi.org/10.1109/TASL.2012.2199109]

88. Odić, A.; Tkalčič, M.; Tasič, J.F.; Košir, A. Predicting and Detecting the Relevant Contextual Information in a Movie-Recommender System. Interact. Comput.; 2013; 25, pp. 74-90. [DOI: https://dx.doi.org/10.1093/iwc/iws003]

89. Wang, X.; Wang, Y. Improving Content-Based and Hybrid Music Recommendation Using Deep Learning. Proceedings of the 22nd ACM International Conference on Multimedia; rlando, FL, USA, 3–7 November 2014; pp. 627-636.

90. Bogdanov, D.; Haro, M.; Fuhrmann, F.; Xambó, A.; Gómez, E.; Herrera, P. Semantic Audio Content-Based Music Recommendation and Visualization Based on User Preference Examples. Inf. Process. Manag.; 2013; 49, pp. 13-33. [DOI: https://dx.doi.org/10.1016/j.ipm.2012.06.004]

91. Colombo-Mendoza, L.O.; Valencia-García, R.; Rodríguez-González, A.; Alor-Hernández, G.; Samper-Zapater, J.J. RecomMetz: A Context-Aware Knowledge-Based Mobile Recommender System for Movie Showtimes. Expert Syst. Appl.; 2015; 42, pp. 1202-1222. [DOI: https://dx.doi.org/10.1016/j.eswa.2014.09.016]

92. Vall, A.; Dorfer, M.; Eghbal-zadeh, H.; Schedl, M.; Burjorjee, K.; Widmer, G. Feature-Combination Hybrid Recommender Systems for Automated Music Playlist Continuation. User Model User-Adap. Inter.; 2019; 29, pp. 527-572. [DOI: https://dx.doi.org/10.1007/s11257-018-9215-8]

93. Huh, J.-H.; Otgonchimeg, S.; Seo, K. Advanced Metering Infrastructure Design and Test Bed Experiment Using Intelligent Agents: Focusing on the PLC Network Base Technology for Smart Grid System. J. Supercomput.; 2016; 72, pp. 1862-1877. [DOI: https://dx.doi.org/10.1007/s11227-016-1672-4]

94. Lee, S.; Jeong, H.; Ko, H. Classical Music Specific Mood Automatic Recognition Model Proposal. Electronics; 2021; 10, 2489. [DOI: https://dx.doi.org/10.3390/electronics10202489]

95. Wang, Z.; Yu, X.; Feng, N.; Wang, Z. An Improved Collaborative Movie Recommendation System Using Computational Intelligence. J. Vis. Lang. Comput.; 2014; 25, pp. 667-675. [DOI: https://dx.doi.org/10.1016/j.jvlc.2014.09.011]

96. Sánchez-Moreno, D.; Gil González, A.B.; Muñoz Vicente, M.D.; López Batista, V.F.; Moreno García, M.N. A Collaborative Filtering Method for Music Recommendation Using Playing Coefficients for Artists and Users. Expert Syst. Appl.; 2016; 66, pp. 234-244. [DOI: https://dx.doi.org/10.1016/j.eswa.2016.09.019]

97. He, J.; Chu, W.W. A Social Network-Based Recommender System (SNRS). Data Mining for Social Network Data; Memon, N.; Xu, J.J.; Hicks, D.L.; Chen, H. Annals of Information Systems Springer: Boston, MA, USA, 2010; Volume 12, pp. 47-74. ISBN 978-1-4419-6286-7

98. Tsur, O.; Rappoport, A. What’s in a Hashtag? Content Based Prediction of the Spread of Ideas in Microblogging Communities. Proceedings of the Fifth ACM International Conference on Web Search and Data Mining—WSDM ’12; New York, NY, USA, 8–12 February 2012; 643.

99. Wang, Z.; Sun, L.; Zhu, W.; Yang, S.; Li, H.; Wu, D. Joint Social and Content Recommendation for User-Generated Videos in Online Social Network. IEEE Trans. Multimed.; 2013; 15, pp. 698-709. [DOI: https://dx.doi.org/10.1109/TMM.2012.2237022]

100. Kazienko, P.; Musial, K.; Kajdanowicz, T. Multidimensional Social Network in the Social Recommender System. IEEE Trans. Syst. Man Cybern. Part A Syst. Hum.; 2011; 41, pp. 746-759. [DOI: https://dx.doi.org/10.1109/TSMCA.2011.2132707]

101. Sedhain, S.; Menon, A.K.; Sanner, S.; Xie, L.; Braziunas, D. Low-rank linear cold-start recommendation from social data. Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence; Menlo Park, CA, USA, 4–9 February 2017; pp. 1502-1508.

102. Davoodi, E.; Kianmehr, K.; Afsharchi, M. A Semantic Social Network-Based Expert Recommender System. Appl. Intell.; 2013; 39, pp. 1-13. [DOI: https://dx.doi.org/10.1007/s10489-012-0389-1]

103. Kesorn, K.; Juraphanthong, W.; Salaiwarakul, A. Personalized Attraction Recommendation System for Tourists Through Check-In Data. IEEE Access; 2017; 5, pp. 26703-26721. [DOI: https://dx.doi.org/10.1109/ACCESS.2017.2778293]

104. Sun, Y.; Fan, H.; Bakillah, M.; Zipf, A. Road-Based Travel Recommendation Using Geo-Tagged Images. Comput. Environ. Urban Syst.; 2015; 53, pp. 110-122. [DOI: https://dx.doi.org/10.1016/j.compenvurbsys.2013.07.006]

105. Smirnov, A.V.; Kashevnik, A.M.; Ponomarev, A. Context-Based Infomobility System for Cultural Heritage Recommendation: Tourist Assistant—TAIS. Pers. Ubiquit. Comput.; 2017; 21, pp. 297-311. [DOI: https://dx.doi.org/10.1007/s00779-016-0990-0]

106. Tsai, C.-Y.; Chung, S.-H. A Personalized Route Recommendation Service for Theme Parks Using RFID Information and Tourist Behavior. Decis. Support Syst.; 2012; 52, pp. 514-527. [DOI: https://dx.doi.org/10.1016/j.dss.2011.10.013]

107. Ruotsalo, T.; Haav, K.; Stoyanov, A.; Roche, S.; Fani, E.; Deliai, R.; Mäkelä, E.; Kauppinen, T.; Hyvönen, E. SMARTMUSEUM: A Mobile Recommender System for the Web of Data. J. Web Semant.; 2013; 20, pp. 50-67. [DOI: https://dx.doi.org/10.1016/j.websem.2013.03.001]

108. Al-Hassan, M.; Lu, H.; Lu, J. A Semantic Enhanced Hybrid Recommendation Approach: A Case Study of e-Government Tourism Service Recommendation System. Decis. Support Syst.; 2015; 72, pp. 97-109. [DOI: https://dx.doi.org/10.1016/j.dss.2015.02.001]

109. Nilashi, M.; bin Ibrahim, O.; Ithnin, N.; Sarmin, N.H. A Multi-Criteria Collaborative Filtering Recommender System for the Tourism Domain Using Expectation Maximization (EM) and PCA–ANFIS. Electron. Commer. Res. Appl.; 2015; 14, pp. 542-562. [DOI: https://dx.doi.org/10.1016/j.elerap.2015.08.004]

110. Logesh, R.; Subramaniyaswamy, V. Exploring Hybrid Recommender Systems for Personalized Travel Applications. Cognitive Informatics and Soft Computing; Mallick, P.K.; Balas, V.E.; Bhoi, A.K.; Zobaa, A.F. Advances in Intelligent Systems and Computing Springer: Singapore, 2019; Volume 768, pp. 535-544. ISBN 9789811306167

111. Colomo-Palacios, R.; García-Peñalvo, F.J.; Stantchev, V.; Misra, S. Towards a Social and Context-Aware Mobile Recommendation System for Tourism. Pervasive Mob. Comput.; 2017; 38, pp. 505-515. [DOI: https://dx.doi.org/10.1016/j.pmcj.2016.03.001]

112. Galhotra, B.; Dewan, A. Impact of COVID-19 on Digital Platforms and Change in E-Commerce Shopping Trends. Proceedings of the 2020 Fourth International Conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud) (I-SMAC); Coimbatore, India, 7–9 October 2020; pp. 861-866.

113. Zhang, X.; Liu, H.; Chen, X.; Zhong, J.; Wang, D. A Novel Hybrid Deep Recommendation System to Differentiate User’s Preference and Item’s Attractiveness. Inf. Sci.; 2020; 519, pp. 306-316. [DOI: https://dx.doi.org/10.1016/j.ins.2020.01.044]

114. Lu, Y.; Zhao, L.; Wang, B. From Virtual Community Members to C2C E-Commerce Buyers: Trust in Virtual Communities and Its Effect on Consumers’ Purchase Intention. Electron. Commer. Res. Appl.; 2010; 9, pp. 346-360. [DOI: https://dx.doi.org/10.1016/j.elerap.2009.07.003]

115. Sulikowski, P.; Zdziebko, T.; Hussain, O.; Wilbik, A. Fuzzy Approach to Purchase Intent Modeling Based on User Tracking For E-Commerce Recommenders. Proceedings of the 2021 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE); Luxembourg, 11–14 July 2021; pp. 1-8.

116. Jiang, L.; Cheng, Y.; Yang, L.; Li, J.; Yan, H.; Wang, X. A trust-based collaborative filtering algorithm for E-commerce recommendation system. J. Ambient. Intell. Humaniz. Comput.; 2019; 10, pp. 3023-3034. [DOI: https://dx.doi.org/10.1007/s12652-018-0928-7]

117. Sarwar, B.; Karypis, G.; Konstan, J.; Reidl, J. Item-Based Collaborative Filtering Recommendation Algorithms. Proceedings of the Tenth International Conference on World Wide Web—WWW ’01; Hong Kong, China, 1–5 May 2001; pp. 285-295.

118. Hwangbo, H.; Kim, Y.S.; Cha, K.J. Recommendation System Development for Fashion Retail E-Commerce. Electron. Commer. Res. Appl.; 2018; 28, pp. 94-101. [DOI: https://dx.doi.org/10.1016/j.elerap.2018.01.012]

119. Ho, S.Y.; Bodoff, D.; Tam, K.Y. Timing of Adaptive Web Personalization and Its Effects on Online Consumer Behavior. Inf. Syst. Res.; 2011; 22, pp. 660-679. [DOI: https://dx.doi.org/10.1287/isre.1090.0262]

120. Guo, Y.; Yin, C.; Li, M.; Ren, X.; Liu, P. Mobile E-Commerce Recommendation System Based on Multi-Source Information Fusion for Sustainable e-Business. Sustainability; 2018; 10, 147. [DOI: https://dx.doi.org/10.3390/su10010147]

121. Sulikowski, P.; Zdziebko, T. Deep Learning-Enhanced Framework for Performance Evaluation of a Recommending Interface with Varied Recommendation Position and Intensity Based on Eye-Tracking Equipment Data Processing. Electronics; 2020; 9, 266. [DOI: https://dx.doi.org/10.3390/electronics9020266]

122. Castro-Schez, J.J.; Miguel, R.; Vallejo, D.; López-López, L.M. A Highly Adaptive Recommender System Based on Fuzzy Logic for B2C E-Commerce Portals. Expert Syst. Appl.; 2011; 38, pp. 2441-2454. [DOI: https://dx.doi.org/10.1016/j.eswa.2010.08.033]

123. Tröster, G. The Agenda of Wearable Healthcare. Yearb. Med. Inf.; 2005; 14, pp. 125-138. [DOI: https://dx.doi.org/10.1055/s-0038-1638446]

124. Apple. 2021; Use Emergency SOS on Your Apple Watch; Available online: https://support.apple.com/en-us/HT206983 (accessed on 27 April 2021).

125. Mishra, T.; Wang, M.; Metwally, A.A.; Bogu, G.K.; Brooks, A.W.; Bahmani, A.; Alavi, A.; Celli, A.; Higgs, E.; Dagan-Rosenfeld, O. et al. Pre-Symptomatic Detection of COVID-19 from Smartwatch Data. Nat. Biomed. Eng.; 2020; 4, pp. 1208-1220. [DOI: https://dx.doi.org/10.1038/s41551-020-00640-6] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/33208926]

126. Zheng, J.W.; Zhang, Z.B.; Wu, T.H.; Zhang, Y. A Wearable Mobihealth Care System Supporting Real-Time Diagnosis and Alarm. Med. Bio. Eng. Comput.; 2007; 45, pp. 877-885. [DOI: https://dx.doi.org/10.1007/s11517-007-0221-y] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/17619091]

127. Raghupathi, W.; Raghupathi, V. Big Data Analytics in Healthcare: Promise and Potential. Health Inf. Sci. Syst.; 2014; 2, 3. [DOI: https://dx.doi.org/10.1186/2047-2501-2-3] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/25825667]

128. Rehman, A.; Naz, S.; Razzak, I. Leveraging Big Data Analytics in Healthcare Enhancement: Trends, Challenges and Opportunities. Multimed. Syst.; 2021; 1, pp. 1-33. [DOI: https://dx.doi.org/10.1007/s00530-020-00736-8]

129. Duan, L.; Street, W.N.; Xu, E. Healthcare Information Systems: Data Mining Methods in the Creation of a Clinical Recommender System. Enterp. Inf. Syst.; 2011; 5, pp. 169-181. [DOI: https://dx.doi.org/10.1080/17517575.2010.541287]

130. Chen, J.; Li, K.; Rong, H.; Bilal, K.; Yang, N.; Li, K. A Disease Diagnosis and Treatment Recommendation System Based on Big Data Mining and Cloud Computing. Inf. Sci.; 2018; 435, pp. 124-149. [DOI: https://dx.doi.org/10.1016/j.ins.2018.01.001]

131. Thong, N.T.; Son, L.H. HIFCF: An Effective Hybrid Model between Picture Fuzzy Clustering and Intuitionistic Fuzzy Recommender Systems for Medical Diagnosis. Expert Syst. Appl.; 2015; 42, pp. 3682-3701. [DOI: https://dx.doi.org/10.1016/j.eswa.2014.12.042]

132. Ko, H.; Huh, J.-H. Electronic Solutions for Artificial Intelligence Healthcare. Electronics; 2021; 10, 2421. [DOI: https://dx.doi.org/10.3390/electronics10192421]

133. Iwendi, C.; Khan, S.; Anajemba, J.H.; Bashir, A.K.; Noor, F. Realizing an Efficient IoMT-Assisted Patient Diet Recommendation System Through Machine Learning Model. IEEE Access; 2020; 8, pp. 28462-28474. [DOI: https://dx.doi.org/10.1109/ACCESS.2020.2968537]

134. Yang, L.; Hsieh, C.-K.; Yang, H.; Pollak, J.P.; Dell, N.; Belongie, S.; Cole, C.; Estrin, D. Yum-Me: A Personalized Nutrient-Based Meal Recommender System. ACM Trans. Inf. Syst.; 2017; 36, pp. 1-31. [DOI: https://dx.doi.org/10.1145/3072614]

135. Sezgin, E.; Ozkan, S. A systematic literature review on Health Recommender Systems. Proceedings of the 2013 E-Health and Bioengineering Conference (EHB); Iasi, Romania, 21–23 November 2013; [DOI: https://dx.doi.org/10.1109/ehb.2013.6707249]

136. Sanchez Bocanegra, C.L.; Sevillano Ramos, J.L.; Rizo, C.; Civit, A.; Fernandez-Luque, L. HealthRecSys: A Semantic Content-Based Recommender System to Complement Health Videos. BMC Med. Inf. Decis. Mak.; 2017; 17, 63. [DOI: https://dx.doi.org/10.1186/s12911-017-0431-7]

137. Wiesner, M.; Pfeifer, D. Health Recommender Systems: Concepts, Requirements, Technical Basics and Challenges. IJERPH; 2014; 11, pp. 2580-2607. [DOI: https://dx.doi.org/10.3390/ijerph110302580] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/24595212]

138. Abbas, K.; Afaq, M.; Ahmed Khan, T.; Song, W.-C. A Blockchain and Machine Learning-Based Drug Supply Chain Management and Recommendation System for Smart Pharmaceutical Industry. Electronics; 2020; 9, 852. [DOI: https://dx.doi.org/10.3390/electronics9050852]

139. Chen, R.-C.; Huang, Y.-H.; Bau, C.-T.; Chen, S.-M. A Recommendation System Based on Domain Ontology and SWRL for Anti-Diabetic Drugs Selection. Expert Syst. Appl.; 2012; 39, pp. 3995-4006. [DOI: https://dx.doi.org/10.1016/j.eswa.2011.09.061]

140. Tikhomirov, V.; Dneprovskaya, N.; Yankovskaya, E. Three Dimensions of Smart Education. Smart Education and Smart e-Learning; Uskov, V.L.; Howlett, R.J.; Jain, L.C. Smart Innovation, Systems and Technologies Springer International Publishing: Cham, Switzerland, 2015; Volume 41, pp. 47-56. ISBN 978-3-319-19874-3

141. Lin, J.; Pu, H.; Li, Y.; Lian, J. Intelligent Recommendation System for Course Selection in Smart Education. Procedia Comput. Sci.; 2018; 129, pp. 449-453. [DOI: https://dx.doi.org/10.1016/j.procs.2018.03.023]

142. Zhu, Z.-T.; Yu, M.-H.; Riezebos, P. A Research Framework of Smart Education. Smart Learn. Environ.; 2016; 3, 4. [DOI: https://dx.doi.org/10.1186/s40561-016-0026-2]

143. Wan, S.; Niu, Z. An E-Learning Recommendation Approach Based on the Self-Organization of Learning Resource. Knowl.-Based Syst.; 2018; 160, pp. 71-87. [DOI: https://dx.doi.org/10.1016/j.knosys.2018.06.014]

144. Shu, J.; Shen, X.; Liu, H.; Yi, B.; Zhang, Z. A Content-Based Recommendation Algorithm for Learning Resources. Multimed. Syst.; 2018; 24, pp. 163-173. [DOI: https://dx.doi.org/10.1007/s00530-017-0539-8]

145. Chen, Y.; Li, X.; Liu, J.; Ying, Z. Recommendation System for Adaptive Learning. Appl. Psychol. Meas.; 2018; 42, pp. 24-41. [DOI: https://dx.doi.org/10.1177/0146621617697959]

146. Klašnja-Milićević, A.; Vesin, B.; Ivanović, M.; Budimac, Z. E-Learning Personalization Based on Hybrid Recommendation Strategy and Learning Style Identification. Comput. Educ.; 2011; 56, pp. 885-899. [DOI: https://dx.doi.org/10.1016/j.compedu.2010.11.001]

147. Dwivedi, P.; Bharadwaj, K.K. E-Learning Recommender System for a Group of Learners Based on the Unified Learner Profile Approach. Expert Syst.; 2015; 32, pp. 264-276. [DOI: https://dx.doi.org/10.1111/exsy.12061]

148. Wu, D.; Lu, J.; Zhang, G. A Fuzzy Tree Matching-Based Personalized E-Learning Recommender System. IEEE Trans. Fuzzy Syst.; 2015; 23, pp. 2412-2426. [DOI: https://dx.doi.org/10.1109/TFUZZ.2015.2426201]

149. Obeid, C.; Lahoud, I.; El Khoury, H.; Champin, P.-A. Ontology-Based Recommender System in Higher Education. Proceedings of the Companion Proceedings of The Web Conference 2018; Lyon, France, 23–27 April 2018; pp. 1031-1034.

150. Gulzar, Z.; Leema, A.A.; Deepak, G. PCRS: Personalized Course Recommender System Based on Hybrid Approach. Procedia Comput. Sci.; 2018; 125, pp. 518-524. [DOI: https://dx.doi.org/10.1016/j.procs.2017.12.067]

151. Esteban, A.; Zafra, A.; Romero, C. Helping University Students to Choose Elective Courses by Using a Hybrid Multi-Criteria Recommendation System with Genetic Optimization. Knowl.-Based Syst.; 2020; 194, 105385. [DOI: https://dx.doi.org/10.1016/j.knosys.2019.105385]

152. Serrano-Guerrero, J.; Herrera-Viedma, E.; Olivas, J.A.; Cerezo, A.; Romero, F.P. A Google Wave-Based Fuzzy Recommender System to Disseminate Information in University Digital Libraries 2.0. Inf. Sci.; 2011; 181, pp. 1503-1516. [DOI: https://dx.doi.org/10.1016/j.ins.2011.01.012]

153. Tejeda-Lorente, Á.; Porcel, C.; Peis, E.; Sanz, R.; Herrera-Viedma, E. A Quality Based Recommender System to Disseminate Information in a University Digital Library. Inf. Sci.; 2014; 261, pp. 52-69. [DOI: https://dx.doi.org/10.1016/j.ins.2013.10.036]

154. He, Q.; Pei, J.; Kifer, D.; Mitra, P.; Giles, L. Context-aware citation recommendation. Proceedings of the 19th International Conference on World Wide Web—WWW ’10; Raleigh, NC, USA, 26–30 April 2010; [DOI: https://dx.doi.org/10.1145/1772690.1772734]

155. Wang, D.; Liang, Y.; Xu, D.; Feng, X.; Guan, R. A Content-Based Recommender System for Computer Science Publications. Knowl.-Based Syst.; 2018; 157, pp. 1-9. [DOI: https://dx.doi.org/10.1016/j.knosys.2018.05.001]

156. Park, N.; Roman, R.; Lee, S.; Chung, J.E. User Acceptance of a Digital Library System in Developing Countries: An Application of the Technology Acceptance Model. Int. J. Inf. Manag.; 2009; 29, pp. 196-209. [DOI: https://dx.doi.org/10.1016/j.ijinfomgt.2008.07.001]

157. Jeong, H. An Investigation of User Perceptions and Behavioral Intentions towards the E-Library. Libr. Collect. Acquis. Tech. Serv.; 2011; 35, pp. 45-60. [DOI: https://dx.doi.org/10.1080/14649055.2011.10766298]

158. Achakulvisut, T.; Acuna, D.E.; Ruangrong, T.; Kording, K. Science Concierge: A Fast Content-Based Recommendation System for Scientific Publications. PLoS ONE; 2016; 11, e0158423. [DOI: https://dx.doi.org/10.1371/journal.pone.0158423]

159. Färber, M.; Jatowt, A. Citation Recommendation: Approaches and Datasets. Int. J. Digit. Libr.; 2020; 21, pp. 375-405. [DOI: https://dx.doi.org/10.1007/s00799-020-00288-2]

160. Brian, D. Netflix Subscriber and Growth Statistics: How Many People Watch Netflix in 2021?; Available online: https://backlinko.com/netflix-users (accessed on 25 February 2021).

161. Statista. Disney+’s Number of Subscribers Worldwide from 1st Quarter 2020 to 2nd Quarter 2021; Available online: https://www.statista.com/statistics/1095372/disney-plus-number-of-subscribers-us/ (accessed on 19 August 2021).

162. BBC News. Netflix Launches UK Film and TV Streaming Service; Available online: https://web.archive.org/web/20120109175608/http://www.bbc.co.uk/news/technology-16467432 (accessed on 9 January 2012).

163. PR Newswire. Netflix Launches in Sweden, Denmark, Norway and Finland; Available online: https://web.archive.org/web/20141127211314/http://www.prnewswire.com/news-releases/netflix-launches-in-sweden-denmark-norway-and-finland-174749581.html (accessed on 27 November 2014).

164. Variety. Netflix’s ‘House of Cards’ Falls Prey to Piracy; Available online: https://variety.com/2013/digital/news/pirates-swarm-netflixs-house-of-cards-1200333171/ (accessed on 4 April 2013).

165. Gupta, G.; Singharia, K. Consumption of OTT Media Streaming in COVID-19 Lockdown: Insights from PLS Analysis. Vision; 2021; 25, pp. 36-46. [DOI: https://dx.doi.org/10.1177/0972262921989118]

166. RyanMegAnousha. Shaken Studios. Empty Theaters. What Hollywood Lost during the Pandemic; Available online: https://www.latimes.com/entertainment-arts/business/story/2020-12-09/everything-hollywood-lost-during-the-pandemic (accessed on 9 December 2020).

167. Insider Intelligence. Top OTT VOD Streaming Services in 2021 by Viewer Count and Growth; Available online: https://www.insiderintelligence.com/insights/ott-vod-video-streaming-services/ (accessed on 21 July 2021).

168. Statista. Facebook Keeps on Growing; Available online: https://www.statista.com/chart/10047/facebooks-monthly-active-users/ (accessed on 4 February 2021).

169. Prabhakaran. 50+ Instagram Stats Everyone Must Know in 2020; Available online: https://www.connectivasystems.com/instagram-stats-2020/ (accessed on 1 April 2020).

170. Statista. Number of Monthly Active Twitter Users Worldwide from 1st Quarter 2010 to 1st Quarter 2019; Available online: https://www.statista.com/statistics/282087/number-of-monthly-active-twitter-users/ (accessed on 31 December 2021).

171. Hughes, D.J.; Rowe, M.; Batey, M.; Lee, A. A Tale of Two Sites: Twitter vs. Facebook and the Personality Predictors of Social Media Usage. Comput. Hum. Behav.; 2012; 28, pp. 561-569. [DOI: https://dx.doi.org/10.1016/j.chb.2011.11.001]

172. Highfield, T.; Leaver, T. A Methodology for Mapping Instagram Hashtags. First Monday; 2014; 20, pp. 1-11. [DOI: https://dx.doi.org/10.5210/fm.v20i1.5563]

173. Sarah, F. The Inside Story of How Facebook Acquired Instagram; Available online: https://onezero.medium.com/the-inside-story-of-how-facebook-acquired-instagram-318f244f1283 (accessed on 5 August 2020).

174. Dave, S. Chart of the Day: Instagram Is Now Bigger Than Twitter; Available online: https://www.businessinsider.com.au/chart-of-the-day-instagram-is-now-bigger-than-twitter-2014-12 (accessed on 11 December 2014).

175. Li, S.; Takahashi, S.; Yamada, K.; Takagi, M.; Sasaki, J. Analysis of SNS Photo Data Taken by Foreign Tourists to Japan and a Proposed Adaptive Tourism Recommendation System. Proceedings of the 2017 International Conference on Progress in Informatics and Computing (PIC); Nanjing, China, 15–17 December 2017; pp. 323-327.

176. Chao, L.; Jian, Y.; Xiang, L.; Hui, C.J. A Social Network System Oriented Hybrid Recommendation Model. Proceedings of the 2012 2nd International Conference on Computer Science and Network Technology; Changchun, China, 29–31 December 2012; pp. 901-906.

177. Jeong, O.-R. SNS-Based Recommendation Mechanisms for Social Media. Multimed Tools Appl; 2015; 74, pp. 2433-2447. [DOI: https://dx.doi.org/10.1007/s11042-014-1884-7]

178. Statista. New Year’s Peak Illustrates Airbnb’s Growing Stature; Available online: https://lb-aps-frontend.statista.com/chart/20386/guests-staying-at-airbnb-appartments-on-new-years-eve/ (accessed on 2 January 2020).

179. Siegler, M.G. Airbnb Cozies Up to Facebook to Help You Feel More at Home When Away from Home; Available online: https://techcrunch.com/2011/05/09/airbnb-social-connections/ (accessed on 10 May 2011).

180. Kontogianni, A.; Alepis, E. Smart Tourism: State of the Art and Literature Review for the Last Six Years. Array; 2020; 6, 100020. [DOI: https://dx.doi.org/10.1016/j.array.2020.100020]

181. Balasaraswathi, M.; Srinivasan, K.; Udayakumar, L.; Sivasakthiselvan, S.; Sumithra, M.G. Big Data Analytic of Contexts and Cascading Tourism for Smart City. Mater. Today Proc.; 2020; S2214785320377129. [DOI: https://dx.doi.org/10.1016/j.matpr.2020.10.132]

182. Yoo, C.W.; Goo, J.; Huang, C.D.; Nam, K.; Woo, M. Improving Travel Decision Support Satisfaction with Smart Tourism Technologies: A Framework of Tourist Elaboration Likelihood and Self-Efficacy. Technol. Forecast. Soc. Chang.; 2017; 123, pp. 330-341. [DOI: https://dx.doi.org/10.1016/j.techfore.2016.10.071]

183. Tribe, J.; Mkono, M. Not Such Smart Tourism? The Concept of e-Lienation. Ann. Tour. Res.; 2017; 66, pp. 105-115. [DOI: https://dx.doi.org/10.1016/j.annals.2017.07.001]

184. Kim, I.-S.; Jeong, C.-S.; Jung, T.-W.; Kang, J.-K.; Jung, K.-D. AR Tourism Recommendation System Based on Character-Based Tourism Preference Using Big Data. Int. J. Internet Broadcasting Commun.; 2021; 13, pp. 61-68. [DOI: https://dx.doi.org/10.7236/IJIBC.2021.13.1.61]

185. Statista. Net Revenue of Amazon from 1st Quarter 2007 to 1st Quarter 2021; Available online: https://www.statista.com/statistics/273963/quarterly-revenue-of-amazoncom/ (accessed on 19 July 2021).

186. Statista. Annual Revenue of Alibaba Group from Financial Year 2011 to 2021; Available online: https://www.statista.com/statistics/225614/net-revenue-of-alibaba/ (accessed on 15 September 2021).

187. Linden, G.; Smith, B.; York, J. Amazon.Com Recommendations: Item-to-Item Collaborative Filtering. IEEE Internet Comput.; 2003; 7, pp. 76-80. [DOI: https://dx.doi.org/10.1109/MIC.2003.1167344]

188. Smith, B.; Linden, G. Two Decades of Recommender Systems at Amazon.Com. IEEE Internet Comput.; 2017; 21, pp. 12-18. [DOI: https://dx.doi.org/10.1109/MIC.2017.72]

189. Wang, J.; Huang, P.; Zhao, H.; Zhang, Z.; Zhao, B.; Lee, D.L. Billion-Scale Commodity Embedding for E-Commerce Recommendation in Alibaba. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining; London, UK, 19–23 August 2018; pp. 839-848.

190. Chen, Q.; Zhao, H.; Li, W.; Huang, P.; Ou, W. Behavior Sequence Transformer for E-Commerce Recommendation in Alibaba. Proceedings of the 1st International Workshop on Deep Learning Practice for High-Dimensional Sparse Data; Anchorage, Alaska, 5 August 2019; pp. 1-4.

191. Statista. Apple Watch Installed Base Worldwide from 2015 to 2020; Available online: https://www.statista.com/statistics/1221051/apple-watch-users-worldwide/ (accessed on 17 March 2021).

192. Statista. Fitbit’s Community Grows Amid Stalling Device Sales; Available online: https://www.statista.com/chart/13083/worldwide-fitbit-shipments/ (accessed on 4 August 2020).

193. Dino, G. Huffpost. Fitbit Recalls Fitbit Force after Complaints of Severe Rashes; Available online: https://www.huffpost.com/entry/fitbit-force-recall_n_4832771 (accessed on 21 February 2014).

194. Mallory, H. Apple Study Finds Watch Can Detect More Types of Irregular Heartbeats; Available online: https://www.mobihealthnews.com/news/apple-study-finds-watch-can-detect-more-types-irregular-heartbeats (accessed on 29 September 2021).

195. Statista. Global Apple Watch shipments from 2017 to 2019. Available online: https://www.statista.com/statistics/526005/global-apple-watch-shipments-forecast/ (accessed on 31 December 2021).

196. Zoë, B. Apple Topped the Swiss in Holiday Watch Shipments; Available online: https://www.businessinsider.com/apple-watch-sells-more-than-swiss-watches-charts-2018-2 (accessed on 13 February 2018).

197. Elearningfeeds. eLearning Statistics and Trends: 2020; Available online: https://elearningfeeds.com/elearning-statistics-and-trends-2020/ (accessed on 14 December 2020).

198. Miran, P.; Sunil, J.; Seungin, K. A Study on Application and Prospect with Prevalence Tablet PC -An Approach of Humanities, Science and Design. J. Korea Des. Knowl.; 2011; 18, pp. 84-93. Available online: http://www.dbpia.co.kr/journal/articleDetail?nodeId=NODE01772204 (accessed on 30 June 2011).

199. Jisu, K.; Ko, H. Comparison of the Convolutional Neural Network models with an art dataset. Proceedings of the 17th International Conference on Multimedia Information Technology and Applications (MITA 2021); Seogwipo, Korea, 5–7 July 2021; pp. 117-120.