Developing a Framework for Evaluating and Predicting Management Innovation in Public Research Institutions

Abstract

As the external environment changes rapidly, organizations need management innovation to adapt to and exploit change as an opportunity. To innovate, it is necessary to evaluate management innovation, because if an organization can measure the degree of management innovation, it can also achieve it. Moreover, if management innovation is predictable, profits can be maximized, and costs can be minimized by allocating efficient resources and establishing appropriate strategies. Therefore, this study attempts to predict the management innovation in public research institutions. Basic data mining and ensemble data mining techniques were used for the prediction. This analysis targeted public research institutes in South Korea. The results showed that the predictive power of public research institutions with high innovation was high. This study suggests that management innovation can be predicted in highly innovative public research institutions. Furthermore, this study’s framework can be applied to other industries.

1. Introduction

Innovation is one of the most addressed topics among practitioners as well as academics. Innovation is a key component of an organization’s competitive advantage [1]. An organization with successful innovation can gain a significant competitive advantage [2]. Sustainable innovation is a solution for organizations to cope with rapidly changing business environments [3]. Therefore, many organizations are striving to innovate. Sustainable innovation is the process of pursuing sustainable development, and sustainable development provides a positive social and economic impact [4]. However, surprisingly, most research has tended to address innovation as the development of new technologies, services or products [5]. Consequently, technological innovation has dominated innovation research, with related concepts, such as product development, radical and incremental innovation, diffusion and adoption, receiving the most attention [1]. In summary, much of the research to date has been devoted to understanding how companies can foster technological innovation [6]. However, rising trade barriers, rising transaction costs and a global market downturn are forcing companies to find other areas to innovate as a means to gain and maintain a competitive advantage [7]. As competition intensifies and technological change accelerates, companies should consider non-technological innovations that are more difficult to replicate and can contribute to a long-term competitive advantage [8]. This requires not only innovations in new products and technologies but also management innovations such as changes in the nature of management within the company [1]. From this point of view, management innovation enables companies to gain a competitive advantage through non-technical innovation [8].

Management innovation refers to the introduction of management practices, processes and structures to achieve organizational objectives [9]. Recent research highlighted the importance of management innovation to company performance, both as a complement to technological innovation [10] and as an independent phenomenon [11]. Management innovation can include changing organizational form, applying new management practices and developing talent, with the effect of leveraging a company’s knowledge base and improving organizational performance [12]. Management transformation involves changing the ‘how and what’ that managers set as the direction, make decisions, coordinate activities and motivate people [13]. These changes are manifested by new management practices, structures or processes [14]. In conclusion, management innovation is usually aimed at increasing the efficiency of internal organizational processes [15]. They are context-dependent [11], are ambiguous and difficult to replicate and represent an important source of competitive advantage [16]. Success is determined by how these management innovations are applied to the unique context of an organization, and, therefore, the evaluation of management innovations to confirm success is becoming more important [17]. In summary, management innovation aims to increase the efficiency of an organization’s internal process, and the evaluation of management innovation is essential for successful management innovation. The evaluation of innovation is broadly important for two reasons. First, it helps identify the expected benefits. Second, it helps companies or organizations measure their ability to remain competitive and acquire competitiveness through sustainable innovation [18]. Recent research on innovation evaluation is developing into research that predicts innovation as well as measurement [19]. Predicting innovation in an organization or a firm is imperative because it helps efficiently allocate resources through predictions, and the establishment of appropriate strategies can maximize profits and minimize damage [20]. To evaluate management innovation, it is necessary to define management innovation and explore the elements that comprise it. However, the factors for evaluating management innovation varied across studies, so the criteria were also formed differently. Therefore, there was a limit where innovation was evaluated differently, even in the same group according to the set evaluation criteria [21]. In this study, after analyzing the performance to overcome the limitations of existing studies, the degree of management innovation is evaluated by decomposing the formula.

The methodology used as a methodology to evaluate management innovation through a formula for performance is the Malmquist productivity index (MPI). The MPI is mainly used to analyze productivity, and decomposing the formula for productivity can identify technological innovation and management innovation. The Malmquist productivity index evaluates management innovation by measuring internal process efficiency changes [22,23]. Based on the evaluation of management innovation, management innovation is predicted through data mining. Data mining is the most commonly used quantitative method for predicting the occurrence of events through quantitative methods [24]. Data mining involves a series of processes to extract useful knowledge by exploring and modeling previously unknown patterns or rules in large amounts of data using complex statistical analyses or model-building techniques. It is mainly used to make predictions, such as demand forecasting in business management, financial forecasting in economics and weather forecasting in climatology [25].

The subject of analysis in this study was set as public research institutes in Korea. In various fields, management innovation in public research institutes is emerging as an important issue. Public research institutes prioritize the production of the public good of scientific knowledge and the knowledge necessary for a nation’s competitive advantage, without maximizing profits beyond market logic [26]. This innovation by public research institutes plays an important role in national growth and securing technological capabilities [27]. Therefore, various countries are striving to innovate through public research institutes [28]. In addition, since public research institutes are operated using taxes, management innovation is desperately needed. In addition, audit organizations are encouraging public research institutes to achieve management innovation rather than lax operation.

In summary, the purpose of this study is as follows. First, the management innovation of Korean public research institutes is evaluated. In order to overcome the limitations of existing research, this study evaluates management innovation using the MPI. Second, based on the evaluated management innovation, various data are synthesized to predict the management innovation of public research institutes in Korea. Data mining techniques are utilized to make predictions; in addition, this study aims to identify the data mining technique with the highest predictive power using more than 40 data mining techniques, rather than focusing on a specific set.

Therefore, this study evaluates and predicts management innovation in Korean public research institutes. The implications that can be obtained through these studies are as follows. (1) This study evaluates and predicts management innovation to gain a competitive advantage, and the methodology used in this study is applicable to other domains. (2) It is possible to provide policy implications for policymakers by evaluating and predicting management innovation in public research institutes in Korea. Based on this, it is possible to establish a quick and accurate strategy for allocating resources efficiently.

The remainder of this paper is organized as follows. Section 2 introduces a literature review and related studies. Section 3 describes the research framework and explains the techniques used in this study. Section 4 presents the results of the research framework. Building on these results, Section 5 proposes further explorations and explains their implications. Lastly, the limitations of this study are described.

2. Literature Review and Related Work

2.1. Management Innovation

Innovation is seen as a major driver of progress and prosperity at the individual firm level and for the economy as a whole [29]. In particular, the ability to innovate is increasingly central, as studies have shown that innovative firms tend to exhibit higher profitability, greater market value, better credit ratings and greater viability [30]. The old paradigm of industrial innovation based on technological inventions today entails organizational innovation [31], management innovation [32,33] and many other types of innovation.

However, most existing research has addressed how organizations can foster technological innovation [6]. Technological innovation can be defined at various levels [34]. Technological innovation involves the creation and adoption of new ideas for physical equipment, technologies, tools or systems that extend a firm’s capabilities into operational processes and production systems [10,34]. Therefore, at this level, it can be defined as the generation and adoption of new ideas for operational processes, production systems, products and services [1]. However, these technological innovations are accelerating, and non-technical innovations are attracting attention because they are relatively easy to replicate [8]. Non-technical forms of innovation include administrative innovation, organizational innovation and management innovation [1]. The scope of administrative innovation, organizational innovation and management innovation is not the same. Administrative innovation has a narrower focus than organizational innovation [14]. Compared to managerial innovation, administrative innovation generally narrows the scope of innovation centered around resource allocation, organizational structure and human resource policy [35], excluding operational and marketing management [9]. The concept of management innovation is more inclusive as it refers to a change in the way business is conducted [33]. Additionally, organizational innovation is used as a concept encompassing management innovation and technological innovation [36].

Management innovation reflects changes in the way management tasks are performed, including traditional processes (i.e., departures from existing processes). Management innovation also includes changing organizational form, applying new management practices and developing talent, with the effect of leveraging the firm’s knowledge base and improving organizational performance [1,12]. Through this, the goal of management innovation is to catch up with external innovations and increase the efficiency of internal processes. Through this, organizations can increase productivity and competitiveness [33] and enable continuous growth.

In summary, technological innovation means a change in external technology, and management innovation means how much it responds to the degree of external technological innovation. In other words, technological innovation occurs first, and management innovation is made to meet the degree of technological innovation. Where there are low levels of management innovation, adjustments in management practices, processes, structures and techniques are not adequately aligned with new technological knowledge in ways that enable the firm to achieve innovation success [1].

By decomposing the formula of the MPI, the same value of technological innovation and management innovation can be obtained. Productivity is determined by multiplying the catch-up index and the frontier shift index [22]. The catch-up index is an indicator that measures how far an organization is from the production frontier through internal coordination or innovative operation and learning and knowledge dissemination effects, market competitiveness, cost structure and improved facility utilization [23]. The frontier shift index changes through changes in the technological system or social system, such as technological innovation or policy change. In particular, many changes occur through the development of technology. Therefore, in this study, the productivity of public research institutes is identified through the MPI, and management innovation is evaluated based on the identified productivity.

2.2. Innovation in Research Institutions

The spread of innovation is a prerequisite for a long-term increase in production and wealth [37]. To favor diffusion, the organizational environment must allow knowledge to cross institutional borders freely and not remain confined to its point of origin. Innovation activities occur mainly in research institutes. Innovation in research creates more added value than innovation in service or manufacturing industries [38]. Innovation is a powerful tool to create value by enhancing efficiency and addressing societal challenges. By improving resource efficiency and developing new solutions to pressing challenges, businesses and organizations can benefit from innovation.

Previous studies have focused on public research institutions. There are several reasons for this. These institutions play a key role in science and technology policies [39] and the products they produce provide the country with a comparative advantage [40]. A nation’s intellectual asset is maintained or increased by continuously conducting research that has been abandoned by several research institutes because it is relatively unprofitable [41]. In particular, public research institutes play a large role as cradles for inventions and innovation creation, improving a nation’s sustainable competitiveness by producing and disseminating scientific research within the national economic system [28]. Therefore, many researchers have conducted studies on innovation in public research institutes [42]. The innovation of The Industrial Technology Research Institute (ITRI), Taiwan’s largest research institute, was analyzed, and the relationship between innovation in research institutes and industrial innovation was found to have a very positive effect [43]. In addition, as a result of examining the relationship between Chinese research institutes and local industries, the higher the innovation of local industries, the higher the innovation of research institutes [44]. In summary, innovation in research institutes can be considered important, as it has been found to have a positive impact on various macroeconomic organizations.

2.3. Prediction Using Data Mining

Data mining is the most commonly used quantitative methodology for predicting event occurrences. It is used in many fields to predict the probability of an event occurring by analyzing data patterns. It identifies data patterns using various techniques, including search and prediction, through characterization and differentiation, frequent pattern mining, association analysis, correlation, classification and regression, cluster analysis, outliers, artificial neural networks, genetic algorithms, decision trees and connection analysis [45]. To date, research on predicting efficiency and productivity through data mining has mainly been conducted in the financial and educational industries. Public research institutes are yet to predict innovation. Owing to time and physical constraints, existing studies have compared and evaluated limited data mining techniques rather than attempting predictions using these various techniques.

Data mining has also been widely used to predict innovations in many fields. It has also been used to predict product innovation [46,47,48] and also to predict innovation within an organization [49]. It is also used to predict service innovation of companies [50,51], and data mining is also used to predict the innovation in companies [52].

In this study, data mining techniques were used to predict the target variables. Data mining can be largely divided into traditional data mining techniques and relatively recent techniques [53]. Traditional data mining techniques include decision trees and regression analysis, whereas relatively recent data mining techniques include Support Vector Machines (SVMs) and artificial neural networks. The main decision tree techniques include J48, Random Tree, Hoeffding Tree, and REP Tree; however, most studies used only one or two techniques to compare and analyze predictive power. Therefore, this study compared and analyzed the predictive power of various decision tree techniques, which are traditional data mining methods. Further, the predictive powers of regression analysis, SVMs and artificial neural networks were compared. In addition, there are studies using traditional data mining techniques and relatively recent data mining techniques; however, those that amplify and analyze results using ensemble methods are insufficient. An ensemble is a method for amplifying the resulting value by combining various classifiers. The most commonly used techniques are Boosting, Bagging and Random Subspace. The use of ensemble techniques has the advantage of increasing the predictive power of existing data mining techniques, but there is insufficient research on which method further amplifies the predictive power, and the productivity prediction of public research institutes has not been studied. In this study, 41 data mining techniques were used for the analysis. It primarily uses data mining techniques based on decision trees, regression analysis, SVM and artificial neural networks.

Logistic regression is a probability model that predicts the likelihood of an event using a linear combination of independent variables. Logistic regression is not significantly different from general linear regression. Similar to a general regression analysis, the relationship between the independent and dependent variables is expressed as a function to predict future events. However, unlike linear regression, logistic regression uses categorical data as dependent variables.

Artificial neural networks are statistical learning algorithms inspired by neural networks in biology (especially the brain among the animal central nervous systems), machine learning and cognitive science. Artificial neural networks refer to an entire model in which artificial neurons (nodes) that form a network through a combination of synapses change the binding strength of synapses through learning.

Support Vector Machine (SVM) is a field in machine learning that is used as a supervised learning model for data analysis. It is primarily used for classification and regression analyses. When a set of data is provided, the SVM algorithm generates a non-probabilistic binary linear classification model that determines which group will be included in the new data input based on a given set of data. The generated classification model is represented by the boundaries between the data, and the SVM algorithm generates boundaries that can best classify most of the input data. They can be classified into two categories: linear and nonlinear.

A decision tree is a support tool that schematizes decision-making rules and their results into a tree structure. Decision trees are primarily used in operational science, especially in decision analysis, to identify strategies that can produce results closest to the goal. They are used in many fields because of their excellent visualization ability. Because decision trees are an old technique, many decision tree techniques that combine various analyses have been developed.

An ensemble model combines multiple classifiers through various methods to achieve better predictive power than a single classifier. Studies on such ensemble models have recently been conducted [54,55]. Ensemble models include Bagging, Boosting and Random Subspace.

Bagging is a method of making multiple copies of learning algorithms, learning each algorithm and combining the results. In other words, Bagging learns each of the n copied learning algorithms for the n datasets created by Bootstrapping, and the results are then combined in a manner, such as majority voting or simple abstraction. Bagging exhibits an effective performance when small changes in data have a large effect on the results of the learning algorithms. In summary, the use of unstable learning algorithms has a significant effect. A simple schematic is shown in Figure 1 below.

Boosting is a method of repeating the step of creating a new classification rule by performing intensive classification again to increase the classification accuracy of learning data that were incorrectly classified using automatic classification techniques. Adaptive Boost (AdaBoost) is mainly used during boosting, which is a method of weighting and learning specific patterns that contribute the most to causing performance degradation of learning results among learning datasets and combining the results step by step. A simple schematic is shown in Figure 2 below.

The Random Subspace technique is an ensemble technique proposed by Ho [56], which seeks to generate diversity in classifiers through changes in the learning data. However, unlike Bagging, the Random Subspace technique is a method of constructing a randomly selected subset of input variables from the feature set and then using each of them to learn different classifiers and incorporate their results. A simple schematic is shown in Figure 3 below.

Therefore, this study attempts to select techniques with relatively high predictive power by predicting management innovation using a large amount of data related to public research institutes and original and ensemble data mining techniques.

3. Research Framework

The management innovation prediction framework for public research institutes comprises three main phases. First, the management innovation was extracted to achieve the purpose of this study. Data mining was then performed by setting the extracted management innovation index as the target feature. Finally, the predictive power of data mining was evaluated using various evaluation techniques.

3.1. Evaluating Management Innovation

Management innovation is an important component of performance measurement as most models have components that measure the innovation process [57]. Among the performance-measuring models, the Malmquist index analysis model, which measures productivity, can measure management innovation [22,23,58]. The model can be divided into the catch-up index (CU) and frontier shift (FS) [22]. Catch-up (CU) can be described as a change in internal efficiency. Frontier shift (FS) is the means by which the maximum efficiency line changes in accordance with the external environment or technology. Catch up is an indicator of how close a firm is to an efficient frontier line [23] and can be interpreted as the extent to which a firm spreads new technologies to increase R&D efficiency. In summary, the frontier line refers to maximum technology, processes and various innovations, and the CU shows how close it is to maximum management innovation. Therefore, through the change in the CU, the level of management innovation is compared with that in previous periods. Using this method, management innovation has been continuously evaluated at the national [58,59] and corporate levels [22,23,60]. Accordingly, a CU can be converted into management innovation.

To quantitatively evaluate management innovation in public research institutions, this study uses the MPI. ${DMU}_k$ is evaluated to be efficient when its optimum $g_k$ = 1; DMUs where $g_k$ > 1 are inefficient. The Decision-Making Unit (DMU) is a case (public research institution). Färe [61] defined an output-based MPI (MI) as

(1)$$\begin{gathered} {\mathrm{M}\mathrm{P}\mathrm{I}}_k\left(x_{t+1},y_{t+1},x_t,y_t\right)={\left(\frac{g_k^t\left(x_{t+1},y_{t+1}\right)}{g_k^t\left(x_t,y_t\right)}\times \frac{g_k^{t+1}\left(x_{t+1},y_{t+1}\right)}{g_k^{t+1}\left(x_t,y_t\right)}\right)}^{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.} \\ =\frac{g_k^t\left(x_{t+1},y_{t+1}\right)}{g_k^t\left(x_t,y_t\right)}{\left(\frac{g_k^t\left(x_t,y_t\right)}{g_k^{t+1}\left(x_t,y_t\right)}\times \frac{g_k^t\left(x_{t+1},y_{t+1}\right)}{g_k^{t+1}\left(x_{t+1},y_{t+1}\right)}\right)}^{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.} \\ =\frac{QF/QB}{PC/PA}{\left(\frac{PC/PA}{PE/PA}\times \frac{QD/QB}{QF/QB}\right)}^{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.} \\ =\frac{QF/QB}{PC/PA}{\left(\frac{PC}{PE}\times \frac{QD}{QF}\right)}^{\raisebox{1ex}{$1$}\!\left/ \!\raisebox{-1ex}{$2$}\right.} \\ ={CU}_k(t,t+1)\times {FS}_k(t,t+1) \end{gathered}$$

where distance function $g_k^t\left(x_t,y_t\right)$ is defined as the efficiency score (during period t in relation to the technology in period t) of ${DMU}_k$, which has inputs $x_t$ and outputs $y_t$.

The method for evaluating management innovation using the above method is as follows. If a frontier exists, as shown in Figure 4, the formula for evaluating management innovation is as follows [62]:

(2)${Management innovation}_k(t,t+1) = \frac{QF/QB}{PC/PA}$

3.2. Forecasting Management Innovation

It is necessary to collect various features for predictive analysis to increase the predictive power of the data mining techniques [63]. All public information in one (ALIO) was used to collect several of these features. This is a system that discloses management information about public institutions in Korea. Several studies have used it as a data source to analyze public institutions in Korea [64,65]. Data that can be obtained through ALIO include financial-, labor- and remuneration-related data. Specifically, these include performance, labor financial status tables, income statements, income and expenditure statuses, new employment and flexible work, average annual salary of employees, welfare expenses and business expenses.

The data mining model used in this study is presented in Table 1 below.

3.3. Evaluating Data Mining Techniques

Five methods were used to evaluate the data mining technique. These include Accuracy, Precision, Recall, F-measure and Area Under the Curve (AUC), which are mainly used to evaluate data mining techniques. If the prediction is true and the actual analysis result is derived as true, it is a true positive (TP); if the actual result is derived as false, it is a false positive (FP). A false negative (FN) occurs if the prediction is false but the actual result is true, and a true negative (TN) occurs if the actual result is false. This is illustrated in the Figure 5 below.

Based on the above figure, Accuracy, Precision, Recall and F-measure were calculated using the following formulae:

(3)$\mathrm{Accuracy}=\frac{TP+TN}{TP+FP+TN+FN}$

(4)$\mathrm{Precision}=\frac{TP}{TP+FP}$

(5)$\mathrm{Recall}=\frac{TP}{TP+FN}$

(6)$\mathrm{F-measure}=2\times \frac{(\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}\times \mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l})}{(\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}+\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l})}$

In the case of Precision and Recall, the amount of data was significantly affected. Accuracy expresses the closeness of the degree of prediction to the actual results. Therefore, the evaluation of the technique was mainly conducted based on accuracy. The F-measure is an indicator of accuracy that integrates precision and recall. In particular, the F-measure evaluates the importance of precision and recall equally to support the judgement of accuracy prediction [66]. The Area Under the Receiver Operating Characteristic (ROC) curve (AUC) considers the balance between the sensitivity and specificity of the model. In the graph, the x axis represents the FP ratio, and the y axis represents the TP ratio. The more biased the graph on the axis, the better the classification of the data mining technique in the ROC curve. In other words, the closer the value of the area under the ROC curve is to 1, the better the performance, and the closer the area is to 0.5, the worse the classification.

4. Result

4.1. Evaluating Management Innovation

The target of this study was a public research institute under the National Science and Technology Research Council (NST), and data on input and output variables were collected for analysis. The NST manages government-funded research institutes related to science and technology in Korea. There are 25 affiliated government-funded research institutes, and 34% of Korea’s total R&D expenditure is invested in government-funded research institutes under the NST. Although there are 25 public research institutes under the NST, data were collected from 23 institutions, as 2 institutes did not provide data. The data were collected from 2014 to 2021. This is because, in 2014, public research institutes in the fields of natural science and engineering were integrated under the NST.

The DEAP Ver 2.1 program [67] was used to analyze productivity. Labor costs and government contributions were used as input variables [28,40] to analyze productivity, and the number of domestic papers, foreign papers, patent registrations and amount of revenue were set as output variables [28,68]. Management innovation was evaluated based on the analyzed MPI. The overall results of the eight-year management innovation values for the 23 public research institutes are presented in Table 2 below.

On average, management innovation decreased in the 2017–2018 period and in the 2019–2020 period. At that time, a major change in internal processes occurred because of changes in policies related to research institutes. Management innovation was judged as high or low based on 1. A value lower than 1 indicates lower management innovation compared with the previous year, and a value higher than 1 indicates higher internal management innovation compared with the previous year. In the case of Research Institute No. 2, management innovation increased in all periods except for 2017–2018. In the case of Research Institute No. 23, management innovation decreased in 2014–2015 and 2015–2016, whereas it increased in the later periods.

The years 2017–2018 brought many changes to government-funded research institutes as the ruling party in Korea changed. Nonregular workers were converted to regular workers, and most changed their internal work processes when a relevant enforcement decree was implemented. Additionally, many changes have occurred in personnel, performance management and internal regulations. In 2019–2020, working from home was initiated because of the global pandemic outbreak. Work processes and regulations have changed owing to unprecedented telecommuting.

4.2. Forecasting Management Innovation

Several variables were collected for predictive analysis. Many variables are required to increase the predictive power of the data mining techniques. ALIO was used to collect data on the variables. ALIO allows the public to comprehensively and easily grasp key information related to the management of public institutions through the Internet. A total of 282 features of the collected data were gathered, including data related to financial position tables, profit and loss statements, income and expenditure status, new employment and flexible work, the average annual salary of employees, welfare expenses and business promotion expenses.

The management innovation in public institutions was predicted using data mining. For the convenience of prediction, the target feature was converted into a binary variable. In management innovation, 1 ≤ high management innovation and 1 > low management innovation.

Based on the generated target features and 282 collected features, the predictive power was analyzed using 41 data mining models. Forty-one data mining models were verified using a 10-fold cross-validation method. The most commonly used data mining technique is a training test set. The training test set divides the entire dataset in a ratio of 5:5 to 9:1 according to the flow of time and then trains the pattern using past data. This was accomplished by checking the predictive power of the test set using a trained pattern. Because the method using the training test set predicts relatively recent data based on relatively recent data (training set), the possibility of errors increases when rapid fluctuations occur. However, the 10-fold cross-validation method cross-validates the entire data into 10 folds. This method can improve the accuracy of small datasets. Of the 10 folds, 9 were designated as the training set and 1 as the test set. Based on the above method, a prediction model was created, prediction was performed and error values were extracted. Subsequently, the test set was included in the training set, and one of the folds in the training set was used as the test set. This is a method for extracting the optimal model based on the error recorded by repeating the process for a total of 10 times. This reduces the error compared with the existing method and can reduce the effect over time.

This study collected 282 variables, 23 researchers, and seven years, but it was insufficient compared to other data mining sets. In the face of new environments due to infectious diseases after 2020, the 10-fold cross-validation method was considered more suitable than the training test method for analysis based on past data. Weka 3.8.6 [69] was used for the analysis. The data mining techniques used in this study are presented in the table below.

4.3. Evaluating Data Mining Technique

This study predicted management innovation in public research institutes. Table 3 presents an evaluation of the data mining model. High-management-innovating public research institutes are also predicted, and an evaluation of the data mining technique to determine this is shown. For Accuracy, the Bagging with DT (REP Tree) model yielded the highest value of 0.758, and for Precision, the Boosting with DT (Random Tree) model yielded the highest value of 0.778. In Recall, 1.000 was derived for the Hoeffding Tree (DT). The Bagging with DT (REP Tree) model was best derived at 0.939, and in AUC, the Bagging with the Bayes Net model was best derived at 0.693. A prediction of low-management-innovating public research institutes was conducted, and an evaluation of the data mining model to determine this is shown in the table below. As a result, somewhat lower Precision, Recall and F-measure results were obtained than those for the prediction of highly innovative institutions. The highest Precision and Recall values were 0.500 and 0.594, respectively. For the F-measure, the Boosting with DT (Hoeffding Tree) technique was the highest (0.493). Overall, the predictive power of low-management-innovating public research institutes was relatively insufficient compared to that of high-management-innovating public research institutes.

However, the predictive power of public research institutes with low management innovation levels is not powerful. Most of the indicators showed values of 0.5, indicating a lower predictive power than prediction through coin tosses. However, research institutes with high management innovation showed high values of 0.75 or higher in most indicators. Therefore, it is advantageous to predict high-management-innovating research institutes.

5. Discussion

In this study, management innovation in public research institutes was evaluated and its predictive power was presented by conducting research to predict innovativeness. As a result of the evaluation, institutions with increasing or decreasing management innovation showed consistent patterns. The pattern for laboratories with increased management innovation is to reduce fixed pay and increase performance-related pay (Research Institutes 2 and 4). Research Institutes 2 and 4 saw a 10% drop in fixed wages but a 20% increase in performance-related remuneration. There is no significant change in total wages; however, a high level of management innovation appears to occur after internally changing the wage rate. These results suggest the same findings as previous studies, showing that performance increases as performance-related pay increases [70,71]. This is consistent with studies in which performance decreased as performance-related pay decreased. Additionally, there was an increase in management innovation in research institutions with increased event support costs (Research Institutes 4 and 11). The cost of supporting events at Research Institutions 4 and 11 increased by approximately 20% each year during the period of increased management innovation (from an average of KRW 56 million to KRW 74 million). Event support expenses are correlated with the number of workshops. It can be assumed that workshops gather members of the research institute to share know-how and processes and that management innovation occurs through this sharing process [72,73]. Finally, management innovation increased when a flexible work system was established (Research Institute 9). In the case of Research Institute 9, management innovation was low but increased after the flexible work system was introduced. This is the same as studies showing that flexible work increases company management innovation [74,75,76]. The effect of the flexible working system is especially evident when it is applied to research positions, which can be seen as a characteristic of research. In addition, there is a difference in the working hours that individuals prefer, and it is considered that higher efficiency and productivity are shown by working at the preferred time.

As a result of predicting management innovation in public research institutes, many existing studies have explained the superiority of ensemble data mining techniques, which have been widely used to amplify predictive power. However, based on the results of this study, the data mining technique, in which the Accuracy was derived below 0.5, showed no significant change in the results. In addition, the evaluation items below 0.5 did not exceed 0.5. These results prove that ensemble data mining does not amplify the results unconditionally.

6. Conclusions

This study evaluated and predicted the management innovation in public research institutes from 2014 to 2021 and verified data mining techniques with high predictive power. The DEAP (Ver 2.1) program was used to evaluate management innovation. Weka (Ver 3.8.6) was used to predict the management innovation derived from the above process. More than 280 features were collected using ALIO to predict management innovation. The data mining technique compared and evaluated the predictive power using 41 techniques with the original and ensemble techniques.

Through the evaluation, the management innovation of each public research institution was extracted, and the analysis found that management innovation decreased on average in 2017–2018 and 2019–2020. It is thought that the decreases in 2017–2018 were due to changes in internal processes with the new Korean regime, and in 2019–2020, the management innovation seemed to change by introducing a new process due to the COVID-19 pandemic. In addition, research institutes have identified several management innovation characteristics. The management innovation of research institutes that introduced flexible work systems, increased performance-related pay and expanded event support was noticeable. Performance-related pay increased through changes to internal regulations in consultation with internal members, and the increase in event support expansion increased the number of workshops due to the increased need for internal workshops. It is evident that the introduction of the flexible work system is tailored to the internal environment as the external environment changes.

As a result of predicting management innovation in public research institutes using data mining techniques, the overall predictive power for high management innovation was high, but the predictive power for low management innovation was low. Because the predictive power was low, the ensemble technique was limited. The data mining technique with the highest Accuracy and F-measure for high management innovation was derived as Bagging with DT (REP Tree). The highest levels of precision were observed for DT (Random Tree), Recall (Hoeffding Tree) and AUC (Bagging with Bayes Net). In summary, the management innovation of public research institutes was measured, the results were predicted through data mining and the predicted results for high management innovation were excellent. These results make it possible to predict public research institutes with high management innovation so that they can be benchmarked against other public research institutes. In addition, less management can be undertaken for public research institutes where high management innovation is expected, which enables the efficient use of scarce resources.

The academic implications of this study are as follows. Many previous studies attempted to measure management innovation in research institutes. Therefore, various criteria were established to evaluate management innovation. However, after evaluating the overall performance of the research institute, it was mathematically decomposed to evaluate management innovation. This will affect future research on management innovation predictions. Second, the ensemble technique did not amplify the results unconditionally. In particular, this study proves that if the evaluation index for data mining does not exceed 0.5, it does not exceed 0.5, even if the ensemble technique is used. This presents a challenge for academically studying the ensemble technique. Third, the framework of this study can be applied not only to public research institutes but also to other fields. Only the criteria for performance differ, and there are academic implications in that a series of processes leading to prediction can be equally applied to all fields.

The practical implications of this study are as follows. The management innovation in public research institutes was evaluated and predicted, and the method used in the study will predict and evaluate the innovation in various public research institutes. In addition, in order to enhance management innovation, the evaluation of innovation must be preceded, and unlike the existing method of evaluating innovation, productivity is derived and then decomposed mathematically to evaluate management innovation, so there are relatively few errors. Therefore, practitioners of research institutes who want to increase management innovation can refer to this study to evaluate the management innovation of each research institute and then reflect policies on innovation to research institutes. Since the National Science and Technology Research Council, which manages public research institutes, can predict high management innovation, it can intensively manage research institutes other than those that have produced high management innovation. Through this, the resources required for the operation of public research institutes can be efficiently allocated. In addition, the method used in this study can be applied when evaluating or predicting management innovation in other fields.

The limitations of this study and directions for future research are as follows. First, the management innovation of government-funded research institutes was predicted using 282 variables. However, it is considered that, among these, there are fewer than 282 variables that have a great influence on the prediction of management innovation. Therefore, it is necessary to derive variables that exert the maximum predictive power with a minimum number of variables using various methods. If this can be achieved, it will be possible to overcome the temporal and physical limitations of prediction. Therefore, future studies should focus on feature selection to reduce the number of variables.

Second, with the recent development of data mining techniques, many more are being developed. In this study, management innovation was predicted using 41 data mining techniques, including existing and ensemble techniques, that amplify the results of existing techniques. The analysis showed that the predictive power was excellent for high management innovation but low for low management innovation. However, for a target variable with low predictive power, if a technique developed other than the 41 featured in this study is used, the results may be different. Therefore, in future studies, research that analyzes not only the data mining technique used in this study but also other techniques can be conducted.

There is a lack of management innovation evaluations and predictions by research institutes. The methodology presented in this study, which measures management innovation at both the research institute and industry levels, provides useful information on sustainable management innovation.

Author Contributions

Conceptualization, K.P. and J.H.; methodology, J.H.; software, K.P. and J.H.; validation, K.P., J.C. and J.H.; formal analysis, K.P.; investigation, J.H.; resources, J.H.; data curation, J.C.; writing—original draft preparation, K.P.; writing—review and editing, K.P.; visualization, K.P.; supervision, J.H.; project administration, J.C. All authors have read and agreed to the published version of the manuscript.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Not applicable.

Acknowledgments

This paper was written based on Park Kyungbo’s doctoral dissertation (2023).

Conflicts of Interest

The authors declare no conflict of interest.

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figures and Tables

Generate image description

Figure 1. Bagging model.

Generate image description

Figure 2. Boosting model.

Generate image description

Figure 3. Random Subspace.

Generate image description

Figure 4. The frontier shifting with period changing.

Generate image description

Figure 5. Evaluation metrics for the data mining technique.

Table 1

Data mining technique.

No.	Data Mining Technique	No.	Data Mining Technique
1	Bayes Net	22	Bagging with Naïve Bayes
2	Naïve Bayes	23	Bagging with Logistic
3	Logistic	24	Bagging with NN
4	ANN	25	Bagging with SVM
5	SVM	26	Bagging with DT (Hoeffding Tree)
6	DT (Hoeffding Tree)	27	Bagging with DT(J48)
7	DT (J48)	28	Bagging with DT (LMT)
8	DT (LMT)	29	Bagging with DT (Random Tree)
9	DT (Random Tree)	30	Bagging with DT (REP Tree)
10	DT (REP Tree)	31	Random Subspace with Bayes Net
11	Boosting with Bayes Net	32	Random Subspace with Naïve Bayes
12	Boosting with Naïve Bayes	33	Random Subspace with Logistic
13	Boosting with Logistic	34	Random Subspace with NN
14	Boosting with NN	35	Random Subspace with SVM
15	Boosting with SVM	36	Random Subspace with DT (Hoeffding Tree)
16	Boosting with DT (Hoeffding Tree)	37	Random Subspace with DT (J48)
17	Boosting with DT(J48)	38	Random Subspace with DT (LMT)
18	Boosting with DT (LMT)	39	Random Subspace with DT (Random Tree)
19	Boosting with DT (Random Tree)	40	Random Subspace with DT (REP Tree)
20	Boosting with DT (REP Tree)	41	Random Forest
21	Bagging with Bayes Net

Table 2

Annual management innovation of each public research institute.

Institutions	2014–2015	2015–2016	2016–2017	2017–2018	2018–2019	2019–2020	2020–2021
1	1.15	1.01	0.98	0.79	1.50	0.75	0.97
2	1.50	1.35	1.24	0.80	1.18	1.12	1.19
3	1.00	1.00	1.00	1.00	1.00	1.00	1.00
4	0.95	0.95	1.11	1.00	1.00	1.00	1.00
5	1.03	0.92	0.95	0.99	1.04	1.01	0.97
6	1.00	1.00	1.00	1.00	0.98	0.86	1.00
7	1.00	1.00	1.00	1.00	0.89	1.12	1.00
8	1.00	1.00	1.00	1.00	1.00	1.00	1.00
9	0.91	1.00	1.21	1.00	1.00	1.00	1.02
10	1.01	1.00	1.00	1.00	0.90	1.10	1.00
11	0.92	0.98	0.86	1.10	1.10	0.74	0.96
12	0.81	0.89	1.04	0.87	1.01	1.00	0.90
13	1.00	1.00	1.00	1.00	1.00	0.99	1.00
14	0.85	0.61	0.60	0.97	1.09	1.18	0.74
15	1.00	1.00	1.00	1.00	1.00	1.00	1.00
16	1.14	0.90	0.96	1.00	1.08	0.93	1.00
17	1.63	0.87	1.17	1.14	1.07	0.92	1.17
18	0.99	1.01	1.00	1.00	1.00	1.00	1.00
19	0.87	0.96	1.09	1.08	1.04	0.90	0.99
20	1.00	1.00	1.00	1.00	0.95	1.05	1.00
21	1.00	1.00	1.00	1.00	1.00	1.00	1.00
22	0.47	1.79	0.92	0.93	1.22	1.11	0.92
23	0.91	0.97	1.17	1.04	1.01	1.02	1.02
Average	1.01	1.01	1.01	0.99	1.04	0.99	1.00

Table 3

Evaluation of data mining technique to predict management innovation of public research institutes.

Data Mining		High-Management Innovation				Low-Management Innovation
No.	Accuracy	Precision	Recall	F-Measure	AUC	Precision	Recall	F-Measure	AUC
1	0.663	0.706	0.909	0.765	0.465	0.143	0.031	0.051	0.381
2	0.543	0.731	0.576	0.644	0.519	0.367	0.563	0.444	0.579
3	0.576	0.729	0.652	0.688	0.539	0.382	0.406	0.394	0.568
4	0.552	0.696	0.727	0.711	0.447	0.290	0.281	0.286	0.517
5	0.608	0.708	0.773	0.739	0.483	0.457	0.500	0.478	0.592
6	0.717	0.717	1.000	0.835	0.435	0.000	0.000	0.000	0.475
7	0.630	0.750	0.727	0.738	0.574	0.421	0.500	0.457	0.612
8	0.750	0.765	0.939	0.844	0.604	0.444	0.250	0.320	0.472
9	0.717	0.778	0.848	0.812	0.617	0.469	0.469	0.469	0.593
10	0.684	0.713	0.939	0.810	0.452	0.000	0.000	0.000	0.475
11	0.652	0.724	0.833	0.775	0.583	0.500	0.094	0.158	0.472
12	0.576	0.696	0.727	0.711	0.447	0.395	0.531	0.460	0.530
13	0.576	0.729	0.652	0.688	0.539	0.382	0.406	0.394	0.568
14	0.684	0.740	0.864	0.797	0.541	0.438	0.467	0.452	0.464
15	0.641	0.732	0.788	0.759	0.487	0.407	0.344	0.373	0.545
16	0.608	0.714	0.758	0.735	0.515	0.439	0.563	0.493	0.607
17	0.652	0.730	0.818	0.771	0.552	0.500	0.438	0.467	0.683
18	0.663	0.754	0.788	0.770	0.618	0.355	0.344	0.349	0.464
19	0.641	0.770	0.712	0.740	0.587	0.467	0.438	0.452	0.585
20	0.750	0.772	0.924	0.841	0.634	0.409	0.281	0.333	0.523
21	0.684	0.734	0.879	0.800	0.693	0.500	0.281	0.360	0.546
22	0.576	0.721	0.667	0.693	0.481	0.381	0.500	0.432	0.564
23	0.652	0.750	0.773	0.761	0.570	0.300	0.281	0.290	0.568
24	0.641	0.712	0.770	0.740	0.587	0.344	0.355	0.349	0.464
25	0.663	0.727	0.848	0.783	0.569	0.379	0.344	0.361	0.593
26	0.663	0.733	0.833	0.780	0.448	0.404	0.594	0.481	0.595
27	0.684	0.740	0.864	0.797	0.461	0.435	0.313	0.364	0.576
28	0.706	0.747	0.894	0.814	0.621	0.381	0.250	0.302	0.565
29	0.739	0.756	0.939	0.838	0.656	0.500	0.250	0.333	0.550
30	0.758	0.730	0.985	0.939	0.483	0.444	0.125	0.195	0.458
31	0.695	0.711	0.970	0.821	0.471	0.143	0.031	0.051	0.380
32	0.532	0.717	0.576	0.639	0.500	0.353	0.563	0.434	0.568
33	0.597	0.716	0.727	0.722	0.554	0.290	0.281	0.286	0.517
34	0.608	0.712	0.760	0.735	0.515	0.435	0.313	0.364	0.550
35	0.652	0.713	0.864	0.781	0.490	0.370	0.313	0.339	0.565
36	0.521	0.727	0.716	0.722	0.550	0.000	0.000	0.000	0.507
37	0.673	0.750	0.818	0.783	0.618	0.448	0.406	0.426	0.616
38	0.717	0.738	0.939	0.827	0.562	0.375	0.188	0.250	0.548
39	0.695	0.726	0.924	0.813	0.570	0.417	0.313	0.357	0.544
40	0.706	0.714	0.985	0.828	0.599	0.000	0.000	0.000	0.504
41	0.717	0.727	0.970	0.831	0.566	0.000	0.000	0.000	0.475

References

1. Volberda, H.W.; Van Den Bosch, F.A.; Heij, C.V. Management innovation: Management as fertile ground for innovation. Eur. Manag. Rev.; 2013; 10, pp. 1-15. [DOI: https://dx.doi.org/10.1111/emre.12007]

2. Rosenberg, N. Studies on Science and the Innovation Process: Selected Works by Nathan Rosenberg; World Scientific: Washington, DC, USA, 2009.

3. Elkhwesky, Z.; El Manzani, Y.; Elbayoumi Salem, I. Driving hospitality and tourism to foster sustainable innovation: A systematic review of COVID-19-related studies and practical implications in the digital era. Tour. Hosp. Res.; 2022; [DOI: https://dx.doi.org/10.1177/14673584221126792]

4. Calik, E.; Bardudeen, F. A measurement scale to evaluate sustainable innovation performance in manufacturing organizations. Procedia Cirp; 2016; 40, pp. 449-454. [DOI: https://dx.doi.org/10.1016/j.procir.2016.01.091]

5. Mol, M.J.; Birkinshaw, J. The role of external involvement in the creation of management innovations. Organ. Stud.; 2014; 35, pp. 1287-1312. [DOI: https://dx.doi.org/10.1177/0170840614539313]

6. Crossan, M.M.; Apaydin, M. A multi-dimensional framework of organizational innovation: A systematic review of the literature. J. Manag. Stud.; 2010; 47, pp. 1154-1191. [DOI: https://dx.doi.org/10.1111/j.1467-6486.2009.00880.x]

7. Khosravi, P.; Newton, C.; Rezvani, A. Management innovation: A systematic review and meta-analysis of past decades of research. Eur. Manag. J.; 2019; 37, pp. 694-707. [DOI: https://dx.doi.org/10.1016/j.emj.2019.03.003]

8. Teece, D.J. Explicating dynamic capabilities: The nature and microfoundations of (sustainable) enterprise performance. Strateg. Manag. J.; 2007; 28, pp. 1319-1350. [DOI: https://dx.doi.org/10.1002/smj.640]

9. Birkinshaw, J.; Hamel, G.; Mol, M.J. Management innovation. Acad. Manag. Rev.; 2008; 33, pp. 825-845. [DOI: https://dx.doi.org/10.5465/amr.2008.34421969]

10. Damanpour, F.; Schneider, M. Characteristics of innovation and innovation adoption in public organizations: Assessing the role of managers. J. Public Adm. Res. Theory; 2009; 19, pp. 495-522. [DOI: https://dx.doi.org/10.1093/jopart/mun021]

11. Mol, M.J.; Birkinshaw, J. The sources of management innovation: When firms introduce new management practices. J. Bus. Res.; 2009; 62, pp. 1269-1280. [DOI: https://dx.doi.org/10.1016/j.jbusres.2009.01.001]

12. Jansen, J.J.; Van den Bosch, F.A.; Volberda, H.W. Exploratory innovation, exploitative innovation, and ambidexterity: The impact of environmental and organizational antecedents. Schmalenbach Bus. Rev.; 2005; 57, pp. 351-363. [DOI: https://dx.doi.org/10.1007/BF03396721]

13. Vaccaro, I.G.; Jansen, J.J.; Van Den Bosch, F.A.; Volberda, H.W. Management innovation and leadership: The moderating role of organizational size. J. Manag. Stud.; 2012; 49, pp. 28-51. [DOI: https://dx.doi.org/10.1111/j.1467-6486.2010.00976.x]

14. Vaccaro, A.; Parente, R.; Veloso, F.M. Knowledge management tools, inter-organizational relationships, innovation and firm performance. Technol. Forecast. Soc. Change; 2010; 77, pp. 1076-1089. [DOI: https://dx.doi.org/10.1016/j.techfore.2010.02.006]

15. Walker, R.M.; Damanpour, F.; Devece, C.A. Management innovation and organizational performance: The mediating effect of performance management. J. Public Adm. Res. Theory; 2011; 21, pp. 367-386. [DOI: https://dx.doi.org/10.1093/jopart/muq043]

16. Damanpour, F.; Aravind, D. Managerial innovation: Conceptions, processes and antecedents. Manag. Organ. Rev.; 2012; 8, pp. 423-454. [DOI: https://dx.doi.org/10.1111/j.1740-8784.2011.00233.x]

17. Ansari, S.S.; Krop, P. Incumbent performance in the face of a radical innovation: Towards a framework for incumbent challenger dynamics. Res. Policy; 2012; 41, pp. 1357-1374. [DOI: https://dx.doi.org/10.1016/j.respol.2012.03.024]

18. Boons, F.; Lüdeke-Freund, F. Business models for sustainable innovation: State-of-the-art and steps towards a research agenda. J. Clean. Prod.; 2013; 45, pp. 9-19. [DOI: https://dx.doi.org/10.1016/j.jclepro.2012.07.007]

19. Edeh, F.O.; Zayed, N.M.; Nitsenko, V.; Brezhnieva-Yermolenko, O.; Negovska, J.; Shtan, M. Predicting innovation capability through knowledge management in the banking sector. J. Risk Financ. Manag.; 2022; 15, 312. [DOI: https://dx.doi.org/10.3390/jrfm15070312]

20. Drake, A.R.; Haka, S.F.; Ravenscroft, S.P. Cost system and incentive structure effects on innovation, efficiency and profitability in teams. Account. Rev.; 1999; 74, pp. 323-345. [DOI: https://dx.doi.org/10.2308/accr.1999.74.3.323]

21. Nieves, J.; Segarra-Ciprés, M. Management innovation in the hotel industry. Tour. Manag.; 2015; 46, pp. 51-58. [DOI: https://dx.doi.org/10.1016/j.tourman.2014.06.002]

22. Hashimoto, A.; Haneda, S. Measuring the change in R&D efficiency of the Japanese pharmaceutical industry. Res. Policy; 2008; 37, pp. 1829-1836.

23. Jang, H.; Lee, S.; Suh, E. A comparative analysis of the change in R&D efficiency: A case of R&D leaders in the technology industry. Technol. Anal. Strateg. Manag.; 2016; 28, pp. 886-900.

24. Romero, C.; Ventura, S. Data mining in education. Wiley Interdiscip. Rev. Data Min. Knowl. Discov.; 2013; 3, pp. 12-27. [DOI: https://dx.doi.org/10.1002/widm.1075]

25. Kumar, V.; Minz, S. Feature selection: A literature review. SmartCR; 2014; 4, pp. 211-229. [DOI: https://dx.doi.org/10.6029/smartcr.2014.03.007]

26. Porter, M.E. The structure within industries and companies’ performance. Rev. Econ. Stat.; 1979; 61, pp. 214-227. [DOI: https://dx.doi.org/10.2307/1924589]

27. Senker, J. Changing organisation of public-sector research in Europe—Implications for benchmarking human resources in RTD. Sci. Public Policy; 2001; 28, pp. 277-284. [DOI: https://dx.doi.org/10.3152/147154301781781390]

28. Coccia, M. Analysis and classification of public research institutes. World Rev. Sci. Technol. Sustain. Dev.; 2006; 3, pp. 1-16. [DOI: https://dx.doi.org/10.1504/WRSTSD.2006.008759]

29. Tushman, M.; Nadler, D. Organizing for innovation. Calif. Manag. Rev.; 1986; 28, pp. 74-92. [DOI: https://dx.doi.org/10.2307/41165203]

30. Czarnitzki, D.; Kraft, K. Firm leadership and innovative performance: Evidence from seven EU countries. Small Bus. Econ.; 2004; 22, pp. 325-332. [DOI: https://dx.doi.org/10.1023/B:SBEJ.0000022209.72378.fe]

31. Wang, W.; Liu, Y. Does University-industry innovation community affect firms’ inventions? The mediating role of technology transfer. J. Transfer.; 2022; 47, pp. 906-935. [DOI: https://dx.doi.org/10.1007/s10961-021-09887-w]

32. Birkinshaw, J.M.; Mol, M.J. How management innovation happens. MIT Sloan Manag. Rev.; 2006; 47, pp. 81-88.

33. Hamel, G. The why, what, and how of management innovation. Harv. Bus. Rev.; 2006; 84, 72.

34. Damanpour, F. The adoption of technological, administrative, and ancillary innovations: Impact of organizational factors. J. Manag.; 1987; 13, pp. 675-688. [DOI: https://dx.doi.org/10.1177/014920638701300408]

35. Evan, W. Organizational lag. Hum. Organ.; 1966; 25, pp. 51-53. [DOI: https://dx.doi.org/10.17730/humo.25.1.v7354t3822136580]

36. Kimberly, J.R.; Evanisko, M.J. Organizational innovation: The influence of individual, organizational, and contextual factors on hospital adoption of technological and administrative innovations. Acad. Manag. J.; 1981; 24, pp. 689-713. [DOI: https://dx.doi.org/10.2307/256170]

37. Grossman, G.M.; Helpman, E. Innovation and Growth in the Global Economy; MIT Press: Cambridge, MA, USA, 1993.

38. Nordberg, M.; Campbell, A.; Verbeke, A. Using customer relationships to acquire technological innovation: A value-chain analysis of supplier contracts with scientific research institutions. J. Bus. Res.; 2003; 56, pp. 711-719. [DOI: https://dx.doi.org/10.1016/S0148-2963(01)00256-9]

39. Robin, S.; Schubert, T. Cooperation with public research institutions and success in innovation: Evidence from France and Germany. Res. Policy; 2013; 42, pp. 149-166. [DOI: https://dx.doi.org/10.1016/j.respol.2012.06.002]

40. Coccia, M. Research performance and bureaucracy within public research labs. Scientometrics; 2009; 79, pp. 93-107. [DOI: https://dx.doi.org/10.1007/s11192-009-0406-2]

41. Etzkowitz, H.; Leydesdorff, L. The dynamics of innovation: From National Systems and “Mode 2” to a Triple Helix of university–industry–government relations. Res. Policy; 2000; 29, pp. 109-123. [DOI: https://dx.doi.org/10.1016/S0048-7333(99)00055-4]

42. Leydesdorff, L. The triple helix: An evolutionary model of innovations. Res. Policy; 2000; 29, pp. 243-255. [DOI: https://dx.doi.org/10.1016/S0048-7333(99)00063-3]

43. Hsu, C.-W. Formation of industrial innovation mechanisms through the research institute. Technovation; 2005; 25, pp. 1317-1329. [DOI: https://dx.doi.org/10.1016/j.technovation.2004.06.002]

44. Chen, K.; Kenney, M. Universities/research institutes and regional innovation systems: The cases of Beijing and Shenzhen. World Dev.; 2007; 35, pp. 1056-1074. [DOI: https://dx.doi.org/10.1016/j.worlddev.2006.05.013]

45. Atluri, G.; Karpatne, A.; Kumar, V. Spatio-temporal data mining: A survey of problems and methods. ACM Comput. Surv. (CSUR); 2018; 51, pp. 1-41. [DOI: https://dx.doi.org/10.1145/3161602]

46. Zhang, H.; Rao, H.; Feng, J. Product innovation based on online review data mining: A case study of Huawei phones. Electron. Commer. Res.; 2018; 18, pp. 3-22. [DOI: https://dx.doi.org/10.1007/s10660-017-9279-2]

47. Su, C.-T.; Chen, Y.-H.; Sha, D. Linking innovative product development with customer knowledge: A data-mining approach. Technovation; 2006; 26, pp. 784-795. [DOI: https://dx.doi.org/10.1016/j.technovation.2005.05.005]

48. Zheng, L.J.; Xiong, C.; Chen, X.; Li, C.-S. Product innovation in entrepreneurial firms: How business model design influences disruptive and adoptive innovation. Technol. Forecast. Soc. Change; 2021; 170, 120894. [DOI: https://dx.doi.org/10.1016/j.techfore.2021.120894]

49. Escamilla-Fajardo, P.; Núñez-Pomar, J.; Parra-Camacho, D. Does the organizational climate predict the innovation in sports clubs?. J. Entrep. Public Policy; 2019; 8, pp. 103-121. [DOI: https://dx.doi.org/10.1108/JEPP-03-2019-104]

50. Choo, A.; Narayanan, S.; Srinivasan, R.; Sarkar, S. Introducing goods innovation, service innovation, or both? Investigating the tension in managing innovation revenue streams for manufacturing and service firms. J. Oper. Manag.; 2021; 67, pp. 704-728. [DOI: https://dx.doi.org/10.1002/joom.1136]

51. Lee, J.; Kao, H.-A.; Yang, S. Service innovation and smart analytics for industry 4.0 and big data environment. Procedia Cirp; 2014; 16, pp. 3-8. [DOI: https://dx.doi.org/10.1016/j.procir.2014.02.001]

52. Figueiredo, R.; Magalhães, C.; Huber, C. How to Predict the Innovation to SMEs? Applying the Data Mining Process to the Spinner Innovation Model. Soc. Sci.; 2023; 12, 75. [DOI: https://dx.doi.org/10.3390/socsci12020075]

53. Barak, S.; Modarres, M. Developing an approach to evaluate stocks by forecasting effective features with data mining methods. Expert Syst. Appl.; 2015; 42, pp. 1325-1339. [DOI: https://dx.doi.org/10.1016/j.eswa.2014.09.026]

54. Dietterich, T.G. Machine-learning research. AI Mag.; 1997; 18, 97.

55. Krawczyk, B.; Minku, L.L.; Gama, J.; Stefanowski, J.; Woźniak, M. Ensemble learning for data stream analysis: A survey. Inf. Fusion; 2017; 37, pp. 132-156. [DOI: https://dx.doi.org/10.1016/j.inffus.2017.02.004]

56. Ho, T.K. The random subspace method for constructing decision forests. IEEE Trans. Pattern Anal. Mach. Intell.; 1998; 20, pp. 832-844.

57. Ivanov, C.-I.; Avasilcăi, S. Performance measurement models: An analysis for measuring innovation processes performance. Procedia-Soc. Behav. Sci.; 2014; 124, pp. 397-404. [DOI: https://dx.doi.org/10.1016/j.sbspro.2014.02.501]

58. Taskin, F.; Zaim, O. Catching-up and innovation in high-and low-income countries. Econ. Lett.; 1997; 54, pp. 93-100. [DOI: https://dx.doi.org/10.1016/S0165-1765(97)00004-9]

59. Madden, G.; Savage, S.J. Telecommunications productivity, catch-up and innovation. Telecommun. Policy; 1999; 23, pp. 65-81. [DOI: https://dx.doi.org/10.1016/S0308-5961(98)00076-7]

60. Zhang, X.; Tone, K.; Lu, Y. Impact of the local public hospital reform on the efficiency of medium-sized hospitals in Japan: An improved slacks-based measure data envelopment analysis approach. Health Serv. Res.; 2018; 53, pp. 896-918. [DOI: https://dx.doi.org/10.1111/1475-6773.12676]

61. Färe, R.; Grosskopf, S.; Norris, M.; Zhang, Z. Productivity growth, technical progress, and efficiency change in industrialized countries. Am. Econ. Rev.; 1994; 84, pp. 66-83.

62. Tone, K. Malmquist productivity index: Efficiency change over time. Handbook on Data Envelopment Analysis; Springer: New York, NY, USA, 2004; pp. 203-227.

63. Jothi, N.; Husain, W. Data mining in healthcare—A review. Procedia Comput. Sci.; 2015; 72, pp. 306-313. [DOI: https://dx.doi.org/10.1016/j.procs.2015.12.145]

64. Shin, S.H.; Kim, E.Y.; Ha, S.; Nam, Y.H. Public Disclosure on State-Owned Enterprises: Integrated Disclosure System on SOEs in South Korea-All Public Information In-One (ALIO) System: A Case Study; The World Bank Group: Washington, DC, USA, 2022; pp. 1-19.

65. Oh, Y.; Lee, K. External control mechanisms and red tape: Testing the roles of external audit and evaluation on red tape in quasi-governmental organizations. Int. Rev. Adm. Sci.; 2022; 88, pp. 355-372. [DOI: https://dx.doi.org/10.1177/0020852320974097]

66. Kubat, M.; Matwin, S. Addressing the Curse of Imbalanced Training Sets: One-sided Selection. 1997; pp. 179-186. Available online: https://sci2s.ugr.es/keel/pdf/specific/congreso/kubat97addressing.pdf (accessed on 8 March 2023).

67. Coelli, T. A guide to DEAP version 2.1: A data envelopment analysis (computer) program. Cent. Effic. Product. Anal. Univ. N. Engl. Aust.; 1996; 96, pp. 1-49.

68. Lee, B.L.; Worthington, A.C. A network DEA quantity and quality-orientated production model: An application to Australian university research services. Omega; 2016; 60, pp. 26-33. [DOI: https://dx.doi.org/10.1016/j.omega.2015.05.014]

69. Hall, M.; Frank, E.; Holmes, G.; Pfahringer, B.; Reutemann, P.; Witten, I.H. The WEKA data mining software: An update. ACM SIGKDD Explor. Newsl.; 2009; 11, pp. 10-18. [DOI: https://dx.doi.org/10.1145/1656274.1656278]

70. Eldridge, C.; Palmer, N. Performance-based payment: Some reflections on the discourse, evidence and unanswered questions. Health Policy Plan.; 2009; 24, pp. 160-166. [DOI: https://dx.doi.org/10.1093/heapol/czp002]

71. Chamberlin, R.; Wragg, T.; Haynes, G.; Wragg, C. Performance-related pay and the teaching profession: A review of the literature. Res. Pap. Educ.; 2002; 17, pp. 31-49. [DOI: https://dx.doi.org/10.1080/02671520110102534]

72. Lilleoere, A.M.; Holme Hansen, E. Knowledge-sharing enablers and barriers in pharmaceutical research and development. J. Knowl. Manag.; 2011; 15, pp. 53-70. [DOI: https://dx.doi.org/10.1108/13673271111108693]

73. Huysman, M.; De Wit, D. Practices of managing knowledge sharing: Towards a second wave of knowledge management. Knowl. Process Manag.; 2004; 11, pp. 81-92. [DOI: https://dx.doi.org/10.1002/kpm.192]

74. Azeem, M.M.; Kotey, B. Innovation in SMEs: The role of flexible work arrangements and market competition. Int. J. Hum. Resour. Manag.; 2021; 34, pp. 1-36. [DOI: https://dx.doi.org/10.1080/09585192.2021.1961162]

75. Rainey Jr, G.W.; Wolf, L. The organizationally dysfunctional consequences of flexible work hours: A general overview. Public Pers. Manag.; 1982; 11, pp. 165-175. [DOI: https://dx.doi.org/10.1177/009102608201100209]

76. Coenen, M.; Kok, R.A. Workplace flexibility and new product development performance: The role of telework and flexible work schedules. Eur. Manag. J.; 2014; 32, pp. 564-576. [DOI: https://dx.doi.org/10.1016/j.emj.2013.12.003]