APPLICATION OF ARTIFICIAL NEURAL NETWORKS TO COST FACTORS STIMULATING INNOVATION – THE CASE OF SLOVAKIA

Abstract

Artificial neural networks are means of processing complex data, and they use a number of interconnected processors and computational paths for their work. The architecture of the human brain inspires artificial neural networks, they are able to "learn" and analyse complex sets of data that would be difficult for simpler algorithms to handle. This work will show what a neural network is, its mathematical model, and how the neural network method is implemented in solving the interrelationships between inputs and outputs in the companies of the Central Europe. One hundred fifty-five small and medium-sized Slovak enterprises are the subject of modelling, where the inputs are the costs of training employees and the costs of science and research in the post-Soviet country. The output is the company's net turnover and it is analysed using the SPSS program. The first part is a description of neural networks; it explains what a neural network is and its classification. The second part of the methodology describes the methods used in the processing of the scientific article. It also mentioned the SPSS program, which was used on artificial neural networks. The following section presents the results of the analysis and the importance of the inputs are available. In the discussions section, the possibilities of further use of the given issue and finally the summary are discussed.

Headnote

Abstract: Artificial neural networks are means of processing complex data, and they use a number of interconnected processors and computational paths for their work. The architecture of the human brain inspires artificial neural networks, they are able to "learn" and analyse complex sets of data that would be difficult for simpler algorithms to handle. This work will show what a neural network is, its mathematical model, and how the neural network method is implemented in solving the interrelationships between inputs and outputs in the companies of the Central Europe. One hundred fifty-five small and medium-sized Slovak enterprises are the subject of modelling, where the inputs are the costs of training employees and the costs of science and research in the post-Soviet country. The output is the company's net turnover and it is analysed using the SPSS program. The first part is a description of neural networks; it explains what a neural network is and its classification. The second part of the methodology describes the methods used in the processing of the scientific article. It also mentioned the SPSS program, which was used on artificial neural networks. The following section presents the results of the analysis and the importance of the inputs are available. In the discussions section, the possibilities of further use of the given issue and finally the summary are discussed.

Keywords: artificial neural networks, research & development, knowledge management

JEL Classification: O32, D80, C45

1.Introduction

From a historical point of view, the emergence of science is a specific attribute of civilization. It is the interaction between people, and later cooperation at the level of enlightenment and communities, that is the tool for disseminating scientific knowledge and technical solutions. (David and Foray, 2003)

The scientific and technical revolution took place in several phases. The first phase began in the 19th and 20th centuries when there was a rapid development of science and technology. We also refer to this phase as the process of creating the Industrial Society, because it resulted in a change in the social structure within European states (Castells, 2014). Mass production, the expansion of the railways, and the critical discovery and subsequent use of electricity were typical for this period. (Gersbach et al., 2019) With the gradual rapid development of automobile and air transport came changes in the way of communication. It has become more accessible and has significantly accelerated the way information flows. The industrial phase not only provided enormous tools for foreign policy cooperation but also contributed to increasing the quality of life and the average age of the population. (Kaul et al., 1999; Calabro et al., 2019)

The impact of science and technology on society has been at the forefront of research by many political scientists, sociologists, philosophers, and theologians since the beginning of the modern age (Ritzer and Stepnisky, 2017). Their views ranged from enthusiasm to damn science. Nevertheless, science has become a part of our daily lives and perhaps a source of stress. For example, in war conflicts, military technology rather symbolizes fear, horror and possible extinction (Russell, 2016). We must also not forget the new forms of addiction associated with technological conveniences, in the way of minors, who spend much of their free time behind the screens of their electronic devices. Besides, it has led to an increase in competition and investment in this factor has had a positive impact on companies' incomes. (Alter, 2017; Bloom et al., 2019)

In this work, we investigated the relationships between inputs and outputs. One of the inputs was the cost of science and research. Because of the importance of technological innovation for economic growth, countries seek to stimulate and attract research and development that leads to innovation. (Lømo, 2018)

However, designing cost-effective methods to support technological innovation has become significantly more difficult as the world economy has become more interconnected. (Graetz and Doud, 2013) A richer characterization of the innovation process has been developed by scientists such as Gibbons and Johnston (1975), Kline and Rosenberg (1986), Nelson (1990) and von Hippel (1988). This concept represents a more interactive relationship in which public research sometimes leads to the development of new technologies and sometimes focuses on problems caused by previous developments or buyer feedback. From this point of view, industrial innovation is based on a complicated process in which basic research does not have to play an initiating role and sometimes no role at all. (Cohen et al., 2002)

Science and technology are among the main factors in improving the competitiveness of the economy and long-term economic growth. For this reason, it makes sense to support a significant development factor. (Fang et al., 2017) It is necessary to procure sufficient inputs and then check the outputs so that the science and technology system fulfils its function correctly. In addition to human resources, funds spent on science and technology are one of the main inputs to the science and technology system (Sealey and Lindley, 1977). To make the use of science and technology as efficient as possible, all businesses must be aware of the importance of supporting science and technology and therefore seek to increase spending on science and technology. Resources constituting total expenditures on science and technology, business and public funds, have defined expected outputs and a defined purpose of their use. (Verbeek et al., 2002)

The main prerequisite for the development of science and technology is available human resources. They are an essential condition for the successful development of the knowledge society in full and, ultimately, they are a necessary prerequisite for increasing the competitiveness of the Slovak economy and faster modernization of the whole community. Separately in the interest of long-term development and use of science and technology, education is one of the pillars of the knowledge society because the requirement for the development of the knowledge society is specially educated human resources. (Malik, 2018)

The system of funding science in developed countries relies on a strong preference for priorities. State research and development programs are a tool with which the state can support research in a specific area (Busch and Lacy, 2019). The State Research and Development Program focuses on the field of science and technology, in which research and development should be strengthened to increase its economic and social benefits. The aim is to achieve a first-class level of research results and international recognition in the field. The state research and development program is implemented in the form of solving research and development projects. (Szambelan and Jiang, 2020) In Slovakia, state government research and development programs are approved by the government on the proposal of the Minister of Education after previous public discussion and discussion in the Science and Technology Council. The Intermediate Body under the Managing Authority for the Operational Program Research and Development is the Agency of the Ministry of Education, Science, Research and Sport of the Slovak Republic for the EU Structural Funds. (Kluvankova-Oravska, 2004)

The neuron receives signals through its dendrites and transmits a generated signal in the body along its axon. Each neuron can have many dendrites, but only one axon. There are synapses at the ends of these fibers. (Zhang, 2019) A synapse is a functional structure between the dendrite of one neuron and the axon of another. When the signal reaches the end of the synapses, chemicals called neurotransmitters released into the gap within the neurons (synaptic cleft). There are receptors at the beginning of the second neuron. When a neurotransmitter relinquishes the relevant receptor, it emits an electrical impulse. The efficiency of a synapse can vary according to the signals that pass along it. The more frequent this transition, the more effective the synapse becomes. We can call this an act of learning. (Lomo, 2018)

The neural network itself is not an algorithm, but rather a framework for many various machine learning algorithms that work together to process complex data inputs. It is a description of the model that is inspired by the functioning of human neurons and their processing of inputs from the human senses. (Liu et al., 2019)

Machine learning algorithms that use neural networks do not need to be programmed using specific rules that determine what to expect from the input. Instead, we can "teach" these rules to a neural network and then apply that knowledge to unknown data. (Hertz, 2018)

A neural network consists of layers of neurons. These interconnected by edges synapses that transmit the signal. Each edge has its rating (weight), and this indicates the strength of the connection in the network. Neurons arranged in groups called layers. Each neuron in the layer works in logical parallelism. Information is transferred from one layer to another in serial operations. (Arredondo-Velazquez et al., 2020)

Classification is a problem in machine learning, where a function learns to map xi objects to one of the predefined classes y and the training data set D has an associated classyi assigned to each sample xi. D = {(x1, y1); (X2, y2);...; (Xn, yn)}. An object xi is usually an m-dimensional vector whose components are called input variables. In most cases there is no functional relation y = f (x) and therefore the relation x, y must be described by the probability distribution P (x, y) and according to the theory of statistical distribution the corresponding class y is selected as maximization of the posterior probability distribution. (Zhu et al., 2018)

2.Methodology

A total of 155 Slovak small and medium-sized enterprises were analyzed. We obtained data that allowed us to apply an artificial neural network. The inputs that were used in the analysis were the costs of research and development and the costs of education. The output was the net turnover of the SME. We found out which input factor is more significant concerning the output, and therefore the neural network technique was chosen. The aim was to find out the mutual relations and connections between individual input factors with hidden layer and hidden layer and output. This is the main technique that was used in the research paper and processed using the SPSS program. (Paunov, 2016)

Another method used for processing a scientific paper is the method of analysis, which was mainly used in the study of neural networks. Subsequently, it is a method of synthesis, ie, combining knowledge. (Schaffer et al., 1992)

The technique for which we decided to apply in our work is an artificial neural network. ANN (artificial neural networks) systems are based on a collection of connected units called artificial neurons. Each synapse (the connection between neurons) can transfer a signal to another neuron. The receiving neuron can then process the signal and send the signal to another neurone in the chain of artificial neurons. (Agatonovic-Kustrin and Beresford, 2000) Basic concepts of artificial neural network (ANN) modeling and its application in pharmaceutical research. Journal of pharmaceutical and biomedical analysis, 22(5), 717-727.). The state of neurons is represented by real numbers in the interval <0; 1>. These neurons can have different parameters. (Izhikevich, 2003) An example is the weight of the link. This determines the strength with which the synapse signal will be sent to the next neuron.

The first generation of ANN consisted of McCulloch-Pitts threshold neurons, a conceptually straightforward model. In this model, a neuron emits a binary "high" signal if the sum of the incoming weighted signals is higher than a specified limit. (Maass, 1999) Although these neurons can only generate digital output, they have been successfully used in efficient ANNs such as multilayer perceptrons (perceptron is a computer model designed to represent and simulate the brain's ability to recognize or discriminate) and Hopfield networks. Each function that has a boolean output can be calculated using a multilayer perceptron with a single hidden layer. These networks are universal for digital computing. (Maass, 1999)

The multilayer perceptron is a supervised method using a feedforward architecture, and it can have multiple hidden layers. One or more dependent variables may be defined, which may be scale, categorization or a succession. If the dependent variable has a scale measurement level, then the neural network prognosticates connected values that approximate the "real" cost of some continuous input function. If the dependent variable is categorical, the neural network is used to analyze cases into the "best" category based on input predictors. (McCormick et al., 2013)

Second-generation neurons do not use a boundary to calculate their output signal, but a continuous activation function that makes these neurons suitable for analogue input and output. A typical example of an activating role is a sigmoid or hyperbolic tangent. As an example of neural networks consisting of such neurons, we mention recurrent neural networks. Networks using second-generation neurons are more efficient than networks using their first-generation transit. If if we give them a boundary function on the output layer of the neural network, they are universal for digital calculate. (Scriptor, 2004) They can do so with fewer neurons compared to first-generation neurons. The signals generated by the first- and second-generation neurons can be considered as the normalized firing frequencies of the neuron at a given time. Biological neurons have a fundamental firing frequency (the mean pulse rate), and a continuous activation function can model these mean frequency values. As a result, second-generation neurons are more similar to biological neurons, making them more efficient than first-generation neurons (Maass, 1997).

3.Results

This work aims to determine the relationship between the analyzed inputs and outputs. The output is the net turnover of small and medium-sized enterprises in the Slovak Republic in the number of 155. The inputs are two, one of which represents the costs of employee training, conferences, language courses, seminars, training, etc. Another input is the cost of research and development, including the cost of researchers' salaries, the cost of purchasing software, subscribing to scientific journals and more.

Artificial neural networks were used to model inputs concerning output. This technique is attributed to the context of regression, which requires a sophisticated approach, such as a probit or logit. In our case, the dependent variable net turnover of enterprises, and we want to explain this by the cost of employee training and the cost of research and development.

The analysis was performed using SPSS software, which is also equipped with the ability to implement artificial neural networks analysis. The procedure is that it is first necessary to enter the data that will be subject to interpretation. Then it is essential to select the path in the program: analysis - neural networks - multilayer perceptron. Within this section, in the Variables part, you need to specify individual variables. In this case, net turnover will be variable, and the cost of R&D and training represent covariates. Then in the Network Structure part, it is necessary to select the Description; diagram; Synaptic Weights. In the Network Performance section, it is crucial to choose Model summary and Classification results. Based on this choice of parameters, it is possible to identify which of the variables play a more critical and which a less significant role. The result can be compared to the Beta coefficient of the regression analysis.

In this case, in the training phase, 111 units received, which represents 71.6% and the remaining 28.4% were subsequently tested, precisely 44 units (see Table 1).

Another critical part of the output from SPSS is the generated model (see Figure 1), in which two units in 1 hidden layer were selected. Depending on how highlighted the lines are, they highlight the relationships. In the input layer and hidden layer, there is an error term of the bias, which in this case is a more robust frame of the defect. Therefore, the highlighted blue line running from the bias in the input layer to the hidden layer will be difficult to interpret.

The following Table 3 shows the quality of the model. Low values of error percentage in the training (0.306%) and testing (0.501%) phase increase the quality of the generated model.

The next Table 3 addresses in more detail about the importance of relationships in numerical terms and is linked to the results of Figure 1, which provides a graphical representation. The upper left part expresses the connections from the input factors to the hidden layer and the second part of the table points from the hidden layer to the output.

Afterwards, a sensitivity analysis is performed, which calculates the importance of each predictor in determining the neural network. The study is based on combined samples for training and testing. Thus, a table was created, showing the significance and normalized significance for each predictor.

Subsequently, Table 3 shows the importance score of the analyzed inputs. Based on the results, we can conclude that the input costs of research and development have a more significant impact on the net turnover of small and medium-sized enterprises than the costs of employee training. However, this difference is only slight.

4.Discussions

The study presents a significant contribution, especially in terms of the use of technically artificial neural networks on the practical example of 155 companies in the Slovak Republic. The results indicate the results of the interrelationships between inputs and outputs. Inputs, in a way, represent the cost of innovation, and the output is the net turnover of companies. In the future, these inputs could be extended by others or specified in more detail already established. The advantage of neural networks is that they can work with both quantitative data and qualitative data. Currently, neural networks are used in almost every field, and this technique has great potential. Therefore, in this area, we recommend a more extensive analysis, uniquely when identifying other components that could represent inputs to the costs of innovation or specify and analyze the existing ones. At the same time, it would be necessary to address a larger sample of companies and to disaggregate the data in terms of the sector to which they belong. At the same time, in this case, it is likely that in other industries, companies will encounter different types of innovation costs, the expenditure of which represents a turnover for the company. Therefore, it would be necessary to establish cooperation with companies and jointly identify individual incentives. This study is an entry point to this issue, and there is still much room for improvement

5.Conclusion

This research aimed to determine the effectiveness of artificial neural networks in solving the relationships between inputs and outputs. A total of 155 companies within the Slovak Republic were analyzed who were willing to cooperate in the creation of the database and provided the necessary data needed for processing. A review of the literature suggested that neural networks outperform all other classifiers in terms of accuracy. The results also showed that the most reliable innovative input is the cost of research and development, followed closely by the value of staff training.

Acknowledgment

This paper is an output of the science project VEGA 1/0708/18 named: "Aspects of use of the SoLoMo marketing concept to enhance awareness of eco-innovations".

Sidebar

Cite as: Zauskova, A., Lyakina, M., Tretyak, V., Miklencicova, R. (2020). Application of artificial neural networks to cost factors stimulating innovation - The case of Slovakia. Ekonomicko-manazerske spektrum, 14(1), 97-105.

Avaliable at: dx.doi.org/10.26552/ems.2020.1.97-105

References

References

Agatonovic-Kustrin, S. and Beresford, R. (2000). Basic concepts of artificial neural network (ANN) modeling and its application in pharmaceutical research. Journal of Pharmaceutical and Biomedical Analysis, 22(5), pp. 717727.

Alter, A. (2017). Irresistible: The rise of addictive technology and the business of keeping us hooked. Penguin.

Arredondo-Velazquez, M., Diaz-Carmona, J., Torres-Huitzil, C., Padilla-Medina, A. and Prado-Olivarez, J. (2020). A streaming architecture for convolutional neural networks based on layer operations chaining. Journal of Real-Time Image Processing, pp. 1-19.

Bloom, N., Van Reenen, J. and Williams, H. (2019). A toolkit of policies to promote innovation. Journal of economic perspectives, 55(3), pp. 163-184.

Busch, L. M. and Lacy, W. B. (2019). Science, agriculture, and the politics of research. Routledge.

Calabro, A., Vecchiarini, M., Gast, J. et al. (2019). Innovation in family firms: A systematic literature review and guidance for future research. International Journal of Management Review, 21(3), pp. 317-335.

Castells, M. (2014). Technopoles of the world: The making of 21st century industrial complexes. Routledge.

Cohen, W. M., Nelson, R. R. and Walsh, J. P. (2002). Links and impacts: The influence of public research on industrial R&D. Management Science, 48(1), pp. 1-23.

David, P. A. and Foray, D. (2003). Economic fundamentals of the knowledge society. Policy Futures in Education, 1(1), pp. 20-49.

Fang, L. H., Lerner, J. and Wu, Ch. (2017). Intellectual property rights protection, ownership, and innovation: Evidence from China. Review of Financial Studies, 30(7), pp. 2446-2477.

Gersbach, H., Schetter, U. and Schneider, M. T. (2019). Taxation, innovation and entrepreneurship. Economic Journal, 129(620), pp. 1731-1781.

Graetz, M. J. and Doud, R. (2013). Technological innovation, international competition, and the challenges of international income taxation. Columbia Law Review, pp. 347-445.

Hertz, J. A. (2018). Introduction to the theory of neural computation. CRC Press.

von Hippel, E. (1988). The sources of innovation. 1988. New York, NY: Oxford University.

Johnston, R. and Gibbons, M. (1975). Characteristics of information usage in technological innovation. IEEE Transactions on Engineering Management, (1), pp. 27-34.

Izhikevich, E. M. (2003). Simple model of spiking neurons. IEEE Transactions on Neural Networks, 14(6), 15691572.

Kaul, I., Grungberg, I. and Stern, M. A. (1999). Global public goods. Global Public Goods, pp. 450.

Kline, S. J. and Rosenberg, N. (1986). An Overview of Innovation. The Positive Sum Game, Washington. DC.

Kluvankova-Oravska, T. (2004). Structural funds the challenge for sustainable development and regional disparity in the Slovak Republic. The European Journal of Social Science Research, 17(1), pp. 61-73.

Liu, W., Wang, Z., Liu, X., Zeng, N., Liu, Y. and Alsaadi, F. E. (2017). A survey of deep neural network architectures and their applications. Neurocomputing, 234, pp. 11-26.

Lømo, T. (2018). Discovering long-term potentiation (LTP)-recollections and reflections on what came after. Acta Physiologica, 222(2), pp. e12921.

Maass, W. (1997). Networks of spiking neurons: The third generation of neural network models. Neural Networks, 10(9), pp. 1659-1671.

Maass, W. (1999). Computing with spiking neurons. Pulsed Neural Networks, 2, pp. 55-85.

Malik, R. S. (2018). Educational Challenges in 21st Century and Sustainable Development. Journal of Sustainable Development Education and Research, 2(1), pp. 9-20.

McCormick, K., Abbott, D., Brown, M. S., Khabaza, T. and Mutchler, S. R. (2013). IBM SPSS modeler cookbook. Packt Publishing.

Nelson, D. B. (1990). Stationarity and persistence in the GARCH (1, 1) model. Econometric Theory, 6(3), pp. 318334.

Paunov, C. (2016). Corruption's asymmetric impacts on firm innovation. Journal of Development Economics, 118, pp. 216-231.

Ritzer, G. and Stepnisky, J. (2017). Modern sociological theory. Sage publications.

Russell, B. (2016). The impact of science on society. Routledge.

Schaffer, J. D., Whitley, D. and Eshelman, L. J. (1992). Combinations of genetic algorithms and neural networks: A survey of the state of the art. Proceedings COGANN-92: International Workshop on Combinations of Genetic Algorithms and Neural Networks, pp. 1-37.

Scriptor, A. M. G. G. (2004). A new paradigm for neurocomputing using the Boolean complete neuron (Doctoral dissertation, University of Colorado at Colorado Springs).

Sealey, Jr. C. W. and Lindley, J. T. (1977). Inputs, outputs, and a theory of production and cost at depository financial institutions. The Journal of Finance, 32(4), pp. 1251-1266.

Szambelan, S. M. and Jiang, Y. D. (2020). Effectual control orientation and innovation performance: Clarifying implications in the corporate context. Small Business Economics, 54(3), pp. 865-882.

Verbeek, A., Debackere, K., Luwel, M. and Zimmermann, E. (2002). Measuring progress and evolution in science and technology-I: The multiple uses of bibliometric indicators. International Journal of Management Reviews, 4(2), pp. 179-211.

Zhang, J. (2019). Basic Neural Units of the Brain: Neurons, Synapses and Action Potential. arXiv preprint arXiv:1906.01703.

Zhu, D., Yao, H., Jiang, B. and Yu, P. (2018). Negative log likelihood ratio loss for deep neural network classification. arXiv preprint arXiv:1804.10690.