Computing Two Heuristic Shrinkage Penalized Deep Neural Network Approach

Abstract

Linear models are not always able to sufficiently capture the structure of a dataset. Sometimes, combining predictors in a non-parametric method, such as deep neural networks (DNNs), would yield a more flexible modeling of the response variables in the predictions. Furthermore, the standard statistical classification or regression approaches are inefficient when dealing with more complexity, such as a high-dimensional problem, which usually suffers from multicollinearity. For confronting these cases, penalized non-parametric methods are very useful. This paper proposes two heuristic approaches and implements new shrinkage penalized cost functions in the DNN, based on the elastic-net penalty function concept. In other words, some new methods via the development of shirnkaged penalized DNN, such as ${\mathrm{DNN}}_{\mathrm{elastic-net}}$ and ${\mathrm{DNN}}_{\mathrm{ridge}\&\mathrm{bridge}}$, are established, which are strong rivals for ${\mathrm{DNN}}_{\mathrm{Lasso}}$ and ${\mathrm{DNN}}_{\mathrm{ridge}}$. If there is any dataset grouping information in each layer of the DNN, it may be transferred using the derived penalized function of elastic-net; other penalized DNNs cannot provide this functionality. Regarding the outcomes in the tables, in the developed DNN, not only are there slight increases in the classification results, but there are also nullifying processes of some nodes in addition to a shrinkage property simultaneously in the structure of each layer. A simulated dataset was generated with the binary response variables, and the classic and heuristic shrinkage penalized DNN models were generated and tested. For comparison purposes, the DNN models were also compared to the classification tree using GUIDE and applied to a real microbiome dataset.