Content area
Deep neural networks have been shown to provide exceptional performance in a multitude of applications. Recent developments lead to a streamlining of their applicability to many areas beyond research, where the development and training of specific models has become a common task for practitioners, comparable to the early days of software development. Despite their impressive capabilities, deep neural networks are highly complex, containing up to billions of learned parameters. This property prevents any deep human understanding of the inner workings of the prediction strategies of these models, adequately earning them the title of “black box predictors”. This is particularly problematic as these models may be subject to Clever Hans behavior, which describes a correct prediction within the boundary of the known data distribution for the wrong reason, caused by causally unrelated, spurious correlations within the training data. The field of explainable artificial intelligence aims to unveil the decision strategies of deep neural networks in a humanly interpretable way. Here, the field's advancement has provided us with various tools of feature attribution, such as layer-wise relevance analysis, which analyze models in terms of the contributions of their inputs to the model’s output.
In the face of the practical adoption of deep neural networks, this thesis aims to provide a framework for the debugging of deep neural networks with respect to Clever Hans behavior based on these tools of explainable artificial intelligence. We demonstrate the practicality of feature attribution methods, specifically layer-wise relevance attribution, to conduct a manual analysis of a patch-based convolutional neural network trained on video data. This provides us with insights into a potential Clever Hans behavior caused by the pre-processing of its data, which we can leverage to improve the model’s performance. Subsequently, we provide a semi-automated solution to the identification of Clever Hans behavior by extending an approach called spectral relevance analysis. Based on the labels provided by our solution, we introduce the class artifact compensation framework, which includes two approaches to verify and mitigate the previously identified Clever Hans behavior. Given the dependence of the identification process on feature attribution approaches, we theoretically and empirically establish that these can be arbitrarily modified without changing their output on the known data. Subsequently, we provide an approach based on our theoretical insights to robustify feature attribution methods against this vulnerability. Finally, we provide a pipeline of three software frameworks, Zennit for feature attribution, CoRelAy for extended spectral relevance analysis, and ViRelAy for the visualization and interactive identification of Clever Hans behavior. We draw a parallel to software development and the necessity for debugging tools, and argue that readily available debugging tools are equally required for the development and learning of deep neural networks. With our introduced workflow, analysis and software frameworks, we hope to provide a basis for such debugging tools, in order to boost research in the field of explainable artificial intelligence and enable practitioners to develop reliable and trustworthy models.