Insights and Applications From Data Driven Representation Learning

Abstract

This dissertation is an exploration of data-driven discovery, inspired by neuroscientific studies of the brain. Each of the three projects listed will describe a different domain of input data (self-driving video, medical image, and biological audio), and how investigating neural network behavior trained on that data can reveal insights for each underlying task. Chapter 2 assess motion selectivity in a self-driving network trained to predict two output tasks: steering and motor. We show how different control conditions can define temporal behavior, as well as how frame order is only implicitly learned if relevant for the task, even if the frame order is present in the input and output training data. Chapter 3 assesses self-supervisedly learned representations from retinal fundus images. We show how these learned representations can drive a voting scheme classifier to match supervised and human expert baselines for disease severity prediction in this field, minimizing the bias enforced from clinically relevant ground truth labels. These representations can be further probed to discover mislabeled or easily confused data, as well as phenotype groupings in retinal images that pertain to other pathology and physiology of the subject. These imply NPID and cluster analysis tools could aid clinicians organize and label data from multiple tasks, an expensive process that requires uncommon expertise. Similarly, Chapter 4 extends this idea about data-driven learning to the audio domain. Here, self-supervisedly learned representations from zebra finch data yielded feature encodings that were functionally relevant for classifying vocalization calls, driving a voting scheme classifier to match supervised baseline performance on a generally difficult task of intra-species audio discrimination. We convert audio waveforms to spectrogram image representations of sound signals, and train a CNN on these inputs, so we can probe these visually-defined audio features. To do so, we assessed how neuronal behavioral preferences can be described by a mid-level representation space of audio (the modulation power spectrum), as well as how these features compare to mid-level audio features correlated with zebra finch brain activity. Data-driven algorithms can learn representations with minimal bias, so commonalities between artificial and biological neural systems imply similar encodings are optimally learned. All in all, this dissertation has evaluated deep learning applied on a host of real world tasks aside from standard datasets curated for computer vision. Though each project requires a different lens for explaining functionally salient behavior, we offer data-driven insights into each underlying task that seem to be consistent with experimental findings in neuroscience and medicine.