Weight versus Node Perturbation Learning in Temporally Extended Tasks: Weight Perturbation Often Performs Similarly or Better

Abstract

Biological constraints often impose restrictions on plasticity rules such as locality and reward-based rather than supervised learning. Two learning rules that comply with these restrictions are weight (WP) and node (NP) perturbation. NP is often used in learning studies, in particular, as a benchmark; it is considered to be superior to WP and more likely neurobiologically realized, as the number of weights and, therefore, their perturbation dimension typically massively exceed the number of nodes. Here, we show that this conclusion no longer holds when we take two properties into account that are relevant for biological and artificial neural network learning: First, tasks extend in time and/or are trained in batches. This increases the perturbation dimension of NP but not WP. Second, tasks are (comparably) low dimensional, with many weight configurations providing solutions. We analytically delineate regimes where these properties let WP perform as well as or better than NP. Furthermore, we find that the changes in weight space directions that are irrelevant for the task differ qualitatively between WP and NP and that only in WP gathering batches of subtasks in a trial decreases the number of trials required. This may allow one to experimentally distinguish which of the two rules underlies a learning process. Our insights suggest new learning rules which combine for specific task types the advantages of WP and NP. If the inputs are similarly correlated, temporally correlated perturbations improve NP. Using numerical simulations, we generalize the results to networks with various architectures solving biologically relevant and standard network learning tasks. Our findings, together with WP’s practicability, suggest WP as a useful benchmark and plausible model for learning in the brain.

Alternate abstract:

Plain Language Summary

Neural networks that receive unspecific feedback about their performance can still learn in a reward-based manner. Two common reward-based rules are weight and node perturbation learning, which start by altering the network weights and the neuronal activity, respectively. Thereafter, perturbations that improve network performance are appropriately consolidated while others are inverted. Weight perturbation is considered much less efficient; therefore, node perturbation has become a standard benchmark for reward-based learning. Furthermore, it is considered the more plausible model for learning in biological systems, although its neurobiological implementation appears more difficult. In contrast, we show that for broad classes of tasks, weight perturbation performs similarly or better, such that it is useful as a benchmark and a plausible candidate for learning in the brain.

Specifically, we consider tasks with two features that are common in biology and artificial neural network learning: First, the tasks extend in time or are presented in batches during training. Second, the neural network dynamics use few of their available dimensions. We find that then the argument in favor of node perturbation no longer holds. In particular, weight perturbation performs comparably well or better in various biologically relevant and standard artificial neural network learning applications. We explain our findings using intuitive arguments and rigorous mathematical analysis. Observed qualitative differences in the error and weight changes during learning may allow one to distinguish the two rules in biological systems.

The demonstrated, unexpected performances push the field of basic reward-based learning into a new direction, prompting weight perturbation’s use in concrete biological modeling and as a benchmark, and sparking the development of new weight perturbation-related algorithms.