Content area
Full text
Abstract- It was recently shown that mathematical objects called "directional fibers" can be used to systematically locate distinct fixed points in recurrent neural networks. Here, we extend that work, showing for the first time that directional fibers apply to many dynamical systems, including gradient flows in non-convex optimization. First, we extend previous algorithms and empirically demonstrate improved performance. Next, we generalize a formal correctness guarantee from prior work, which was specific to recurrent neural networks, to a broader class of problems. Finally, we apply directional fibers to a non-convex optimization problem, showing frequent recovery of the global optimum. We conclude that directional fibers are a promising new approach to fixed point location and non-convex optimization.
Keywords: Directional Fibers, Numerical Path Following, Fixed Points, Nonlinear Equations, Non-convex Optimization
(ProQuest: ... denotes formulae omitted.)
1. Introduction
Global solution of systems of non-linear equations, and of non-convex optimization problems, is a core challenge encountered widely across science and engineering. Modern computing techniques have facilitated various approaches to this challenge, such as combinatorial optimization heuristics [1], rigorous branch and bound techniques [2], [3], evolutionary computation [4], quantum algorithms [5], and repeated, randomized local search [6]. However, this remains an active research area, as all existing approaches suffer from various limitations in their generality, practicality, empirical performance, or theoretical guarantees. Therefore, it remains important to search for new approaches that can shed new insight on this grand challenge, and which have the potential to eventually lead to more effective global solution methods.
One new approach that we recently proposed is based on mathematical objects called "directional fibers" [7]. A directional fiber is a certain set of points in the domain of a vector-valued function, which contains the roots of the function. For differentiable functions that map between Euclidean spaces, directional fibers are typically curves that can be numerically traversed, which enables a systematic way of enumerating distinct roots. As such, directional fibers are a rather general root-finding apparatus. This includes finding fixed points in many dynamical systems and stationary points in many optimization problems. However, in previous work, directional fibers were only implemented, theoretically analyzed, and experimentally studied for the special case of fixed point location in recurrent neural networks (RNNs).
This paper extends that previous work to a...