On the Design of a Framework for Large-Scale Exploratory Graph Analytics

Abstract

Large-scale exploratory graph analytics merges data science with high-performance computing to extract critical insights from network-representable data. Data scientists routinely analyze data from the natural, social, and computing sciences by representing it as networks, or graphs, where objects become vertices and their relationships become edges. This representation allows data scientists to add graph analytics to their toolbox. However, designing tools for large-scale exploratory graph analytics is challenging due to the complexities of graph algorithms, such as high communication in distributed systems and large memory demands. These challenges can lead to overly complex software, which limits usability and development to a small group of specialists. Frameworks for large-scale exploratory graph analytics should be designed to integrate high-performance computing with user-friendly interfaces, both at the developer and user levels, minimizing the need for extensive technical training.

This dissertation introduces Arachne, a revolutionary, open-source framework for large-scale exploratory graph analytics that is usable through Python and built on C, C++, and Chapel. Arachne provides kernels for a variety of graph algorithms that take advantage of both shared-memory and distributed-memory computing architectures, using a highly distributable and parallelizable graph data structure. As the first framework of its kind, Arachne provides large-scale exploratory graph analytic capabilities regardless of the system it runs on by adapting its graph kernels specifically to their runtime environment. This allows data scientists to run high-performance graph analysis anywhere—from their personal work systems to supercomputers.