A performance modeling-based HADOOP configuration tuning strategy

Abstract

Hadoop that is a distributed system infrastructure provides a stable and reliable set of interfaces for applications. This paper proposes a Hadoop configuration parameter tuning method H-Tune, it is based on integrated learning modeling and meta-heuristic optimization. The experimental results show that the performance model can accurately predict the running time of MapReduce applications. After tuning with the Hadoop configuration parameter method proposed in this paper, the average acceleration ratio is 9.6 times and 1.5 times, respectively, and the performance of MapReduce application has been significantly improved.