A Reinforcement Learning-based Approach to in-Pool Memory Allocation for Distributed Heterogeneous Memory Pools

Abstract

Conference Title: 2024 7th International Conference on Mechatronics and Computer Technology Engineering (MCTE)

Conference Start Date: 2024 Aug. 23

Conference End Date: 2024 Aug. 25

Conference Location: Guangzhou, China

The conventional memory allocation method in distributed heterogeneous memory pool mainly uses the Spark Shuffle skew tuning execution algorithm to calculate the allocation parameters, which is vulnerable to changes in temporary storage task nodes, resulting in excessive memory overflow in the pool. Therefore, a memory allocation method in distributed heterogeneous memory pool using reinforcement learning is proposed. That is, the memory allocation manager in the distributed heterogeneous memory pool is designed by using reinforcement learning, and the memory transmission optimization mechanism in the distributed heterogeneous memory pool is generated, thus realizing the memory allocation in the heterogeneous memory pool. The experimental results show that after using the memory allocation method in the distributed heterogeneous memory pool reinforcement learning pool designed in this paper, the memory overflow in the pool under different types of memory pools is low, which proves that the designed memory allocation method has good allocation effect, high efficiency, and certain application value, and has made certain contributions to improving the quality of distributed heterogeneous storage.