Content area
As monitoring technologies and data collection methodologies advance, landslide disaster data reflects attributes such as diverse sources, heterogeneity, substantial volumes, and stringent real-time requirements. To bolster the data support capabilities for the monitoring, prevention, and management of landslide disasters, the efficient integration of multi-source heterogeneous data is of paramount importance. The present study proposes an innovative approach to integrate multi-source landslide disaster data by combining the Flink-oriented framework with load balancing task scheduling based on an improved particle swarm optimization (APSO) algorithm. It utilizes Flink’s streaming processing capabilities to efficiently process and store multi-source landslide data. To tackle the issue of uneven cluster load distribution during the integration process, the APSO algorithm is proposed to facilitate cluster load balancing. The findings indicate the following: (1) The multi-source data integration method for landslide disaster based on Flink and APSO proposed in this article, combined with the structural characteristics of landslide disaster data, adopts different integration methods for data in different formats, which can effectively achieve the integration of multi-source landslide data. (2) A multi-source landslide data integration framework based on Flink has been established. Utilizing Kafka as a message queue, a real-time data pipeline was constructed, with Flink facilitating data processing and read/write operations for the database. This implementation achieves efficient integration of multi-source landslide data. (3) Compared to Flink’s default task scheduling strategy, the cluster load balancing strategy based on APSO demonstrated a reduction of approximately 4.7% in average task execution time and an improvement of approximately 5.4% in average system throughput during actual tests using landslide data sets. The research findings illustrate a significant improvement in the efficiency of data integration processing and system performance.
Details
Disasters;
Particle swarm optimization;
Task scheduling;
Data processing;
Datasets;
Algorithms;
Optimization;
Load distribution (forces);
Load distribution;
Data analysis;
Data integration;
Clusters;
Monitoring;
Data collection;
Heterogeneity;
Load balancing;
Efficiency;
Distributed processing;
Geology;
Scheduling;
Big Data;
Remote sensing;
Landslides & mudslides;
Streaming;
Emergency communications systems;
Earthquakes;
Spatial data;
Real time;
Integration
; Li, Mengyu 1 ; Cai, Yingchun 1
1 School of Water Conservancy and Transportation, Zhengzhou University, Zhengzhou 450001, China;