A Scalable Framework for Sensor Data Ingestion and Real-Time Processing in Cloud Manufacturing

Abstract

Cloud Manufacturing enables the integration of geographically distributed manufacturing resources through advanced Cloud Computing and IoT technologies. This paradigm promotes the development of scalable and adaptable production systems. However, existing frameworks face challenges related to scalability, resource orchestration, and data security, particularly in rapidly evolving decentralized manufacturing settings. This study presents a novel nine-layer architecture designed specifically to address these issues. Central to this framework is the use of Apache Kafka for robust, high-throughput data ingestion, and Apache Spark Streaming to enhance real-time data processing. This framework is underpinned by a microservice-based architecture that ensures a high scalability and reduced latency. Experimental validation using sensor data from the UCI Machine Learning Repository demonstrated substantial improvements in processing efficiency and throughput compared with conventional frameworks. Key components, such as RabbitMQ, contribute to low-latency performance, whereas Kafka ensures data durability and supports real-time application. Additionally, the in-memory data processing of Spark Streaming enables rapid and dynamic data analysis, yielding actionable insights. The experimental results highlight the potential of the framework to enhance operational efficiency, resource utilization, and data security, offering a resilient solution suited to the demands of modern industrial applications. This study underscores the contribution of the framework to advancing Cloud Manufacturing by providing detailed insights into its performance, scalability, and applicability to contemporary manufacturing ecosystems.