Terramycelium: a reference architecture for adaptive big data systems

Abstract

Modern organizations generate and process unprecedented volumes of structured, semi-structured, and unstructured data from diverse sources, creating significant architectural and engineering challenges for traditional data processing systems. Industry analyses consistently report failure rates of 60-85% for Big Data projects, with architectural limitations identified as a primary contributing factor. Current reference architectures suffer from monolithic designs, inadequate cross-cutting concerns (security, privacy, metadata), and limited adaptability to evolving data ecosystems. This paper presents Terramycelium, a novel reference architecture for Big Data systems that addresses these limitations through a domain-driven, event-oriented approach. The architecture integrates principles from complex adaptive systems, domain-driven design, distributed systems, and event-driven architectures to enable autonomous domain-specific data ownership while maintaining system-wide coherence through asynchronous event communication. We developed Terramycelium following empirically grounded reference architecture guidelines and evaluated it through two complementary methods: a case-mechanism experiment and expert opinion assessment. The case-mechanism experiments demonstrated the architecture’s capability to process 1.693GB of data with 50-100 second latency, handle 771,305 streaming messages with 0.0000148 second ingestion latency, and maintain stable performance with 24% CPU utilization under high-volume scenarios. Expert evaluation (n=3, 10-32 years experience) validated the architecture’s innovative integration of domain-driven design with data engineering, while identifying implementation complexity and organizational readiness as adoption challenges. Terramycelium contributes a validated approach for building scalable, maintainable Big Data systems that addresses the limitations of existing monolithic architectures while aligning with modern software engineering practices.