Content area
Full Text
Abstract
The advent of big data technologies-and associated hype-can leave data warehouse professionals and business users doubtful but hopeful about leveraging new sources and types of data. This confusion can impact a project's ability to meet expectations. It can also polarize teams into "which one will we use" thinking.
Good architectures address the cost, benefits, and risks of every design decision. Good architectures draw upon existing skills and tools where they make sense and add new ones where needed. We architects always use the right tool for the job.
In this article, we describe the parts of the Hadoop framework that are most relevant to the data warehouse architect and developer. We sort through the reasons an organization should consider big data solutions such as Hadoop and why it's not a battle of which (classic data warehouse or big data) is best. Both can-and should-exist together in the modern data architecture.
Introduction
The concept of data warehousing has been with us for at least 30 years and has reached maturity within IT organizations and among data analysts. In the 1990s, online analytical processing (OLAP) systems allowed analysts to perform operations that might not have been possible in other solutions during that period. However, newer, disruptive technologies have been introduced that change overall system architecture and approaches to large-scale data analysis.
There has been a good deal of us-versus-them controversy in the relational and non-relational database world, mostly due to the mistaken belief that an organization must choose one over the other. As we have seen with many technologies over the decades, finding the right tool for the job is paramount to support business needs. Platform wars rarely benefit our organizations.
Big data technologies have moved beyond the "only for Web start-ups" or "only for scientific use" phase and are now ready to answer real-world business questions.
A Data Story
Many stories used to explain big data and Hadoop use social media and scientific sensor data-all wonderful examples of the divergence from traditional data. However, these examples sometimes leave traditional enterprise users feeling as if there are no applications in their world for these technologies.
Big data isn't just about using new tools; it's about solving problems that could be too expensive to solve...