Streaming Big Data into the enterprise architecture : challenges and opportunities

Business Intelligence and Data Analytics (BI & DA) have been used for years to evaluate bulk data but these technologies reach their limits when the data involved is ever more ephemeral, unstructured and high-volume. Big Data solution(s), which support BI & DA, are enabled by recent advances in technologies and architecture to handle unstructured and high volume data. However, Big Data problems are complex to analyze and solve. Challenges include classifying Big Data so as to be able to choose Big Data solution(s) to fit into the Enterprise Architecture (EA) of an organization. It is important to efficiently deliver real-time analytic processing on constantly changing streaming data and enable descriptive, predictive and prescriptive analytics to support real-time decisions for BI. When a significant amount of data needs to be processed quickly in near real-time to gain insights, this is a form of streaming data. Streaming data changes constantly and can be in many forms e.g. web data, audio/video data and external data. To date organizations have not handled streaming data well. Using streaming data for BI and DA is a new paradigm in analytics and will have impacts on technology infrastructure components and data architectures, since most data is directly generated in digital format today. The technology infrastructure components and the technology architectures as well as data architectures changes must be captured by an organization’s existing EA to enable conducting BI and DA, using a holistic approach at all times, for the successful development and execution of an organization’s strategy. In this paper we discuss streaming of Big Data into EA for BI and DA to capture streaming data and outline key challenges and opportunities in integrating Big Data solutions. We also examine data architecture for streaming Big Data into the EA for BI & DA.