Jumping into the Stream with EMA's Hybrid Data Ecosystem

Written by John Myers, Managing Research Director, Business Intelligence and Data Warehousing

Real-time data is flooding organizations. It comes from customer- and partner-facing websites and mobile apps. It comes from customers via social media posts. It comes from IoT sensor data on connected devices such as fleet vehicles, manufacturing equipment, and utility networks. Some experts and pundits are suggesting that information has become a more valuable resource to the economy than fossil fuels. Data is the new oil…or rather, data is the renewable resource that powers data-driven business.

In this era, as organizations focus on information to power their business models, there is a unique opportunity to utilize streaming datasets to expand how an organization gains informational insight across a range of business dimensions. This insight can impact how organizations view and evaluate their customer base to strengthen and expand revenues. It can improve the operation and management of companies and business units to optimize the allocation of corporate resources, while simultaneously reducing costs. Improved information based on streaming data from products and services can launch entirely new revenue streams.

With this wide range of opportunities to create value with streaming data, enterprises are challenged to implement the supporting technology environments that take advantage of enterprise application, social media, and IoT data across their organization. The problem is not that the individual technical tasks enable digital transformation, but new business streaming models or data-driven exploration taken by themselves are too challenging. The reason organizations struggle to implement these types of next-generation data management environments is because of the combined complexity of:

• Integrating streaming data from thousands of streaming sources
• Managing the workloads associated with multiple usage scenarios such as exploration, operational processing, and analytics
• Presenting information in the correct format, linking real-time processing such as machine learning algorithms into to the day-to-day lives of employees, customers, and partners

These streaming events have great value, but companies should capitalize and link this real-time information to an organization’s operations quickly to truly take advantage of that value. Business stakeholders need to validate the information and insights derived from data streams and pipelines, and technologists need to certify that the exploration environments, inputs to operational processes, and apps supporting value-added insights are in short order. Without a production-ready deployment, business opportunities are missed and confidence in the information and/or application is lost.

To truly take advantage of these data streams, whether they are in an exploration and discovery environment to evaluate and validate data or a streaming data app to take real-time action, enterprises need to ingest, process, and organize that information quickly and effectively. When enterprises want to bring together the disparate data underlying these multiple streaming usage scenarios, they require a coordinated environment to manage and optimize the implementation of data and information in their business. This coordinated environment needs to place the business requirements of next-generation data management at its core and include the “right” technical elements around them.

The Enterprise Management Associates’ (EMA) Hybrid Data Ecosystem (HDE) provides the framework to link the business requirements of next-generation data management environments to support not only historical data at rest, but the flows of streaming data that are quickly inundating organizations. An effective Hybrid Data Ecosystem, like all next-generation data management environments, provides the proper “connective tissue” between multiple data management platforms, data formats, and usage scenarios. By reducing the complexity of managing and coordinating these streaming data flows, an HDE architecture breaks down the barriers to the complexity using technologies that:

• Allow business stakeholders such as analysts and data scientists to access technical functionality like metadata management and data integration pipelines
• Enable technologists within the CIO’s office and the IT department to scale their efforts across multiple projects
• Give IT architects and directors the flexibility to add new platforms without being “locked” into an architectural decision environment