Latency-Aware Cloud Pipelines: Redefining Real-Time Data Integration with Elastic Engineering Models

Abstract

More and more enterprises across verticals are adopting data-driven decision-making behaviors. Managing the whole life cycle of large amounts of enterprise data, which consists of several processes such as replication, synchronization, and transmission, imposes extra burdens on enterprise budget and increases the time-to-market of new interactive and ad-hoc analytics. In parallel, new IT paradigms for deploying services, specifically, Next Generation Cloud Computing in the form of elastic and thin cloud pipelines, have emerged as cost-effective enablers for real time data management and processing. This paradigm breaks the limitation of both proprietary and highly costly large scale on-premise infrastructures and traditional batch cloud service models. However, existing cloud provider pipeline offerings mainly focus on computation and storage, and do not yet expose robust data integration services. This paper presents a comprehensive elastic engineering model that abstracts both the deployments of pipelines, such as ETL, streaming, and data synchronization, or replication, as well as their management, with a focus on maximum flexibility and automation. Based on a design-time model, we investigate data pipeline templating including dependency and latency-aware provisioning and orchestration of pipelines on next generation cloud and cloud hybrid environments. This is complemented with a run-time management engine that dynamically augments the original model with fine-grain execution knowledge and tracks performance. Finally, we demonstrate our approach with a full implementation in the context of two use case areas: Enterprise Information Management and the further externalization of data generated by Internet of Things enabled things, with a focus on transportation.