Portal de Periódicos da CAPES

NMSTREAM: A SCALABLE EVENT-DRIVEN ETL FRAMEWORK FOR PROCESSING HETEROGENEOUS STREAMING DATA

2018; Copernicus Publications; Volume: IV-4; Linguagem: Inglês

10.5194/isprs-annals-iv-4-243-2018

2194-9050

Fuyuan Xiao, C. Li, Zheng Wu, Yun Wu,

Data Quality and Management

Abstract. ETL (Extraction-Transform-Load) tools, traditionally developed to operate offline on historical data for feeding Data-warehouses need to be enhanced to deal with continuously increased streaming data and be executed at network level during data streams acquisition. In this paper, a scalable and web-based ETL system called NMStream was presented. NMStream is based on event-driven architecture and designed for integrating distributed and heterogeneous streaming data by integrating the Apache Flume and Cassandra DB system, and the ETL processes were conducted through the Flume agent object. NMStream can be used for feeding traditional/real-time data-warehouses or data analytic tools in a stable and effective manner.