Portal de Periódicos da CAPES

Efficient resource scheduling for the analysis of Big Data streams

2019; IOS Press; Volume: 23; Issue: 1 Linguagem: Inglês

10.3233/ida-173691

1571-4128

Mahmood Mortazavi-Dehkordi, Kamran Zamanifar,

Distributed and Parallel Computing Systems

The emergence of Big Data has had a profound impact on how data are analyzed. Open source distributed stream processing platforms have gained popularity for analyzing streaming Big Data as they provide low latency required for streaming Big Data applications using cluster resources. However, existi ng resource schedulers are still lacking the efficiency that Big Data analytical applications require. Recent works have already considered streaming Big Data characteristics to improve the efficiency of scheduling in the platforms. Nevertheless, they have not taken into account the specific attributes of analytical applications. This study, therefore, presents Bframework, an efficient resource scheduling framework used by streaming Big Data analysis applications based on cluster resources. Bframework proposes a query model using Directed Graphs (DGs) and introduces operator assignment and operator scheduling algorithms based on a novel partitioning algorithm. Bframework is highly adaptable to the fluctuation of streaming Big Data and the availability of cluster resources. Experiments with the benchmark and well-known real-world queries show that Bframework can significantly reduce the latency of streaming Big Data analysis queries up to about 65%.