Resource Estimation in Distributed Data Stream Processing Systems

DOI

10.2991/wartia-16.2016.361How to use a DOI?

Keywords

Big Data, real-time stream processing system, resource estimation.

Abstract

Distributed data stream processing systems(DSPS) are widely used in real-time massive data processing scenarios for its characteristics of real-time and high throughout. In the real-world DSPS, the fluctuating arrival rate of the input data leads the consuming computing resource of DSPS to be time-variable. To guarantee the performance of DSPS, the accurate prediction of DSPS’s consuming resources is necessary. In this paper, we proposed approaches to make the online prediction of computing resources that DSPS consumes. We monitor the usage of computing resources such as CPU and memory in a DSPS, and use temporal data streams clustering algorithm and linear regression method to make online prediction of CPU resources and memory resources respectively. Our prediction approaches are proved efficient and quickly enough.

Copyright

© 2016, the Authors. Published by Atlantis Press.

Open Access

This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Download article (PDF)

TY  - CONF
AU  - Minglu Fan
AU  - Yi Liang
AU  - Fei Liu
AU  - Mangmang Yang
AU  - Haihua Wang
PY  - 2016/05
DA  - 2016/05
TI  - Resource Estimation in Distributed Data Stream Processing Systems
BT  - Proceedings of the 2016 2nd Workshop on Advanced Research and Technology in Industry Applications
PB  - Atlantis Press
SP  - 1822
EP  - 1825
SN  - 2352-5401
UR  - https://doi.org/10.2991/wartia-16.2016.361
DO  - 10.2991/wartia-16.2016.361
ID  - Fan2016/05
ER  -