A Highly-Efficient Fault Tolerance Method for a Scalable Stream Processing System

DOI

10.2991/caai-17.2017.50How to use a DOI?

Keywords

SPS; dynamic scale out; fault tolerance

Abstract

An effective fault-tolerant mechanism becomes essential to the credible stream processing system (SPS). However, the traditional introduction of fault-tolerant mechanisms has a relatively negative influence on the systemic calculation efficiency. And enhancing the system scale out is an effective way to solve this problem. This paper proposes a highly-efficient fault recovery method that is for an extensible SPS. On the one hand, based on a reasonable partition and upstream backup of its internal calculation status. The SPS can monitor the bottleneck state of the operators and then migrate the upstream backup state of the operators that is in need of scale out to the new nodes to achieve dynamic scale out of the SPS; On the other hand, when a node fault in the SPS, the system can dynamically expand a secondary node for the erroneous one, and then use the upstream backup algorithm to achieve fault recovery. In this paper, a mathematical model is built for the SPS. And based on this, the fault recovery time and computational efficiency of the rapid fault tolerant method are analyzed. Experimental results have shown that the one proposed in this paper is more effective in fault tolerance which gives the system a higher calculation efficiency by reducing the frequency of upstream backup of node state.

Copyright

© 2017, the Authors. Published by Atlantis Press.

Open Access

This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Download article (PDF)

TY  - CONF
AU  - Guanghui Chang
AU  - Peizhen Li
AU  - Guangxia Xu
PY  - 2017/06
DA  - 2017/06
TI  - A Highly-Efficient Fault Tolerance Method for a Scalable Stream Processing System
BT  - Proceedings of the 2017 2nd International Conference on Control, Automation and Artificial Intelligence (CAAI 2017)
PB  - Atlantis Press
SP  - 226
EP  - 230
SN  - 1951-6851
UR  - https://doi.org/10.2991/caai-17.2017.50
DO  - 10.2991/caai-17.2017.50
ID  - Chang2017/06
ER  -