Parallel computation method for incremental cloud services based on Hadoop

DOI

10.2991/iceeecs-16.2016.220How to use a DOI?

Keywords

Parallel computation; incremental cloud services; Hadoop

Abstract

Performance of Hadoop platform is closely related to the task scheduler. Based on the analysis of the existing Hadoop platform's job scheduling algorithm, I propose a multi-queue job scheduling optimization algorithm in the paper. Based on the actual test and analysis of Hadoop platform, it can be seen that the optimization algorithm proposed in this paper can effectively allocate the node resources according to the degree of demand for the node resource. It can achieve the resource sharing among multiple queues. And at the same time, it can also effectively avoid the ping-pong effect brought by resource competition at the same time. Optimization algorithm has greater improvements in execution efficiency than Hadoop default algorithm.

Copyright

© 2016, the Authors. Published by Atlantis Press.

Open Access

This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Download article (PDF)

TY  - CONF
AU  - Yonghong Li
AU  - Na Zhou
AU  - Guofeng Zhao
PY  - 2016/12
DA  - 2016/12
TI  - Parallel computation method for incremental cloud services based on Hadoop
BT  - Proceedings of the 2016 4th International Conference on Electrical & Electronics Engineering and Computer Science (ICEEECS 2016)
PB  - Atlantis Press
SP  - 1146
EP  - 1149
SN  - 2352-538X
UR  - https://doi.org/10.2991/iceeecs-16.2016.220
DO  - 10.2991/iceeecs-16.2016.220
ID  - Li2016/12
ER  -