Improved YARN resource scheduling algorithm based on network load sensing

DOI

10.2991/aest-16.2016.114How to use a DOI?

Keywords

multitask scheduling; YARN; task flow graph; greedy strategy; network resource.

Abstract

Confronted with multitask scheduling, YARN may conduct a precise calculation and allocation towards every single task's computing and memory resources,however it failed to consider the impact on task fulfillment and cluster resources utilization. In this paper, we propose a new algorithm to allocate the data interconnected tasks to the same node or to another closely-located low network load node. This new algorithm firstly builds a task flow graph for the task requests of an application, and then assigns tasks more properly using the greedy strategy, based on the state of network resource. In this way, it significantly improves the management performance of YARN in the case of network congestion or instability. Besides, it consumesless execution time, makes higher utilization of CPU and requires lower network load.

Copyright

© 2016, the Authors. Published by Atlantis Press.

Open Access

This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Download article (PDF)

TY  - CONF
AU  - Xuyang Ding
AU  - Xiaohui Liu
AU  - Rui Zhang
AU  - Shan Tang
AU  - Linqin Sun
AU  - Ying Xie
PY  - 2016/11
DA  - 2016/11
TI  - Improved YARN resource scheduling algorithm based on network load sensing
BT  - Proceedings of the 2016 International Conference on Advanced Electronic Science and Technology (AEST 2016)
PB  - Atlantis Press
SP  - 852
EP  - 859
SN  - 1951-6851
UR  - https://doi.org/10.2991/aest-16.2016.114
DO  - 10.2991/aest-16.2016.114
ID  - Ding2016/11
ER  -