Improving Performance in Hadoop Using Automatic and Predictive Configuration

DOI

10.2991/ceis-16.2016.54How to use a DOI?

Keywords

high performance; dynamic; prediction; big data

Abstract

MapReduce is an effective programming model for analyzing large-scale data. Hadoop-a distributed processing system is widely used nowadays. Improving the task parallelism can be a key point to improve the MapReduce performance in Hadoop. In this paper, we address the problem in two ways. On the one hand we can run the tasks with some dynamic configurations. On the other hand, considering of the difference of tasktracker we use mathematics method to predict the cups' utilization of tasktracker to assign the task. Experimental results on both ways show we can improve the performance in Hadoop by improving the task parallelism.

Copyright

© 2017, the Authors. Published by Atlantis Press.

Open Access

This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Download article (PDF)

TY  - CONF
AU  - Juan Fang
AU  - Hao Sun
AU  - Li-Fu Zhou
AU  - Xing-Tian Ren
AU  - Min Cai
PY  - 2016/11
DA  - 2016/11
TI  - Improving Performance in Hadoop Using Automatic and Predictive Configuration
BT  - Proceedings of the 2016 International Conference on Computer Engineering and Information Systems
PB  - Atlantis Press
SP  - 275
EP  - 278
SN  - 2352-538X
UR  - https://doi.org/10.2991/ceis-16.2016.54
DO  - 10.2991/ceis-16.2016.54
ID  - Fang2016/11
ER  -