Hadoop Performance Tuning based on Parameter Optimization

DOI

10.2991/snce-18.2018.118How to use a DOI?

Keywords

Hadoop, TeraSort, Parameter optimization, Performance tuning

Abstract

In order to better verify that Hadoop performance can be improved through optimization of parameters, we can use the following test methods: benchmarking, stability testing, high availability testing, scalability testing, and security testing. In this paper, the benchmark test method is used to verify the optimization of parameters and to optimize the performance of Hadoop. This article mainly focuses on the 17 parameters in Tab.1. The optimization results are shown in Tab.3. The optimization of the parameters was verified by the execution time of the TeraSort algorithm in the benchmark test. During the experiment, the CPU and memory utilization rate, disk IO and network IO throughput and other indicators were collected. Fig.1-3 fully illustrates the comparison between Hadoop and TeraSort algorithm after parameter default value and parameter adjustment. The experimental results show that after the Hadoop parameters are adjusted and optimized, the Hadoop performance tuning is achieved under certain conditions.

Copyright

© 2018, the Authors. Published by Atlantis Press.

Open Access

This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Download article (PDF)

TY  - CONF
AU  - Wei Wang
AU  - Yong Shi
AU  - Xin Liu
AU  - YiHong Feng
AU  - Ning Tao
PY  - 2018/05
DA  - 2018/05
TI  - Hadoop Performance Tuning based on Parameter Optimization
BT  - Proceedings of the 8th International Conference on Social Network, Communication and Education (SNCE 2018)
PB  - Atlantis Press
SP  - 577
EP  - 580
SN  - 2352-538X
UR  - https://doi.org/10.2991/snce-18.2018.118
DO  - 10.2991/snce-18.2018.118
ID  - Wang2018/05
ER  -