A kind of Prefetching Data Way to Hadoop MapReduce Environments

DOI

10.2991/icmmita-16.2016.234How to use a DOI?

Keywords

Hadoop; MapReduce; Prefetching ;distributed computing.

Abstract

In MapReduce environment, problem of data redundancy,a large number of tasks to be processing and mass data storage come up, in order to solve these problems, we put forward the way of data prefetching , preprocessing of hid the remote data access latency, by adjusting the allocation of resources to Reduce the business, we put forward the method of than before, caused by the Reduce tasks of remote data access performance problems caused by time delay and resource competition system, the hidden by prefetching data the method Reduce task of remote data access latency, and Reduce task control through the resource allocation, to Reduce resource competition caused the Reduce task, the experiment results show that with the default Hadoop graphs and Hadoop Online Prototype (HOP), compared with the method the system performance can be improved by more than 10%.

Copyright

© 2017, the Authors. Published by Atlantis Press.

Open Access

This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Download article (PDF)

TY  - CONF
AU  - Hui Xia
AU  - Peng Wu
PY  - 2017/01
DA  - 2017/01
TI  - A kind of Prefetching Data Way to Hadoop MapReduce Environments
BT  - Proceedings of the 2016 4th International Conference on Machinery, Materials and Information Technology Applications
PB  - Atlantis Press
SN  - 2352-538X
UR  - https://doi.org/10.2991/icmmita-16.2016.234
DO  - 10.2991/icmmita-16.2016.234
ID  - Xia2017/01
ER  -