Huge Data Analysis and Processing Platform based on Hadoop

DOI

10.2991/msmee-17.2017.56How to use a DOI?

Keywords

Huge data analysis, Hadoop, HDFS, MapReduce

Abstract

SGC puts forward the major initiatives of the construction of strong smart grid and three integrates and five big systems, requiring the informationlization further improves to a level on the basis of the inheritance for fruit and deepening application, having even higher requirements especially for huge information data processing, data search grid. In response to meet the needs of the smart grid to the huge amounts of data processing, this article puts forward building dynamic scalable huge amounts of data processing platform on the basis of Hadoop in the virtualization resources management platform through in-depth analysis and research of the two core key technology, HDFS and MapReduce and gives its technical architecture, implementation plan and case analysis. Compared with traditional Hadoop distributed parallel computing system based on the physical machine deployment, to build Hadoop virtual server templates through virtualization platform not only can quickly complete the Hadoop deployment of distributed parallel computing system and can effectively make use of computing resources.

Copyright

© 2017, the Authors. Published by Atlantis Press.

Open Access

This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Download article (PDF)

TY  - CONF
AU  - Yuanbin Li
AU  - Rong Chen
PY  - 2017/05
DA  - 2017/05
TI  - Huge Data Analysis and Processing Platform based on Hadoop
BT  - Proceedings of the 2017 2nd International Conference on Materials Science, Machinery and Energy Engineering (MSMEE 2017)
PB  - Atlantis Press
SP  - 267
EP  - 271
SN  - 2352-5401
UR  - https://doi.org/10.2991/msmee-17.2017.56
DO  - 10.2991/msmee-17.2017.56
ID  - Li2017/05
ER  -