Loop Query of Big Data with Low Transmission Cost

DOI

10.2991/mse-15.2016.60How to use a DOI?

Keywords

fetch operation, loop query, big data, network transmission

Abstract

At present, big data environment in electric power field develops. Our platform has a poor efficiency of data analysis and the main bottleneck is disk I/O time of fetch operations. In order to address the problem, we use the characteristics that several tasks usually execute on the same data sets simultaneously, and propose a Loop Query method based on Fetch operations sharing (LQF). It lets all the operations on the data set to process its each record circularly, which largely improves the efficiency of data analysis. We also present a Data Placement algorithm with low Transmission cost (DPT). It extends the queries to the cloud environment and reduces the result transmission cost of fetch operations. Last extensive experiments show that the proposed LQF algorithm performs 75.4% better than the benchmark solution when the number of tasks is large, and the DPT algorithm has on average 47.6% less execution time than random placement method.

Copyright

© 2016, the Authors. Published by Atlantis Press.

Open Access

This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Download article (PDF)

TY  - CONF
AU  - Yan Ma
AU  - Yufeng Chen
AU  - Gang Kong
AU  - Suhong Chen
PY  - 2016/03
DA  - 2016/03
TI  - Loop Query of Big Data with Low Transmission Cost
BT  - Proceedings of the 2015 International Conference on Mechanical Science and Engineering
PB  - Atlantis Press
SP  - 368
EP  - 372
SN  - 2352-5401
UR  - https://doi.org/10.2991/mse-15.2016.60
DO  - 10.2991/mse-15.2016.60
ID  - Ma2016/03
ER  -