Multi-link query optimization based on heuristic search in data stream

DOI

10.2991/jiaet-18.2018.4How to use a DOI?

Keywords

heuristic search; data stream; undirected weighted graph; connection tree

Abstract

In the stream processing, the connection operation is one of the very important basic operations, and it plays an important role in real-time data query and mining. According to the existing data stream connection technology and combining with the characteristics of coarse granularity processing unit of Sparkstreaming platform, this paper designs a multiple connection query optimization algorithm based on heuristic search. The algorithm first transforms the query into an undirected weighted graph, constructs the connection tree through the heuristic function to optimize the connection order, and combines the sliding window technology to introduce the cache mechanism and reduce the computational amount. Due to the characteristic that during the stream processing the data is coming continuously and constantly, the function evaluation value is periodically updated and the connection tree is rebuilt dynamically, in order to ensure the continuous and efficient connection operation. The experimental part uses the distributed Message Queue Kafka to generate multiple data streams, and uses Sparkstreaming as a consumer to connect operations and verify the feasibility of the algorithm.

Copyright

© 2018, the Authors. Published by Atlantis Press.

Open Access

This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Download article (PDF)

TY  - CONF
AU  - Zhiming Chen
AU  - Fengzhen Wang
AU  - Boyang Liu
PY  - 2018/03
DA  - 2018/03
TI  - Multi-link query optimization based on heuristic search in data stream
BT  - Proceedings of the 2018 Joint International Advanced Engineering and Technology Research Conference (JIAET 2018)
PB  - Atlantis Press
SP  - 19
EP  - 23
SN  - 2352-5401
UR  - https://doi.org/10.2991/jiaet-18.2018.4
DO  - 10.2991/jiaet-18.2018.4
ID  - Chen2018/03
ER  -