Construction and Application of Ship Data Mining Platform Based on Spark

DOI

10.2991/msbda-19.2019.42How to use a DOI?

Keywords

Spark, data mining, Automatic Identification System, platform construction, Data preprocessing

Abstract

With the explosive growth of ship information data brought by the intellectualization and digitization of waterborne traffic, it has become more and more important to process and apply massive ship data quickly and accurately. In view of the growing maturity and wide application of Spark large data technology in data mining field, this paper, against the characteristics of ship AIS data, chooses a series of large data technologies based on Spark to build the AIS data mining platform. According to the basic process of mining and processing, the overall framework of the platform is divided into three modules: database, Spark computing and mining, visualization. The AIS data of Sunda Strait with high ship density is used to process and mine the actual data to verify the availability of the platform. The results show that the platform works well and the three modules can work normally. And the AIS data of the Strait are processed and mined quickly and accurately. According to different requirements, it can display bar charts, polyline charts, histogram, trajectory charts and thermal charts and other commonly used data analysis graphics to achieve the effect of navigation assistance.

Copyright

© 2019, the Authors. Published by Atlantis Press.

Open Access

This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Download article (PDF)

TY  - CONF
AU  - Lei Cao
AU  - Jingfeng Hu
AU  - Ran Li
PY  - 2019/08
DA  - 2019/08
TI  - Construction and Application of Ship Data Mining Platform Based on Spark
BT  - Proceedings of the 2019 International Conference on Modeling, Simulation and Big Data Analysis (MSBDA 2019)
PB  - Atlantis Press
SP  - 277
EP  - 282
SN  - 2352-538X
UR  - https://doi.org/10.2991/msbda-19.2019.42
DO  - 10.2991/msbda-19.2019.42
ID  - Cao2019/08
ER  -