Network music big data platform storage system construction

DOI

10.2991/wartia-17.2017.46How to use a DOI?

Keywords

Big data platform, Online music, HBase, Distributed storage, Sqoop

Abstract

To build the network music big data platform, We need to collect a lot of music resources from the Internet, And store it on big data platforms. Therefore, it is a key problem to build a storage system that is high performance, scalable and capable of supporting big data. The network music data is very large and the metadata structure is not consistent, HBase and other distributed databases are widely used to build it. However, single RowKey index query in HBase is inefficient, it is difficult to meet the demand of big data platform for real-time or quasi-real-time query and statistical analysis. To this end, this paper proposes a solution, First, we migrated to HBase through Sqoop, which stored the music resources stored in multiple relational databases; Then use Phoenix to establish a secondary index query mechanism, build the Phoenix + HBase storage structure, and store management of the heterogeneous music resources; Finally, the read-write performance and query performance of the storage architecture are tested in the Spark parallel computing framework. Compared with the experimental results, the efficiency of Phoenix + HBase query performance is significantly higher than that of single computer and direct call to HBase, and it can operate unstructured data and structured data, which is in line with the requirement of the storage system of music big data platform.

Copyright

© 2017, the Authors. Published by Atlantis Press.

Open Access

This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Download article (PDF)

TY  - CONF
AU  - Ouyang Tong
AU  - Yi-zhen Cao
PY  - 2017/11
DA  - 2017/11
TI  - Network music big data platform storage system construction
BT  - Proceedings of the 3rd Workshop on Advanced Research and Technology in Industry (WARTIA 2017)
PB  - Atlantis Press
SP  - 229
EP  - 236
SN  - 2352-5401
UR  - https://doi.org/10.2991/wartia-17.2017.46
DO  - 10.2991/wartia-17.2017.46
ID  - Tong2017/11
ER  -