Proceedings of the 2022 3rd International Conference on Artificial Intelligence and Education (IC-ICAIE 2022)

<Previous Article In Volume
Next Article In Volume>

Mainstream Big Data Parallel Computing System Performance Optimization

Authors
Yarong Lv1, *
1Information Engineering College, Xi’an Mingde Institute of Technology, Xi’an, 710000, Shaanxi, China
*Corresponding author. Email: lvyr@mdit.edu.cn
Corresponding Author
Yarong Lv
Available Online 27 December 2022.
DOI
10.2991/978-94-6463-040-4_156How to use a DOI?
Keywords
Big Data; Parallel Computing; Parallel Computing System; System Performance Optimization
Abstract

In recent years, with the widespread application of the Internet and information technology in people's production and life, the amount of data generated by all walks of life every day has shown a geometric and explosive growth trend, and the real-time nature of data analysis has become increasingly high. Therefore, big data parallel computing is widely used. The main purpose of this paper is to analyze and research the performance optimization of parallel computing systems based on mainstream big data. This paper mainly analyzes the design requirements and elastic resource scheduling strategy of the big data parallel computing system, introduces the framework and module design and the main functional modules, and preprocesses the data. The experimental results show that as the parallelism of Shuffle increases, the running time of Task decreases. This is because as the degree of parallelism increases, the amount of data processed by a single task decreases and the processing speed becomes faster. However, the proportion of shuffle data read I/O waiting time is the smallest only when the shuffle parallelism is 800, which is optimal.

Copyright
© 2023 The Author(s)
Open Access
Open Access This chapter is licensed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

Download article (PDF)

<Previous Article In Volume
Next Article In Volume>
Volume Title
Proceedings of the 2022 3rd International Conference on Artificial Intelligence and Education (IC-ICAIE 2022)
Series
Atlantis Highlights in Computer Sciences
Publication Date
27 December 2022
ISBN
978-94-6463-040-4
ISSN
2589-4900
DOI
10.2991/978-94-6463-040-4_156How to use a DOI?
Copyright
© 2023 The Author(s)
Open Access
Open Access This chapter is licensed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

Cite this article

risenwbib
TY  - CONF
AU  - Yarong Lv
PY  - 2022
DA  - 2022/12/27
TI  - Mainstream Big Data Parallel Computing System Performance Optimization
BT  - Proceedings of the 2022 3rd International Conference on Artificial Intelligence and Education (IC-ICAIE 2022)
PB  - Atlantis Press
SP  - 1036
EP  - 1042
SN  - 2589-4900
UR  - https://doi.org/10.2991/978-94-6463-040-4_156
DO  - 10.2991/978-94-6463-040-4_156
ID  - Lv2022
ER  -