The Improvement of Network Performance by Using the Technique of PF-RING Zero-copy to Optimize Spark Streaming

DOI

10.2991/cnct-16.2017.9How to use a DOI?

Keywords

Zero copy, PF_RING, Spark Streaming, Adaptive

Abstract

Regarding the problem of lower utilization rate of broadband and data throughput in the real-time computations when Spark Streaming receiving and treating a large number of TCP packet, we proposed a technique of effective zero-copy of IO kernel and optimiazation of user protocol stack. By comparing and analyzing the monitoring data, we can determine that the reason of low utilization rate of network under special circumstances is the large expanse of kernel system which induced by the big data flow frame dealing with a large number of network IO interruptions in unit time. We propose that the realization of protocol stack can be stripped from the kernel of operating system by combining PF_RING, this can both reduce the expanse of system management and avoid the additional data movement. The results demonstrate that comparing with the present open-source version of Spark-1.4.0, we can effectively reduce the time delay, decrease the jitter and improved the utilization rate of the Worker nodes by using the technique of PF-RING zero-copy to optimize Spark Streaming. And this technique can also meet the real-time computation's need when receiving and treating a large number of TCP packets.

Copyright

© 2017, the Authors. Published by Atlantis Press.

Open Access

This is an open access article distributed under the CC BY-NC license (http://creativecommons.org/licenses/by-nc/4.0/).

Download article (PDF)

TY  - CONF
AU  - Feng LIU
AU  - Zhan-rong LI
AU  - Yang GAO
AU  - Sheng-si LUO
PY  - 2016/12
DA  - 2016/12
TI  - The Improvement of Network Performance by Using the Technique of PF-RING Zero-copy to Optimize Spark Streaming
BT  - Proceedings of the International Conference on Computer Networks and Communication Technology (CNCT 2016)
PB  - Atlantis Press
SP  - 62
EP  - 68
SN  - 2352-538X
UR  - https://doi.org/10.2991/cnct-16.2017.9
DO  - 10.2991/cnct-16.2017.9
ID  - LIU2016/12
ER  -