Investigation of Influence Related to Multi-GPUs and Hyperparameters on the ViT Model Based on PyTorch Data Parallelism

DOI

10.2991/978-94-6463-300-9_65How to use a DOI?

Keywords

GPU parallel algorithms; Deep learning; Data parallelism; Vision Transformer; PyTorch

Abstract

Neural networks, which are highly effective models for addressing various real-world challenges, face the challenge of excessive computational requirements caused by the substantial size of the models and the extensive datasets employed during training. In this paper, the purpose is to examine the effects of hyperparameters and factors such as the number and model of Graphics Processing Units (GPUs) on training time and accuracy using the Vision Transformer (ViT) model as a case study within parallel algorithms. The experiments conducted in this study employed PyTorch as the framework and CIFAR-10 as the dataset. The learning rate chosen for the experiments was set to 0.001, while two batch sizes, namely 64 and 32, were explored. Additionally, the study investigated different epoch values, including 1, 5, 10, and 20. To handle the computational workload effectively during training, data parallelism was employed as an approach. In the realm of optimizing the performance of parallel algorithms, numerous methods have been proposed to enhance their efficiency. Consequently, this research places particular emphasis on hyperparameter tuning. The primary objectives of this paper are to examine the impact of GPUs on both training time and model accuracy, as well as to analyze the influence of hyperparameter configuration on the accuracy of the model.

Copyright

© 2023 The Author(s)

Open Access

Open Access This chapter is licensed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

Download article (PDF)

TY  - CONF
AU  - Yibo Feng
AU  - Haoran Zheng
PY  - 2023
DA  - 2023/11/27
TI  - Investigation of Influence Related to Multi-GPUs and Hyperparameters on the ViT Model Based on PyTorch Data Parallelism
BT  - Proceedings of the 2023 International Conference on Image, Algorithms and Artificial Intelligence (ICIAAI 2023)
PB  - Atlantis Press
SP  - 629
EP  - 637
SN  - 2352-538X
UR  - https://doi.org/10.2991/978-94-6463-300-9_65
DO  - 10.2991/978-94-6463-300-9_65
ID  - Feng2023
ER  -