SIBI Dynamic Gesture Translation Using MediaPipe and Long Short-Term Memory in Real-Time

DOI

10.2991/978-94-6463-366-5_6How to use a DOI?

Keywords

sign language recognition; SIBI; LSTM; MediaPipe; real-time

Abstract

Commonly used image processing classification methods like Artificial Neural Network and Convolutional Neural Network are considered successful for sign language identification. However, they perform well only with static data and face limitations in handling sequential and dynamic data like Indonesian Sign Language System (SIBI) sign gestures. To address this, this research uses the Long Short-Term Memory (LSTM) method which has a flexible architecture and can adjust dynamically to accommodate various input sequence lengths, making it reliable in handling sequential data and allowing it to be implemented in real-time systems. This research uses a primary dataset which directly collected by the author, featuring six classes based on question words: “what,” “how,” “how much,” “where,” “why,” and “who.” The 180 original data are augmented into 3060 (510 for each class) with four variations: rotation, zoom in, zoom out, and brightness and contrast adjustments. Data processing utilizes the MediaPipe framework to extract hand landmarks from each data point, saving them as numerical data in NumPy array format. Thus, instead of detecting the entire image susceptible to background noise, detection focuses solely on landmarks indicating hand and finger positions. With a data split of 2616 for training, 153 for testing, and 291 for validation, the model is constructed with three LSTM layers and three Dense layers. This combination yields a categorical accuracy of 99.85%, a loss of 0.0059, validation categorical accuracy of 100%, and validation loss of 0.0064 after 150 training epochs.

Copyright

© 2024 The Author(s)

Open Access

Open Access This chapter is licensed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

Download article (PDF)

TY  - CONF
AU  - Rivano Ardiyan Taufiq Kurniawan
AU  - Wilis Kaswidjanti
PY  - 2024
DA  - 2024/02/02
TI  - SIBI Dynamic Gesture Translation Using MediaPipe and Long Short-Term Memory in Real-Time
BT  - Proceedings of the 2023 1st International Conference on Advanced Informatics and Intelligent Information Systems (ICAI3S 2023)
PB  - Atlantis Press
SP  - 49
EP  - 60
SN  - 1951-6851
UR  - https://doi.org/10.2991/978-94-6463-366-5_6
DO  - 10.2991/978-94-6463-366-5_6
ID  - Kurniawan2024
ER  -