Hand Sign Virtual Reality Data Processing Using Padding Technique
DOI:
https://doi.org/10.52985/insyst.v6i2.395Keywords:
Recurrent Neural Networks (RNNs), Sequential Data, Signal ProcessingAbstract
This study focuses on addressing the challenges of processing hand sign data in Virtual Reality environments, particularly the variability in data length during gesture recording. To optimize machine learning models for gesture recognition, various padding techniques were implemented. The data was gathered using the Meta Quest 2 device, consisting of 1,000 samples representing 10 American Sign Language hand sign movements. The research applied different padding techniques, including pre- and post-zero padding as well as replication padding, to standardize sequence lengths. Long Short-Term Memory networks were utilized for modeling, with the data split into 80% for training and 20% for validation. An additional 100 unseen samples were used for testing. Among the techniques, pre-replication padding produced the best results in terms of accuracy, precision, recall, and F1 score on the test dataset. Both pre- and post-zero padding also demonstrated strong performance but were outperformed by replication padding. This study highlights the importance of padding techniques in optimizing the accuracy and generalizability of machine learning models for hand sign recognition in Virtual Reality. The findings offer valuable insights for developing more robust and efficient gesture recognition systems in interactive Virtual Reality environments, enhancing user experiences and system reliability. Future work could explore extending these techniques to other Virtual Reality interactions.
References
S. D. Sulistyaningrum, R. S. Putri, A. Herawati, and S. Irianto, “Trends of virtual reality for learning empirical evidence from different fields,” Journal of Education and Learning (EduLearn), vol. 16, no. 4, pp. 531–541, Nov. 2022, doi: 10.11591/edulearn.v16i4.20648.
L. Muñoz-Saavedra, L. Miró-Amarante, and M. Domínguez-Morales, “Augmented and virtual reality evolution and future tendency,” Applied Sciences (Switzerland), vol. 10, no. 1, 2020, doi: 10.3390/app10010322.
M. Alizadeh and N. Cowie, “Self-directed learning using VR,” Pacific Journal of Technology Enhanced Learning, vol. 4, no. 1, pp. 10–11, Feb. 2022, doi: 10.24135/pjtel.v4i1.130.
H. R. Spero, I. Vazquez-Lopez, K. Miller, R. Joshaghani, S. Cutchin, and J. Enterkine, “Drones, virtual reality, and modeling: communicating catastrophic dam failure,” Int J Digit Earth, vol. 15, no. 1, pp. 585–605, Dec. 2022, doi: 10.1080/17538947.2022.2041116.
N. Albertini, J. Baldini, A. Dal Pino, F. Lazzari, S. Legnaioli, and V. Barone, “PROTEUS: an immersive tool for exploring the world of cultural heritage across space and time scales,” Herit Sci, vol. 10, no. 1, 2022, doi: 10.1186/s40494-022-00708-3.
G. E. Lysenko, A. V. Shchegolev, B. N. Bogomolov, and D. P. Meshakov, “Virtual reality therapy in patients after trauma surgery: prospective clinical trial,” Annals of Critical Care, vol. 2023, no. 3, 2023, doi: 10.21320/1818-474X-2023-3-58-65.
C. P. Kamm, R. Kueng, and R. Blättler, “Development of a new immersive virtual reality (VR) headset-based dexterity training for patients with multiple sclerosis: Clinical and technical aspects,” Technology and Health Care, pp. 1–12, Aug. 2023, doi: 10.3233/THC-230541.
C. P. Kamm, R. Blättler, R. Kueng, and T. Vanbellingen, “Feasibility and usability of a new home-based immersive virtual reality headset-based dexterity training in multiple sclerosis,” Mult Scler Relat Disord, vol. 71, p. 104525, Mar. 2023, doi: 10.1016/j.msard.2023.104525.
Z. Jiang, A. Meltzer, and X. Zhang, “Using virtual reality to implement disability studies’ advocacy principles: uncovering the perspectives of people with disability,” Disabil Soc, pp. 1–21, Jan. 2023, doi: 10.1080/09687599.2022.2150601.
A. Tariq, T. Rana, and M. Nawaz, “Virtual Reality for Disabled People: A Survey,” in 2018 12th International Conference on Open Source Systems and Technologies (ICOSST), IEEE, Dec. 2018, pp. 17–21. doi: 10.1109/ICOSST.2018.8632182.
N. R. Rahim and A. A. A. Rahim, “A VIRTUAL REALITY APPROACH TO SUPPORT MALAYSIAN SIGN LANGUAGE INTERACTIVE LEARNING FOR DEAF-MUTE CHILDREN,” J Pharm Negat Results, vol. 13, 2022.
K. Sangeethalakshmi, K. G. Shanthi, A. Mohan Raj, S. Muthuselvan, P. Mohammad Taha, and S. Mohammed Shoaib, “Hand gesture vocalizer for deaf and dumb people,” Mater Today Proc, vol. 80, 2023, doi: 10.1016/j.matpr.2021.07.324.
A. Vaitkevičius, M. Taroza, T. Blažauskas, R. Damaševičius, R. Maskeliunas, and M. Woźniak, “Recognition of American Sign Language gestures in a Virtual Reality using Leap Motion,” Applied Sciences (Switzerland), vol. 9, no. 3, 2019, doi: 10.3390/app9030445.
A. Shaw et al., “JengASL: A Gamified Approach to Sign Language Learning in VR,” J WSCG (Plzen), vol. 31, no. 1–2, 2023, doi: 10.24132/JWSCG.2023.4.
G. Park and W. Woo, “Hybrid 3D Hand Articulations Tracking Guided by Classification and Search Space Adaptation,” in Proceedings of the 2018 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2018, 2018. doi: 10.1109/ISMAR.2018.00029.
F. Wen, Z. Zhang, T. He, and C. Lee, “AI enabled sign language recognition and VR space bidirectional communication using triboelectric smart glove,” Nat Commun, vol. 12, no. 1, 2021, doi: 10.1038/s41467-021-25637-w.
N. Takayama, G. Benitez-Garcia, and H. Takahashi, “Masked Batch Normalization to Improve Tracking-Based Sign Language Recognition Using Graph Convolutional Networks,” in Proceedings - 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition, FG 2021, 2021. doi: 10.1109/FG52635.2021.9667007.
F. Reggiswarashari and S. W. Sihwi, “Speech emotion recognition using 2D-convolutional neural network,” International Journal of Electrical and Computer Engineering, vol. 12, no. 6, 2022, doi: 10.11591/ijece.v12i6.pp6594-6601.
F. Alrasheedi, X. Zhong, and P. C. Huang, “Padding Module: Learning the Padding in Deep Neural Networks,” IEEE Access, vol. 11, 2023, doi: 10.1109/ACCESS.2023.3238315.
X. Hu, X. Wei, Y. Gao, W. Zhuang, M. Chen, and H. Lv, “An attention-mechanism-based traffic flow prediction scheme for smart city,” in 2019 15th International Wireless Communications and Mobile Computing Conference, IWCMC 2019, 2019. doi: 10.1109/IWCMC.2019.8766639.
T. E. E. Tju and M. U. Shalih, “Hand Sign Interpretation through Virtual Reality Data Processing,” Jurnal Ilmu Komputer dan Informasi, vol. 17, no. 2, pp. 185–194, Jun. 2024, doi: 10.21609/jiki.v17i2.1280.
“Meta Quest 2: Our Most Advanced New All-in-One VR Headset | Oculus | Meta Store.” Accessed: Aug. 04, 2024. [Online]. Available: https://www.meta.com/quest/products/quest-2/tech-specs/#tech-specs
“Top 10 & 25 American Sign Language Signs for Beginners – The Most Known Top 10 & 25 ASL Signs to Learn First | Start ASL.” Accessed: Aug. 04, 2024. [Online]. Available: https://www.startasl.com/top-10-25-american-sign-language-signs-for-beginners-the-most-know-top-10-25-asl-signs-to-learn-first/
“‘you’ ASL American Sign Language.” Accessed: Aug. 04, 2024. [Online]. Available: https://www.lifeprint.com/asl101/pages-signs/y/you.htm
“‘help’ American Sign Language (ASL).” Accessed: Aug. 04, 2024. [Online]. Available: https://www.lifeprint.com/asl101/pages-signs/h/help.htm
“‘me’ American Sign Language (ASL).” Accessed: Aug. 04, 2024. [Online]. Available: https://www.lifeprint.com/asl101/pages-signs/m/me.htm
“Meta Quest 2: Immersive All-In-One VR Headset | Meta Store | Meta Store.” Accessed: Aug. 04, 2024. [Online]. Available: https://www.meta.com/quest/products/quest-2/#overview
“Unity - Manual: Unity User Manual 2022.3 (LTS).” Accessed: Aug. 04, 2024. [Online]. Available: https://docs.unity3d.com/Manual/index.html
“Unity - Scripting API: CommonUsages.” Accessed: Aug. 04, 2024. [Online]. Available: https://docs.unity3d.com/ScriptReference/XR.CommonUsages.html
“[1808.03314] Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network.” Accessed: Aug. 11, 2024. [Online]. Available: https://arxiv.org/abs/1808.03314
A. Sherstinsky, “Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) network,” Physica D, vol. 404, p. 132306, Mar. 2020, doi: 10.1016/j.physd.2019.132306.
M. K. Suryadewiansyah and T. E. E. Tju, “Naïve Bayes dan Confusion Matrix untuk Efisiensi Analisa Intrusion Detection System Alert,” Jurnal Nasional Teknologi dan Sistem Informasi, vol. 8, no. 2, pp. 81–88, 2022, doi: 10.25077/teknosi.v8i2.2022.81-88.
Downloads
Additional Files
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 INSYST: Journal of Intelligent System and Computation
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
