Implementation of Hand Gesture Recognition as Smart Home Devices Controller
DOI:
https://doi.org/10.52985/insyst.v6i2.372Keywords:
Hand Gesture Recognition, Hand Landmark, Mediapipe, Smart HomeAbstract
Some current virtual assistant products such as Alexa, Siri and Google Home facilitate features to control smart home devices through voice input, which has become increasingly popular in recent years. In addition to voice input, smart home devices can also be monitored and controlled through smartphones or computers using applications that provide users with flexibility. However, both control systems are less efficient, as they consume time and voice input utilization may sometimes not be recognized in crowded conditions. Therefore, this research introduces an application to recognize real-time hand gestures and utilize them for a new control system that provides time and energy efficiency. This application processes images using the Mediapipe framework, generating hand landmark outputs. These landmark outputs are utilized to determine the combination of raised or lowered fingers, which is then used to control smart home devices. The application is developed with ESP32 and ESP01s modules as data receivers from gesture recognition, and ESP32-CAM for image acquisition. Meanwhile, the gesture recognition computation process is executed on a Raspberry Pi 3 Model B. The gesture recognition application achieves good accuracy at 88%, but may experience occasional failures for certain gestures. However, the response time generated by the smart home control system is still relatively long, averaging 7.88 seconds.
References
P. Mtshali and F. Khubisa, “A Smart Home Appliance Control System for Physically Disabled People,” in 2019 Conference on Information Communications Technology and Society (ICTAS), IEEE, Mar. 2019, pp. 1–5. doi: 10.1109/ICTAS.2019.8703637.
A. Mujahid et al., “Real-Time Hand Gesture Recognition Based on Deep Learning YOLOv3 Model,” Applied Sciences, vol. 11, no. 9, p. 4164, May 2021, doi: 10.3390/app11094164.
V. Këpuska and G. Bohouta, “Next-generation of virtual personal assistants (Microsoft Cortana, Apple Siri, Amazon Alexa and Google Home),” in 2018 IEEE 8th Annual Computing and Communication Workshop and Conference (CCWC), 2018, pp. 99–103. doi: 10.1109/CCWC.2018.8301638.
S. P. Yadav, A. Gupta, C. Dos Santos Nascimento, V. de Albuquerque, M. S. Naruka, and S. Singh Chauhan, “Voice-Based Virtual-Controlled Intelligent Personal Assistants,” in 2023 International Conference on Computational Intelligence, Communication Technology and Networking (CICTN), 2023, pp. 563–568. doi: 10.1109/CICTN57981.2023.10141447.
Z. Wu, N. Evans, T. Kinnunen, J. Yamagishi, F. Alegre, and H. Li, “Spoofing and countermeasures for speaker verification: A survey,” Speech Commun, vol. 66, pp. 130–153, 2015, doi: https://doi.org/10.1016/j.specom.2014.10.005.
N. R. Adhinugroho and H. P. Uranus, Perancangan Rumah Cerdas sebagai Aplikasi IoT Berbasis Voice Recognition dan Arduino [Voice Recognition and Arduino Based Smart Home Design as Application of IoT]. 2019.
H. Basanta, Y.-P. Huang, and T.-T. Lee, “Assistive design for elderly living ambient using voice and gesture recognition system,” in 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC), IEEE, Oct. 2017, pp. 840–845. doi: 10.1109/SMC.2017.8122714.
T.-S. Dinh Dong-Luong and Kim, “Smart Home Appliance Control via Hand Gesture Recognition Using a Depth Camera,” in Smart Energy Control Systems for Sustainable Buildings, C. and H. R. J. and J. L. C. Littlewood John and Spataru, Ed., Cham: Springer International Publishing, 2017, pp. 159–172.
R.-J. Wang, S.-C. Lai, J.-Y. Jhuang, M.-C. Ho, and Y.-C. Shiau, “Development of Smart Home Gesture-based Control System,” Sensors and Materials, vol. 33, no. 10, p. 3459, Oct. 2021, doi: 10.18494/SAM.2021.3522.
J. Dai, “Gesture Recognition Based Smart Home Control System,” 2020.
P. Vogiatzidakis and P. Koutsabasis, “Mid-Air Gesture Control of Multiple Home Devices in Spatial Augmented Reality Prototype,” Multimodal Technologies and Interaction, vol. 4, p. 61, Aug. 2020, doi: 10.3390/mti4030061.
A. S. Bankar, A. D. Harale, and K. J. Karande, “Gestures Controlled Home Automation using Deep Learning: A Review,” International Journal of Current Engineering and Technology, vol. 11, no. 06, pp. 617–621, Dec. 2021, doi: 10.14741/ijcet/v.11.6.4.
F. Zhang et al., “MediaPipe Hands: On-device Real-time Hand Tracking,” CoRR, vol. abs/2006.1, 2020, [Online]. Available: https://arxiv.org/abs/2006.10214
B. P. Pratiwi, A. S. Handayani, and S. Sarjana, “Pengukuran Kinerja Sistem Kualitas Udara dengan Teknologi WSN menggunakan Confusion Matrix,” Jurnal Informatika Upgris, vol. 6, no. 2, Jan. 2021, doi: 10.26877/jiu.v6i2.6552.
R. B. Widodo, “Confusion matrix,” in Metode k-Nearest Neighbors Klasifikasi Angka Bahasa Isyarat, Malang: Media Nusa Creative, 2022, ch. 3, pp. 21–23.
Downloads
Additional Files
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 INSYST: Journal of Intelligent System and Computation
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
