Kodály Hand Sign Recognition from Hand Landmarks Using XGBoost
DOI:
https://doi.org/10.56705/ijodas.v7i1.391Keywords:
Angklung, Gesture Recognition, XGBoost, MediaPipe Hands, Kodály Hand SignAbstract
Introduction: Angklung is a traditional Indonesian musical instrument that continues to evolve through digital technology. However, computer vision–based gesture recognition for controlling physical angklung instruments remains limited. This study investigates landmark-based recognition of Kodály hand signs and evaluates its application for real-time angklung interaction. Method: Hand landmarks were extracted using MediaPipe Hands from RGB camera input. Each gesture was represented by 63 normalized numerical features derived from 21 landmarks. The dataset consists of 8,000 images representing eight Kodály gesture classes (Do–Do'). Gesture classification was performed using the Extreme Gradient Boosting (XGBoost) algorithm. Model evaluation applied a subject-independent two-fold scheme using accuracy, precision, recall, F1-score, and confusion matrix analysis. Real-time system trials were conducted under different lighting conditions and capture distances, and TCP communication with an ESP32 controller was evaluated. Results: The model achieved 96.63% accuracy in Fold 1 and 96.40% in Fold 2. Misclassifications were mainly observed between visually similar gestures, particularly La and Mi. Separate real-time system trials showed consistent recognition under bright lighting, while accuracy decreased under dim lighting, especially for Do (90%) and Mi (86.7%). Gesture recognition remained reliable up to approximately 1.5 m. TCP testing over 200 command events recorded 0% failed acknowledgments with a mean round-trip time of 87.36 ms. Conclusion: These indicate that landmark-based Kodály gesture classification using MediaPipe Hands and XGBoost can support real-time angklung interaction under controlled conditions, although improvements are needed for low-light environments and visually similar gestures
Downloads
References
[1] A. R. Wicaksono, J. Subur, and M. Taufiqurrohman, “Design and Development of an Automatic Angklung Robot Based on Microcontroller,” JEEE-U (Journal of Electrical and Electronic Engineering-UMSIDA), vol. 7, no. 2, pp. 107–128, Oct. 2023, doi: 10.21070/jeeeu.v7i2.1669.
[2] I. Nur Hanafi, S. Supriyono, and H. Susanti, “Rancang Bangun Angklung Elektrik dengan Mode Otomatis dan Manual Menggunakan Teknologi Mikrokontroler dan Smartphone,” Infotekmesin, vol. 16, no. 1, pp. 113–119, Jan. 2025, doi: 10.35970/infotekmesin.v16i1.2558.
[3] E. Murpratama, U. Sunarya, and A. Novianti, “ANGKLUNG ROBOT CONTROL SYSTEM BASED ON MICROCONTROLLER,” Jurnal Elektro dan Telekomunikasi Terapan, vol. 6, no. 1, p. 734, Jan. 2020, doi: 10.25124/jett.v6i1.1876.
[4] R. P. Wardana, E. M. Budi, and A. S. Sudarsono, “Development of a Wireless Distributed Real-Time Angklung Robot System,” in 2023 8th International Conference on Instrumentation, Control, and Automation (ICA), IEEE, Aug. 2023, pp. 253–257. doi: 10.1109/ICA58538.2023.10273119.
[5] L. Turchet and P. Casari, “Latency and Reliability Analysis of a 5G-Enabled Internet of Musical Things System,” IEEE Internet Things J., vol. 11, no. 1, pp. 1228–1240, Jan. 2024, doi: 10.1109/JIOT.2023.3288818.
[6] C. Cui, M. S. Sunar, and G. Eg Su, “Deep vision-based real-time hand gesture recognition: a review,” PeerJ Comput. Sci., vol. 11, p. e2921, Jun. 2025, doi: 10.7717/peerj-cs.2921.
[7] H.-H. Li and C.-C. Hsieh, “Dynamic Hand Gesture Recognition Using MediaPipe and Transformer,” in IEEE ICEIB 2025, Basel Switzerland: MDPI, Sep. 2025, p. 22. doi: 10.3390/engproc2025108022.
[8] Y. Meng, H. Jiang, N. Duan, and H. Wen, “Real-Time Hand Gesture Monitoring Model Based on MediaPipe’s Registerable System,” Sensors, vol. 24, no. 19, p. 6262, Sep. 2024, doi: 10.3390/s24196262.
[9] R. S. Anwar, T. Hasanuddin, and S. M. Abdullah, “Sistem Keamanan Pintu Asrama Berbasis Pengenalan Wajah dengan Algoritma Haar Cascade,” Buletin Sistem Informasi dan Teknologi Islam, vol. 3, no. 3, pp. 213–218, Aug. 2022, doi: 10.33096/busiti.v3i3.1197.
[10] M. Oudah, A. Al-Naji, and J. Chahl, “Hand Gesture Recognition Based on Computer Vision: A Review of Techniques,” J. Imaging, vol. 6, no. 8, p. 73, Jul. 2020, doi: 10.3390/jimaging6080073.
[11] M. Z. Fauzi and R. Sarno, “Recognition of Real-Time Angklung Kodály Hand Gesture using Mediapipe and Machine Learning Method,” in ICCoSITE 2023 - International Conference on Computer Science, Information Technology and Engineering: Digital Transformation Strategy in Facing the VUCA and TUNA Era, Institute of Electrical and Electronics Engineers Inc., 2023, pp. 980–985. doi: 10.1109/ICCoSITE57641.2023.10127808.
[12] A. Gülle, C. Akay, and N. B. Uzun, “Zoltán Kodály gives a hand to secondary school students in recorder performance and attitudes toward music in Turkey,” International Journal of Music Education, vol. 39, no. 4, pp. 477–491, Nov. 2021, doi: 10.1177/02557614211005904.
[13] M. Noorunnahar, A. H. Chowdhury, and F. A. Mila, “A tree based eXtreme Gradient Boosting (XGBoost) machine learning model to forecast the annual rice production in Bangladesh,” PLoS One, vol. 18, no. 3, p. e0283452, Mar. 2023, doi: 10.1371/journal.pone.0283452.
[14] M. Imani, A. Beikmohammadi, and H. R. Arabnia, “Comprehensive Analysis of Random Forest and XGBoost Performance with SMOTE, ADASYN, and GNUS Under Varying Imbalance Levels,” Technologies (Basel)., vol. 13, no. 3, Mar. 2025, doi: 10.3390/technologies13030088.
[15] N. Andriyanov and S. Mikhailova, “Improving Gesture Recognition Efficiency with MediaPipe and YOLO-Pose,” The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. XLVIII-2/W9-2025, pp. 13–18, Sep. 2025, doi: 10.5194/isprs-archives-XLVIII-2-W9-2025-13-2025.
[16] T. L. Dang, S. D. Tran, T. H. Nguyen, S. Kim, and N. Monet, “An improved hand gesture recognition system using keypoints and hand bounding boxes,” Array, vol. 16, Dec. 2022, doi: 10.1016/j.array.2022.100251.
[17] C. Bentéjac, A. Csörgő, and G. Martínez-Muñoz, “A comparative analysis of gradient boosting algorithms,” Artif. Intell. Rev., vol. 54, no. 3, pp. 1937–1967, Mar. 2021, doi: 10.1007/s10462-020-09896-5.
[18] S. Bhushan, M. Alshehri, I. Keshta, A. K. Chakraverti, J. Rajpurohit, and A. Abugabah, “An Experimental Analysis of Various Machine Learning Algorithms for Hand Gesture Recognition,” Electronics (Basel)., vol. 11, no. 6, p. 968, Mar. 2022, doi: 10.3390/electronics11060968.
[19] N. Zheng, Y. Li, W. Zhang, and M. Du, “User-Independent EMG Gesture Recognition Method Based on Adaptive Learning,” Front. Neurosci., vol. 16, Mar. 2022, doi: 10.3389/fnins.2022.847180.
[20] P. Zhang, Y. Jia, and Y. Shang, “Research and application of XGBoost in imbalanced data,” Int. J. Distrib. Sens. Netw., vol. 18, no. 6, p. 155013292211069, Jun. 2022, doi: 10.1177/15501329221106935.
[21] P. Sarker, J.-J. Tiang, and A.-A. Nahid, “Metaheuristic-Driven Feature Selection for Human Activity Recognition on KU-HAR Dataset Using XGBoost Classifier,” Sensors, vol. 25, no. 17, p. 5303, Aug. 2025, doi: 10.3390/s25175303.
[22] S. Farhadpour, T. A. Warner, and A. E. Maxwell, “Selecting and Interpreting Multiclass Loss and Accuracy Assessment Metrics for Classifications with Class Imbalance: Guidance and Best Practices,” Remote Sens. (Basel)., vol. 16, no. 3, p. 533, Jan. 2024, doi: 10.3390/rs16030533.
[23] O. Rainio, J. Teuho, and R. Klén, “Evaluation metrics and statistical tests for machine learning,” Sci. Rep., vol. 14, no. 1, p. 6086, Mar. 2024, doi: 10.1038/s41598-024-56706-x.
[24] G. Zeng, “Invariance Properties and Evaluation Metrics Derived from the Confusion Matrix in Multiclass Classification,” Mathematics, vol. 13, no. 16, p. 2609, Aug. 2025, doi: 10.3390/math13162609.
[25] H. Azis, P. Purnawansyah, and N. Alfiyyah, “Multiclass Classification on Nominal Value of Rupiah Banknotes Based on Image Processing,” ILKOM Jurnal Ilmiah, vol. 16, no. 1, pp. 87–99, Apr. 2024, doi: 10.33096/ilkom.v16i1.1784.87-99.
[26] M. A. Mubarak, F. Fattah, and H. Azis, “Prototipe Smart Home Berbasis ESP32 dengan Fitur Keamanan pintu, Lampu, dan AC Otomatis Berbasis IoT,” LINIER: Literatur Informatika dan Komputer, vol. 2, no. 3, pp. 421–437, Oct. 2025, doi: 10.33096/linier.v2i3.3152.
[27] A. Muh. F. Dzikrulkhair, R. Satra, and A. W. Mufila Gafar, “Rancang Bangun Jemuran Pintar Otomatis Berbasis Internet of things (Iot),” LINIER: Literatur Informatika dan Komputer, vol. 2, no. 3, pp. 438–446, Oct. 2025, doi: 10.33096/linier.v2i3.3153.
[28] Y. A. Yunus, R. Satra, and F. Fattah, “Sistem Monitoring Suhu Pada Inkubator Penetas Telur Berbasis IoT,” LINIER: Literatur Informatika dan Komputer, vol. 1, no. 4, pp. 389–394, Dec. 2024, doi: 10.33096/linier.v1i4.2538.
[29] M. Hlayel, H. Mahdin, and H. A. Mohd Adam, “Latency Analysis of WebSocket and Industrial Protocols in Real-Time Digital Twin Integration,” International Journal of Engineering Trends and Technology, vol. 73, no. 1, pp. 120–135, Jan. 2025, doi: 10.14445/22315381/IJETT-V73I1P110.
[30] B. Amirkhanov, G. Amirkhanova, M. Kunelbayev, S. Adilzhanova, and M. Tokhtassyn, “Evaluating HTTP, MQTT over TCP and MQTT over WEBSOCKET for digital twin applications: A comparative analysis on latency, stability, and integration,” International Journal of Innovative Research and Scientific Studies, vol. 8, no. 1, pp. 679–694, Jan. 2025, doi: 10.53894/ijirss.v8i1.4414.
[31] X. Chen, H. Kim, J. M. Aman, W. Chang, M. Lee, and J. Rexford, “Measuring TCP Round-Trip Time in the Data Plane,” in Proceedings of the Workshop on Secure Programmable Network Infrastructure, New York, NY, USA: ACM, Aug. 2020, pp. 35–41. doi: 10.1145/3405669.3405823.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Achmad Zulfikar, Farniwati Fattah, Andi Widya Mufila Gaffar
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
Authors retain copyright and full publishing rights to their articles. Upon acceptance, authors grant Indonesian Journal of Data and Science a non-exclusive license to publish the work and to identify itself as the original publisher.
Self-archiving. Authors may deposit the submitted version, accepted manuscript, and version of record in institutional or subject repositories, with citation to the published article and a link to the version of record on the journal website.
Commercial permissions. Uses intended for commercial advantage or monetary compensation are not permitted under CC BY-NC 4.0. For permissions, contact the editorial office at ijodas.journal@gmail.com.
Legacy notice. Some earlier PDFs may display “Copyright © [Journal Name]” or only a CC BY-NC logo without the full license text. To ensure clarity, the authors maintain copyright, and all articles are distributed under CC BY-NC 4.0. Where any discrepancy exists, this policy and the article landing-page license statement prevail.
