Integration of Yolov8 And Instance Segmentation in The Chinese Sign Language (CSL) Recognition System
DOI:
https://doi.org/10.56705/ijodas.v6i2.247Keywords:
Chinese Sign Language, YOLOv8, Instance Segmentation, Hand Gesture Recognition, Deep LearningAbstract
This research aims to develop an advanced recognition system for Chinese Sign Language (CSL) by integrating YOLOv8 and instance segmentation techniques. Communication through sign language is essential for the deaf community, and although CSL has been standardized in China, recognizing complex hand movements remains a significant challenge. YOLOv8 is employed for real-time object detection, while instance segmentation is used to provide more detailed analysis of hand gestures. This integration seeks to improve hand gesture recognition under varying lighting and background conditions, which is crucial for more effective communication between the deaf community and the wider society. The study evaluates the system’s performance using common metrics such as Mean Average Precision (mAP), precision, recall, and F1-score. The findings indicate that the non-segmentation model performs better than the segmentation model in terms of precision, recall, and mAP, especially when trained with a larger dataset ratio. The non-segmentation model provides faster and more accurate detection, while the segmentation model, despite using the same amount of data, shows potential for more detailed recognition of gestures. Although the segmentation model shows improvements in the F1-score with more detailed accuracy, the non-segmentation model remains superior in overall detection speed and accuracy. This research highlights the importance of integrating YOLOv8 and instance segmentation for improving CSL recognition, with better results on the non-segmentation model for more effective communication for the deaf
Downloads
References
K. Emmorey, “Ten Things You Should Know About Sign Languages,” Curr Dir Psychol Sci, vol. 32, no. 5, pp. 387–394, Oct. 2023, doi: 10.1177/09637214231173071.
A. N. Handayani, M. I. Akbar, H. Ar-Rosyid, M. Ilham, R. A. Asmara, and O. Fukuda, “Design of SIBI Sign Language Recognition Using Artificial Neural Network Backpropagation,” in 2022 2nd International Conference on Intelligent Cybernetics Technology & Applications (ICICyTA), IEEE, Dec. 2022, pp. 192–197. doi: 10.1109/ICICyTA57421.2022.10038205.
T. N. Fitria, “THE USE OF SIGN LANGUAGE AS A MEDIA FOR DELIVERING INFORMATION ON NATIONAL TELEVISION NEWS BROADCASTS,” ELP (Journal of English Language Pedagogy), vol. 9, no. 1, pp. 118–131, Jan. 2024, doi: 10.36665/elp.v9i1.764.
C. Jo, “Overview of Chinese Sign Language,” Chinese Language and Literature, Dec. 2023, doi: 10.57237/j.cll.2023.05.002.
X. Jiang, S. C. Satapathy, L. Yang, S.-H. Wang, and Y.-D. Zhang, “A Survey on Artificial Intelligence in Chinese Sign Language Recognition,” Arab J Sci Eng, vol. 45, no. 12, pp. 9859–9894, Dec. 2020, doi: 10.1007/s13369-020-04758-2.
H. Wang, Y. Chen, Z. Yang, L. Zhu, Y. Zhao, and T. Tian, “Estimation and projection of the burden of hearing loss in China: findings from the Global Burden of Disease Study 2019,” Public Health, vol. 228, pp. 119–127, Mar. 2024, doi: 10.1016/j.puhe.2024.01.004.
X. Chen, L. Su, J. Zhao, K. Qiu, N. Jiang, and G. Zhai, “Sign Language Gesture Recognition and Classification Based on Event Camera with Spiking Neural Networks,” Electronics (Basel), vol. 12, no. 4, p. 786, Feb. 2023, doi: 10.3390/electronics12040786.
D. Wiryany, S. Natasha, R. Kurniawan, J. I. Komunikasi, and M. Bandung, “PERKEMBANGAN TEKNOLOGI INFORMASI DAN KOMUNIKASI TERHADAP PERUBAHAN SISTEM KOMUNIKASI INDONESIA,” 2022.
- Ridwang, A. A. Ilham, I. Nurtanio, and - Syafaruddin, “Dynamic Sign Language Recognition Using Mediapipe Library and Modified LSTM Method,” Int J Adv Sci Eng Inf Technol, vol. 13, no. 6, pp. 2171–2180, Dec. 2023, doi: 10.18517/ijaseit.13.6.19401.
S. C. Mesbahi, M. A. Mahraz, J. Riffi, and H. Tairi, “Hand Gesture Recognition Based on Various Deep Learning YOLO Models,” International Journal of Advanced Computer Science and Applications, vol. 14, no. 4, 2023, doi: 10.14569/IJACSA.2023.0140435.
Mr. J. Rajasekhar, “UNDERSTANDING YOLO: REAL-TIME OBJECT DETECTION EXPLAINED,” INTERANTIONAL JOURNAL OF SCIENTIFIC RESEARCH IN ENGINEERING AND MANAGEMENT, vol. 08, no. 07, pp. 1–9, Jul. 2024, doi: 10.55041/IJSREM36359.
S. Tyagi, P. Upadhyay, H. Fatima, S. Jain, I. Avinash, and K. Sharma, “American Sign Language Detection using YOLOv5 and YOLOv8,” 2023, doi: 10.21203/rs.3.rs-3126918/v1.
S. Daniels, N. Suciati, and C. Fathichah, “Indonesian Sign Language Recognition using YOLO Method,” IOP Conf Ser Mater Sci Eng, vol. 1077, no. 1, p. 012029, Feb. 2021, doi: 10.1088/1757-899X/1077/1/012029.
Z. Q. Zhao, P. Zheng, S. T. Xu, and X. Wu, “Object Detection with Deep Learning: A Review,” Nov. 01, 2019, Institute of Electrical and Electronics Engineers Inc. doi: 10.1109/TNNLS.2018.2876865.
J. C. Caicedo et al., “Data-analysis strategies for image-based cell profiling,” Nat Methods, vol. 14, no. 9, pp. 849–863, Sep. 2017, doi: 10.1038/nmeth.4397.
K. Maharana, S. Mondal, and B. Nemade, “A review: Data pre-processing and data augmentation techniques,” Global Transitions Proceedings, vol. 3, no. 1, pp. 91–99, Jun. 2022, doi: 10.1016/j.gltp.2022.04.020.
M. Guo et al., “Normal Workflow and Key Strategies for Data Cleaning Toward Real-World Data: Viewpoint,” Interact J Med Res, vol. 12, p. e44310, Sep. 2023, doi: 10.2196/44310.
A. N. Handayani, S. Amaliya, M. I. Akbar, M. Z. Wiryawan, Y. W. Liang, and W. C. Kurniawan, “Hand Keypoint-Based CNN for SIBI Sign Language Recognition,” International Journal of Robotics and Control Systems, vol. 5, no. 2, pp. 813–829, 2025, doi: 10.31763/ijrcs.v5i2.1745.
A. N. Handayani, M. I. Akbar, H. A. Rosyid, M. Arifianto, O. Fukuda, and R. A. Asmara, “K-NN for Hand Keypoints Detection Based on SIBI Standard,” in 2024 International Conference on Electrical and Information Technology (IEIT), IEEE, Sep. 2024, pp. 72–77. doi: 10.1109/IEIT64341.2024.10763365.
A. Onan, “SRL-ACO: A text augmentation framework based on semantic role labeling and ant colony optimization,” Journal of King Saud University - Computer and Information Sciences, vol. 35, no. 7, p. 101611, Jul. 2023, doi: 10.1016/j.jksuci.2023.101611.
R. Wulanningrum, A. N. Handayani, and A. P. Wibawa, “Perbandingan Instance Segmentation Image Pada Yolo8,” Jurnal Teknologi Informasi dan Ilmu Komputer, vol. 11, no. 4, pp. 753–760, Aug. 2024, doi: 10.25126/jtiik.1148288.
G. Li, L. Zhou, Q. Tong, Y. Ding, X. Qi, and H. Liu, “A Data Augmentation Approach to Sentiment Analysis of MOOC Reviews,” International Journal of Advanced Computer Science and Applications, vol. 15, no. 8, 2024, doi: 10.14569/IJACSA.2024.01508122.
A. N. Handayani, T. Andriyanto, D. F. Azizah, M. Z. Wiryawan, and H. A. Rosyid, “Comparison of ResNet-50 and EfficientNet-B0 Method for Classification of Indonesian Sign Language System (SIBI) Using Multi Background Dataset,” in 2024 International Conference on Computer Engineering, Network, and Intelligent Multimedia (CENIM), IEEE, Nov. 2024, pp. 1–6. doi: 10.1109/CENIM64038.2024.10882836.
V. R. Joseph, “Optimal ratio for data splitting,” Statistical Analysis and Data Mining: The ASA Data Science Journal, vol. 15, no. 4, pp. 531–538, Aug. 2022, doi: 10.1002/sam.11583.
Q. H. Nguyen et al., “Influence of Data Splitting on Performance of Machine Learning Models in Prediction of Shear Strength of Soil,” Math Probl Eng, vol. 2021, pp. 1–15, Feb. 2021, doi: 10.1155/2021/4832864.
A. Nurhopipah and U. Hasanah, “Dataset Splitting Techniques Comparison For Face Classification on CCTV Images,” IJCCS (Indonesian Journal of Computing and Cybernetics Systems), vol. 14, no. 4, p. 341, Oct. 2020, doi: 10.22146/ijccs.58092.
C.-J. Zhang et al., “Evaluation of the YOLO models for discrimination of the alfalfa pollinating bee species,” J Asia Pac Entomol, vol. 27, no. 1, p. 102195, Mar. 2024, doi: 10.1016/j.aspen.2023.102195.
J. Chen, C. Ji, J. Zhang, Q. Feng, Y. Li, and B. Ma, “A method for multi-target segmentation of bud-stage apple trees based on improved YOLOv8,” Comput Electron Agric, vol. 220, p. 108876, May 2024, doi: 10.1016/j.compag.2024.108876.
E. Lee, B. Park, M.-H. Jeon, H. Jang, A. Kim, and S. Lee, “Data augmentation using image translation for underwater sonar image segmentation,” PLoS One, vol. 17, no. 8, p. e0272602, Aug. 2022, doi: 10.1371/journal.pone.0272602.
Downloads
Published
Issue
Section
License
Authors retain copyright and full publishing rights to their articles. Upon acceptance, authors grant Indonesian Journal of Data and Science a non-exclusive license to publish the work and to identify itself as the original publisher.
Self-archiving. Authors may deposit the submitted version, accepted manuscript, and version of record in institutional or subject repositories, with citation to the published article and a link to the version of record on the journal website.
Commercial permissions. Uses intended for commercial advantage or monetary compensation are not permitted under CC BY-NC 4.0. For permissions, contact the editorial office at ijodas.journal@gmail.com.
Legacy notice. Some earlier PDFs may display “Copyright © [Journal Name]” or only a CC BY-NC logo without the full license text. To ensure clarity, the authors maintain copyright, and all articles are distributed under CC BY-NC 4.0. Where any discrepancy exists, this policy and the article landing-page license statement prevail.
