Leakage-Aware BISINDO Alphabet Classification with Hand-Region DINOv2 Embeddings

Lilis Nur Hayati; Dolly Indra

doi:10.56705/ijodas.v7i2.528

Authors

Lilis Nur Hayati Universitas Muslim Indonesia
Dolly Indra Universitas Muslim Indonesia

DOI:

https://doi.org/10.56705/ijodas.v7i2.528

Keywords:

BISINDO, Sign-Language Recognition, DINOv2, Hand-Region Classification, Group-Disjoint Evaluation, Illumination Robustness

Abstract

Introduction: Visual similarity across images can bias the evaluation of Indonesian Sign Language (BISINDO) alphabet classifiers. This study assessed frozen DINOv2 embeddings and illumination correction using a group-disjoint evaluation protocol. Method: The dataset comprised 19,748 annotated gesture regions from 19,746 images across 26 classes. Full-frame visual-similarity groups were assigned to training, validation, and test partitions containing 14,107, 2,821, and 2,820 instances, respectively. Frozen DINOv2-small embeddings with class-balanced multinomial logistic regression were evaluated under raw, fixed-gamma, and adaptive-gamma conditions against a grayscale reference using principal component analysis and logistic regression. Configuration selection used validation macro-F1, while uncertainty was estimated using 2,000 group-bootstrap resamples. Results and Discussion: The validation-selected raw configuration achieved test accuracy of 90.28% and macro-F1 of 0.8982 (95% CI, 0.8832–0.9130), compared with macro-F1 of 0.4383 for the grayscale reference. Adaptive gamma achieved macro-F1 of 0.9038, but its advantage over raw inputs was small and uncertain (difference, 0.0056; 95% paired CI, −0.0048 to 0.0152). Exploratory brightness tests indicated condition-specific responses to synthetic illumination changes. Conclusion: Frozen DINOv2 embeddings supported static BISINDO gesture-region classification within an auditable evaluation framework. Adaptive gamma did not demonstrate a consistent improvement, and the findings remain limited to annotation-defined regions under the documented grouping protocol.

Downloads

Download data is not yet available.

References

[1] L. N. Hayati, A. N. Handayani, W. S. G. Irianto, R. A. Asmara, and D. Indra, “A Multimodal BISINDO Corpus: Annotated Images and Videos for Indonesian Sign Language Hand Gesture Recognition,” Mendeley Data, ver. 2, 2025. https://doi.org/10.17632/235c78xbmk.2.

[2] L. N. Hayati, A. N. Handayani, W. S. G. Irianto, R. A. Asmara, D. Indra, and N. S. Damanhuri, “Improving Indonesian Sign Alphabet Recognition for Assistive Learning Robots Using Gamma-Corrected MobileNetV2,” Buletin Ilmiah Sarjana Teknik Elektro, vol. 7, no. 3, pp. 350-361, 2025. https://doi.org/10.12928/biste.v7i3.13300.

[3] L. N. Hayati, A. N. Handayani, W. S. G. Irianto, R. A. Asmara, D. Indra, and N. S. Damanhuri, “Real-Time BISINDO Alphabet Recognition via Faster R-CNN Incorporating Skin Tone Diversity as a Classification Feature,” Buletin Ilmiah Sarjana Teknik Elektro, vol. 8, no. 3, pp. 811-823, 2026. https://doi.org/10.12928/biste.v8i3.15587.

[4] S. Dwijayanti, Hermawati, S. I. Taqiyyah, H. Hikmarika, and B. Y. Suprapto, “Indonesia Sign Language Recognition Using Convolutional Neural Network,” International Journal of Advanced Computer Science and Applications, vol. 12, no. 10, pp. 415-422, 2021. https://doi.org/10.14569/IJACSA.2021.0121046.

[5] S. Suharjito, N. Thiracitta, and H. Gunawan, “SIBI Sign Language Recognition Using Convolutional Neural Network Combined with Transfer Learning and Non-Trainable Parameters,” Procedia Computer Science, vol. 179, pp. 72-80, 2021. https://doi.org/10.1016/j.procs.2020.12.011.

[6] I. A. Adeyanju, O. O. Bello, and M. A. Adegboye, “Machine Learning Methods for Sign Language Recognition: A Critical Review and Analysis,” Intelligent Systems with Applications, vol. 12, Art. no. 200056, 2021. https://doi.org/10.1016/j.iswa.2021.200056.

[7] M. Oquab et al., “DINOv2: Learning Robust Visual Features Without Supervision,” Transactions on Machine Learning Research, 2024. [Online]. Available: https://openreview.net/forum?id=a68SUt6zFt.

[8] M. Caron et al., “Emerging Properties in Self-Supervised Vision Transformers,” in Proc. IEEE/CVF International Conference on Computer Vision, 2021, pp. 9650-9660. https://doi.org/10.1109/ICCV48922.2021.00968.

[9] A. Dosovitskiy et al., “An Image Is Worth 16×16 Words: Transformers for Image Recognition at Scale,” in Proc. International Conference on Learning Representations, 2021. [Online]. Available: https://openreview.net/forum?id=YicbFdNTTy.

[10] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A Simple Framework for Contrastive Learning of Visual Representations,” in Proc. 37th International Conference on Machine Learning, 2020, pp. 1597-1607. [Online]. Available: https://proceedings.mlr.press/v119/chen20j.html.

[11] S. Rahman, M. M. Rahman, M. Abdullah-Al-Wadud, G. D. Al-Quaderi, and M. Shoyaib, “An Adaptive Gamma Correction for Image Enhancement,” EURASIP Journal on Image and Video Processing, vol. 2016, Art. no. 35, 2016. https://doi.org/10.1186/s13640-016-0138-1.

[12] C. Shorten and T. M. Khoshgoftaar, “A Survey on Image Data Augmentation for Deep Learning,” Journal of Big Data, vol. 6, Art. no. 60, 2019. https://doi.org/10.1186/s40537-019-0197-0.

[13] J. M. Johnson and T. M. Khoshgoftaar, “Survey on Deep Learning with Class Imbalance,” Journal of Big Data, vol. 6, Art. no. 27, 2019. https://doi.org/10.1186/s40537-019-0192-5.

[14] M. Buda, A. Maki, and M. A. Mazurowski, “A Systematic Study of the Class Imbalance Problem in Convolutional Neural Networks,” Neural Networks, vol. 106, pp. 249-259, 2018. https://doi.org/10.1016/j.neunet.2018.07.011.

[15] M. Sokolova and G. Lapalme, “A Systematic Analysis of Performance Measures for Classification Tasks,” Information Processing & Management, vol. 45, no. 4, pp. 427-437, 2009. https://doi.org/10.1016/j.ipm.2009.03.002.

[16] K. H. Brodersen, C. S. Ong, K. E. Stephan, and J. M. Buhmann, “The Balanced Accuracy and Its Posterior Distribution,” in Proc. 20th International Conference on Pattern Recognition, 2010, pp. 3121-3124. https://doi.org/10.1109/ICPR.2010.764.

[17] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On Calibration of Modern Neural Networks,” in Proc. 34th International Conference on Machine Learning, 2017, pp. 1321-1330. [Online]. Available: https://proceedings.mlr.press/v70/guo17a.html.

[18] B. Efron and R. Tibshirani, “Bootstrap Methods for Standard Errors, Confidence Intervals, and Other Measures of Statistical Accuracy,” Statistical Science, vol. 1, no. 1, pp. 54-75, 1986. https://doi.org/10.1214/ss/1177013815.

[19] B. Barz and J. Denzler, “Do We Train on Test Data? Purging CIFAR of Near-Duplicates,” Journal of Imaging, vol. 6, no. 6, Art. no. 41, 2020. https://doi.org/10.3390/jimaging6060041.

[20] C. Zauner, “Implementation and Benchmarking of Perceptual Image Hash Functions,” M.S. thesis, Upper Austria University of Applied Sciences, Hagenberg, Austria, 2010. [Online]. Available: https://zauner.nllk.net/files/thesis.pdf.

[21] D. R. Roberts et al., “Cross-Validation Strategies for Data with Temporal, Spatial, Hierarchical, or Phylogenetic Structure,” Ecography, vol. 40, no. 8, pp. 913-929, 2017. https://doi.org/10.1111/ecog.02881.

[22] S. Varma and R. Simon, “Bias in Error Estimation When Using Cross-Validation for Model Selection,” BMC Bioinformatics, vol. 7, Art. no. 91, 2006. https://doi.org/10.1186/1471-2105-7-91.

[23] M. R. Romadlon and A. N. Handayani, “YOLOv8 Implementation on British Sign Language System with Edge Detection Extraction,” Indonesian Journal of Data and Science, vol. 6, no. 2, pp. 211-221, 2025. https://doi.org/10.56705/ijodas.v6i2.276.

[24] M. E. Wijaya and A. N. Handayani, “Integration of YOLOv8 and Instance Segmentation in the Chinese Sign Language Recognition System,” Indonesian Journal of Data and Science, vol. 6, no. 2, pp. 241-250, 2025. https://doi.org/10.56705/ijodas.v6i2.247.

[25] A. K. F. Shobiry and R. Puspitasari, “Hybrid Feature Benchmark for Blood Cell Classification Using ResNet50 and EfficientNetV2 Features with SVM and ANN Classifiers via Unsupervised Segmentation,” International Journal of Artificial Intelligence in Medical Issues, vol. 3, no. 2, pp. 110-126, 2025. [Online]. Available: https://jurnal.yoctobrain.org/index.php/ijaimi/article/view/364.

[26] H. Oumarou, “Detection and Classification of Bacterial Skin Infections Using K-Nearest Neighbors Algorithm,” International Journal of Artificial Intelligence in Medical Issues, vol. 1, no. 2, pp. 124-136, 2023. https://doi.org/10.56705/ijaimi.v1i2.149.

[27] A. R. Manga, T. Amaliah, Y. Salim, T. Saitoh, H. Zhang, and H. Azis, “From Embeddings to Ensembles: A Controlled Comparison of ArcFace and Global CNN Features,” in 2026 20th International Conference on Ubiquitous Information Management and Communication, 2026, pp. 1-8. https://doi.org/10.1109/IMCOM69009.2026.11360880.

[28] A. R. Manga, Nirmala, H. Azis, F. Fattah, Y. Salim, and H. Darwis, “ResNet-50 for Flower Image Classification: A Comparative Study of Segmentation and Non-Segmentation Approaches,” in 2025 19th International Conference on Ubiquitous Information Management and Communication, 2025, pp. 1-6. https://doi.org/10.1109/IMCOM64595.2025.10857520.

[29] N. Rismayanti, “Predicting the Rule, Not the Load: A Construct-Validity and Validation-Leakage Audit of Steel Industry Load-Type Classification,” Nusantara Journal of Multidisciplinary Informatics, vol. 1, no. 1, pp. 34-41, 2026. [Online]. Available: https://njomi.org/index.php/njomi/article/view/5.

[30] N. Rismayanti and A. P. Utami, “Improving Multi-Class Classification on 5-Celebrity-Faces Dataset using Ensemble Classification Methods,” Indonesian Journal of Data and Science, vol. 4, no. 2, pp. 120-129, 2023. https://doi.org/10.56705/ijodas.v4i2.78.

[31] S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 6, pp. 1137-1149, 2017. https://doi.org/10.1109/TPAMI.2016.2577031.

[32] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “MobileNetV2: Inverted Residuals and Linear Bottlenecks,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018, pp. 4510-4520. https://doi.org/10.1109/CVPR.2018.00474.

[33] Y. Cui, M. Jia, T.-Y. Lin, Y. Song, and S. Belongie, “Class-Balanced Loss Based on Effective Number of Samples,” in Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 9268-9277. https://doi.org/10.1109/CVPR.2019.00949.

[34] H. Azis, R. A. Jalil, and A. R. Manga, “Comparative Performance of VGG16 and EfficientNetB0-Based Transfer Learning for Brain Tumor Classification,” Knowledge Engineering and Data Science, vol. 8, no. 2, pp. 185-197, 2025. https://doi.org/10.17977/um018v8i22025p185-197.

[35] Y. Salim, A. P. Utami, A. R. Manga, H. Azis, and F. T. Admojo, “Optimal Strategy for Handling Unbalanced Medical Datasets: Performance Evaluation of K-NN Algorithm Using Sampling Techniques,” Knowledge Engineering and Data Science, vol. 7, no. 2, pp. 176-186, 2024. https://doi.org/10.17977/um018v7i22024p176-186.