Deep Neural Network-Based Sign Language Recognition: A Comprehensive Approach Using Transfer Learning with Explainability

  • 2024-09-11 18:17:44
  • A. E. M Ridwan, Mushfiqul Islam Chowdhury, Mekhala Mariam Mary, Md Tahmid Chowdhury Abir
  • 0

Abstract

To promote inclusion and ensuring effective communication for those who relyon sign language as their main form of communication, sign language recognition(SLR) is crucial. Sign language recognition (SLR) seamlessly incorporates withdiverse technology, enhancing accessibility for the deaf community byfacilitating their use of digital platforms, video calls, and communicationdevices. To effectively solve this problem, we suggest a novel solution thatuses a deep neural network to fully automate sign language recognition. Thismethodology integrates sophisticated preprocessing methodologies to optimisethe overall performance. The architectures resnet, inception, xception, and vggare utilised to selectively categorise images of sign language. We prepared aDNN architecture and merged it with the pre-processing architectures. In thepost-processing phase, we utilised the SHAP deep explainer, which is based oncooperative game theory, to quantify the influence of specific features on theoutput of a machine learning model. Bhutanese-Sign-Language (BSL) dataset wasused for training and testing the suggested technique. While training onBhutanese-Sign-Language (BSL) dataset, overall ResNet50 with the DNN modelperformed better accuracy which is 98.90%. Our model's ability to provideinformational clarity was assessed using the SHAP (SHapley AdditiveexPlanations) method. In part to its considerable robustness and reliability,the proposed methodological approach can be used to develop a fully automatedsystem for sign language recognition.

 

Quick Read (beta)

loading the full paper ...