Sign language recognition using spiking neural networks

Chaudhari, Pranav and Vicente-Sola, Alex and Basu, Amlan and Manna, Davide L. and Kirkland, Paul and Di Caterina, Gaetano (2024) Sign language recognition using spiking neural networks. Procedia Computer Science, 235. pp. 2674-2683. ISSN 1877-0509 (https://doi.org/10.1016/j.procs.2024.04.252)

[thumbnail of Chaudhari-etal-PCS-2024-Sign-language-recognition-using-spiking]
Preview
 Text. Filename: Chaudhari-etal-PCS-2024-Sign-language-recognition-using-spiking.pdf
Final Published Version
License: Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 logo
Download (596kB)| Preview

Abstract

In recent years, research in automatic Sign Language Recognition (SLR) has undergone significant progress, serving as a foundational base for developing applications that aim to promote the integration of deaf individuals into society. Most of this progress is owed to the recent developments in deep learning. However, the deployment of conventional Artificial Neural Networks (ANNs) can be hindered by their requirements in terms of computational power and energy consumption. Therefore, to improve the efficiency of current SLR systems, in this work, we propose the use of the increasingly popular Spiking Neural Networks (SNNs), which, on the one hand, provide more energy-efficient computations than conventional ANNs and, on the other hand, are able to process temporal sequences with simpler architectures thanks to their temporal dynamics. To evaluate our method, we utilize WLASL300, the 300-word (300 classes of signs) dataset fromWord-Level American Sign Language, and achieve an improvement in accuracy with the SNN (+2.70%) over the previous state-of-the-art, when working with energy-efficient spiking neurons. Furthermore, we construct a non-spiking version of the same network and evaluate it in a similar manner. Our results demonstrate how the SNN has sparser activations (25% less), thanks to the use of spiking neurons, and therefore can be implemented with a lower power requirement than an ANN version of the same architecture. This work thus demonstrates the possibility of performing SLR in a very effective and efficient way, thus opening up the development of applications that span from the automatic real-time translation of dynamic signs to remote control utilizing sign languages.

ORCID iDs

Chaudhari, Pranav, Vicente-Sola, Alex ORCID logoORCID: https://orcid.org/0000-0002-2370-6562, Basu, Amlan ORCID logoORCID: https://orcid.org/0000-0002-0180-8090, Manna, Davide L. ORCID logoORCID: https://orcid.org/0000-0001-8963-5050, Kirkland, Paul ORCID logoORCID: https://orcid.org/0000-0001-5905-6816 and Di Caterina, Gaetano ORCID logoORCID: https://orcid.org/0000-0002-7256-0897;
CORE (COnnecting REpositories)