Loading...

Automated Lip-Reading robotic system based on convolutional neural network and long short-term memory

Gholipour, A ; Sharif University of Technology | 2021

299 Viewed
  1. Type of Document: Article
  2. DOI: 10.1007/978-3-030-90525-5_7
  3. Publisher: Springer Science and Business Media Deutschland GmbH , 2021
  4. Abstract:
  5. In Iranian Sign Language (ISL), alongside the movement of fingers/arms, the dynamic movement of lips is also essential to perform/recognize a sign completely and correctly. In a follow up of our previous studies in empowering the RASA social robot to interact with individuals with hearing problems via sign language, we have proposed two automated lip-reading systems based on DNN architectures, a CNN-LSTM and a 3D-CNN, on the robotic system to recognize OuluVS2 database words. In the first network, CNN was used to extract static features, and LSTM was used to model temporal dynamics. In the second one, a 3D-CNN network was used to extract appropriate visual and temporal features from the videos. The accuracy rate of 89.44% and 86.39% were obtained for the presented CNN-LSTM and 3D-CNN networks, respectively; which were fairly promising for our automated lip-reading robotic system. Although the proposed non-complex networks did not provide the highest accuracy for this database (based on the literature), 1) they were able to provide better results than some of the more complex and even pre-trained networks in the literature, 2) they are trained very fast, and 3) they are quite appropriate and acceptable for the robotic system during Human-Robot Interactions (HRI) via sign language. © 2021, Springer Nature Switzerland AG
  6. Keywords:
  7. Audition ; Brain ; Complex networks ; Convolution ; Human robot interaction ; Robotics ; Automated lip readings ; Convolutional neural network ; Deep learning ; Dynamic movements ; Lip reading ; Long short-term memory ; Robotic systems ; Sign language ; Social robots ; Long short-term memory
  8. Source: 13th International Conference on Social Robotics, ICSR 2021, 10 November 2021 through 13 November 2021 ; Volume 13086 LNAI , 2021 , Pages 73-84 ; 03029743 (ISSN) ; 9783030905248 (ISBN)
  9. URL: https://link.springer.com/chapter/10.1007/978-3-030-90525-5_7