A Review of Multimodal Human Activity Recognition with Special Emphasis on Classification, Applications, Challenges and Future Directions

HARI MOHAN PANDEY*, Santosh Kumar Yadav, Kamlesh Tiwari, Shaik Ali Akbar

*Corresponding author for this work

Research output: Contribution to journalArticle (journal)peer-review

202 Citations (Scopus)
1835 Downloads (Pure)

Abstract

Human activity recognition (HAR) is one of the most important and challenging problems in the computer vision. It has critical application in wide variety of tasks including gaming, human-robot interaction, rehabilitation, sports, health monitoring, video surveillance, and robotics. HAR is challenging due to the complex posture made by the human and multiple people interaction. Various artefacts that commonly appears in the scene such as illuminations variations, clutter, occlusions, background diversity further adds the complexity to HAR. Sensors for multiple modalities could be used to overcome some of these inherent challenges. Such sensors could include an RGB-D camera, infrared sensors, thermal cameras, inertial sensors, etc. This article introduces a comprehensive review of different multimodal human activity recognition methods where different types of sensors being used along with their analytical approaches and fusion methods. Further, this article presents classification and discussion of existing work within seven rational aspects: (a) what are the applications of HAR; (b) what are the single and multi-modality sensing for HAR; (c) what are different vision based approaches for HAR; (d) what and how wearable sensors based system contributes to the HAR; (e) what are different multimodal HAR methods; (f) how a combination of vision and wearable inertial sensors based system contributes to the HAR; and (g) challenges and future directions in HAR. With a more and comprehensive understanding of multimodal human activity recognition, more research in this direction can be motivated and refined.
Original languageEnglish
Article number106970
JournalKnowledge-Based Systems
Volume223
Early online date17 Apr 2021
DOIs
Publication statusPublished - 8 Jul 2021

Keywords

  • Activity Recognition
  • Computer vison
  • Wearable sensors
  • Fusion of vision and inertial sensors
  • Smart-shoes
  • Multimodality

Fingerprint

Dive into the research topics of 'A Review of Multimodal Human Activity Recognition with Special Emphasis on Classification, Applications, Challenges and Future Directions'. Together they form a unique fingerprint.
  • Skeleton based Human Activity Recognition using ConvLSTM and Guided Feature Learning

    PANDEY, H. M., Yadav, S. K., Tiwari, K. & Akbar, S. A., 1 Oct 2021, (E-pub ahead of print) In: Soft Computing. SOCO-D-21-00092R2.

    Research output: Contribution to journalArticle (journal)peer-review

    Open Access
    File
    64 Citations (Scopus)
    284 Downloads (Pure)

Cite this