Skip to main content
Article
Multi-level feature fusion for multimodal human activity recognition in Internet of Healthcare Things
Information Fusion
  • Md Milon Islam, University of Waterloo
  • Sheikh Nooruddin, University of Waterloo
  • Fakhri Karray, University of Waterloo & Mohamed bin Zayed University of Artificial Intelligence
  • Ghulam Muhammad, King Saud University
Document Type
Article
Abstract

Human Activity Recognition (HAR) has become a crucial element for smart healthcare applications due to the fast adoption of wearable sensors and mobile technologies. Most of the existing human activity recognition frameworks deal with a single modality of data that degrades the reliability and recognition accuracy of the system for heterogeneous data sources. In this article, we propose a multi-level feature fusion technique for multimodal human activity recognition using multi-head Convolutional Neural Network (CNN) with Convolution Block Attention Module (CBAM) to process the visual data and Convolutional Long Short Term Memory (ConvLSTM) for dealing with the time-sensitive multi-source sensor information. The architecture is developed to be able to analyze and retrieve channel and spatial dimension features through the use of three branches of CNN along with CBAM for visual information. The ConvLSTM network is designed to capture temporal features from the multiple sensors’ time-series data for efficient activity recognition. An open-access multimodal HAR dataset named UP-Fall detection dataset is utilized in experiments and evaluations to measure the performance of the developed fusion architecture. Finally, we deployed an Internet of Things (IoT) system to test the proposed fusion network in real-world smart healthcare application scenarios. The findings from the experimental results reveal that the developed multimodal HAR framework surpasses the existing state-of-the-art methods in terms of multiple performance metrics.

DOI
10.1016/j.inffus.2023.01.015
Publication Date
6-1-2023
Keywords
  • Convolutional block attention module,
  • Convolutional long short term memory,
  • Human activity recognition,
  • Internet of things,
  • Multi-head convolutional neural network
Comments

IR Deposit conditions:

OA version (pathway a) Accepted version

24 months embargo

License: CC BY-NC-ND

Must link to publisher version with DOI
Citation Information
Islam, M.M., Noor, S., Karray, F., and Muhammad, G., "Multi-level feature fusion for multimodal human activity recognition in Internet of Healthcare Things", Information Fusion, vol. 94, p. 17-31, doi: 10.1016/j.inffus.2023.01.015