Skip to main content
Article
Jointly-Learnt Networks for Future Action Anticipation Via Self-Knowledge Distillation and Cycle Consistency
IEEE Transactions on Circuits and Systems for Video Technology
  • Md Moniruzzaman
  • Zhaozheng Yin, Missouri University of Science and Technology
  • Zhihai He
  • Ming-Chuan Leu, Missouri University of Science and Technology
  • Ruwen Qin, Missouri University of Science and Technology
Abstract

Future action anticipation aims to infer future actions from the observation of a small set of past video frames. In this paper, we propose a novel Jointly learnt Action Anticipation Network (J-AAN) via Self-Knowledge Distillation (Self-KD) and cycle consistency for future action anticipation. In contrast to the current state-of-the-art methods which anticipate the future actions either directly or recursively, our proposed J-AAN anticipates the future actions jointly in both direct and recursive ways. However, when dealing with future action anticipation, one important challenge to address is the future's uncertainty since multiple action sequences may come from or be followed by the same action. Training an action anticipation model with one-hot-encoded hard labels that assign zero probabilities to incorrect, yet semantically similar actions may not handle the uncertain future. To address this challenge, we design a Self-KD mechanism to train our J-AAN, where the J-AAN gradually distills its own knowledge during the training to soften the hard labels to model the uncertainty on future action anticipation. Furthermore, we design a forward and backward action anticipation framework with our proposed J-AAN based on a cyclic consistency constraint. The forward J-AAN anticipates the future actions from the observed past actions, and the backward J-AAN verifies the anticipation of the forward J-AAN by anticipating the past actions from the anticipated future actions. The proposed method outperforms all the latest state-of-the-art action anticipation methods on the breakfast, 50Salads, and EPIC-Kitchens-55 datasets. This project will be publicly available on https://github.com/MoniruzzamanMd/J-AAN.

Department(s)
Mechanical and Aerospace Engineering
Second Department
Computer Science
Third Department
Engineering Management and Systems Engineering
Keywords and Phrases
  • Computational modeling,
  • Computer vision,
  • Cycle Consistency,
  • Future Action Anticipation,
  • Knowledge engineering,
  • Self-Knowledge Distillation,
  • Target recognition,
  • Task analysis,
  • Training,
  • Uncertainty
Document Type
Article - Journal
Document Version
Final Version
File Type
text
Language(s)
English
Rights
© 2023 Institute of Electrical and Electronics Engineers, All rights reserved.
Publication Date
1-1-2022
Publication Date
01 Jan 2022
Citation Information
Md Moniruzzaman, Zhaozheng Yin, Zhihai He, Ming-Chuan Leu, et al.. "Jointly-Learnt Networks for Future Action Anticipation Via Self-Knowledge Distillation and Cycle Consistency" IEEE Transactions on Circuits and Systems for Video Technology (2022) ISSN: 1558-2205; 1051-8215
Available at: http://works.bepress.com/ming-leu/429/