Skip to main content
Article
D2-Net: Weakly-supervised action localization via discriminative embeddings and denoised activations
arXiv
  • Sanath Narayan, Inception Institute of Artificial Intelligence
  • Hisham Cholakkal, Mohamed bin Zayed University of Artificial Intelligence
  • Munawar Hayat, Monash University
  • Fahad Shahbaz Khan, Mohamed bin Zayed University of Artificial Intelligence & Linköping University
  • Minghsuan Yang, University of California & Google Research & Yonsei University
  • Ling Shao, Inception Institute of Artificial Intelligence
Document Type
Article
Abstract

This work proposes a weakly-supervised temporal action localization framework, called D2-Net, which strives to temporally localize actions using video-level supervision. Our main contribution is the introduction of a novel loss formulation, which jointly enhances the discriminability of latent embeddings and robustness of the output temporal class activations with respect to foreground-background noise caused by weak supervision. The proposed formulation comprises a discriminative and a denoising loss term for enhancing temporal action localization. The discriminative term incorporates a classification loss and utilizes a top-down attention mechanism to enhance the separability of latent foreground-background embeddings. The denoising loss term explicitly addresses the foreground-background noise in class activations by simultaneously maximizing intra-video and inter-video mutual information using a bottom-up attention mechanism. As a result, activations in the foreground regions are emphasized whereas those in the background regions are suppressed, thereby leading to more robust predictions. Comprehensive experiments are performed on multiple benchmarks, including THUMOS14 and ActivityNet1.2. Our D2-Net performs favorably in comparison to the existing methods on all datasets, achieving gains as high as 2.3% in terms of mAP at IoU=0.5 on THUMOS14. Source code is available at https://github.com/naraysa/D2-Net. Copyright © 2020, The Authors. All rights reserved.

DOI
doi.org/10.48550/arXiv.2012.06440
Publication Date
12-11-2020
Keywords
  • Computer Vision and Pattern Recognition (cs.CV)
Disciplines
Comments

Preprint: arXiv

Citation Information
S. Narayan, H. Cholakkal, M. Hayat, F.S. Khan, M. Yang, and L. Shao, "D2-Net: Weakly-supervised action localization via discriminative embeddings and denoised activations", 2020, arXiv:2012.06440