Skip to main content
Energy-based Latent Aligner for Incremental Learning
  • K.J. Joseph, Indian Institute of Technology, Hyderabad, India & Mohamed bin Zayed University of Artificial Intelligence
  • Salman Khan, Mohamed bin Zayed University of Artificial Intelligence & Australian National University, Australia
  • Fahad Shahbaz Khan, Mohamed bin Zayed University of Artificial Intelligence & Linköping University, Sweden
  • Rao Muhammad Anwer, Aalto University, Finland
  • Vineeth N. Balasubramanian, Indian Institute of Technology, Hyderabad, India
Document Type

Deep learning models tend to forget their earlier knowledge while incrementally learning new tasks. This behavior emerges because the parameter updates optimized for the new tasks may not align well with the updates suitable for older tasks. The resulting latent representation mismatch causes forgetting. In this work, we propose ELI: Energy-based Latent Aligner for Incremental Learning, which first learns an energy manifold for the latent representations such that previous task latents will have low energy and the current task latents have high energy values. This learned manifold is used to counter the representational shift that happens during incremental learning. The implicit regularization that is offered by our proposed methodology can be used as a plug-and-play module in existing incremental learning methodologies. We validate this through extensive evaluation on CIFAR-100, ImageNet subset, ImageNet 1k and Pascal VOC datasets. We observe consistent improvement when ELI is added to three prominent methodologies in class-incremental learning, across multiple incremental settings. Further, when added to the state-of-the-art incremental object detector, ELI provides over 5% improvement in detection accuracy, corroborating its effectiveness and complementary advantage to the existing art. Code is available at: © 2022, CC BY.

Publication Date
  • Computer vision,
  • Deep learning,
  • 'current,
  • Aligners,
  • Energy manifolds,
  • Energy-based,
  • High energy value,
  • Incremental learning,
  • Learn+,
  • Learning models,
  • Lower energies,
  • Regularisation,
  • Object detection,
  • Artificial Intelligence (cs.AI),
  • Computer Vision and Pattern Recognition (cs.CV),
  • Machine Learning (cs.LG)

Preprint: arXiv

Archived with thanks to arXiv

Preprint License: CC by 4.0

Uploaded 24 May 2022

Citation Information
K.J. Joseph, S. Khan, F.S. Khan, R.M. Anwer, and V.N. Balasubramanian, "Energy-based Latent Aligner for Incremental Learning", arXiv, Mar 2022, doi: 10.48550/arXiv.2203.14952