Skip to main content
Article
FLECS: A Federated Learning Second-Order Framework via Compression and Sketching
arXiv
  • Artem Agafonov, Moscow Institute of Physics and Technology, Dolgoprudny, Russian Federation & Mohamed bin Zayed University of Artificial Intelligence
  • Dmitry Kamzolov, Moscow Institute of Physics and Technology, Dolgoprudny, Russian Federation & Mohamed bin Zayed University of Artificial Intelligence
  • Rachael Tappenden, University of Canterbury, Christchurch, 8041, New Zealand
  • Alexander Gasnikov, Moscow Institute of Physics and Technology, Dolgoprudny, Russian Federation & Institute for Information Transmission Problems, Moscow, Russian Federation & HSE University, Moscow, Russian Federation
  • Martin Takac, Mohamed Bin Zayed University of Artificial Intelligence
Document Type
Article
Abstract

Inspired by the recent work FedNL (Safaryan et al, FedNL: Making Newton-Type Methods Applicable to Federated Learning), we propose a new communication efficient second-order framework for Federated learning, namely FLECS. The proposed method reduces the high-memory requirements of FedNL by the usage of an L-SR1 type update for the Hessian approximation which is stored on the central server. A low dimensional 'sketch' of the Hessian is all that is needed by each device to generate an update, so that memory costs as well as number of Hessian-vector products for the agent are low. Biased and unbiased compressions are utilized to make communication costs also low. Convergence guarantees for FLECS are provided in both the strongly convex, and nonconvex cases, and local linear convergence is also established under strong convexity. Numerical experiments confirm the practical benefits of this new FLECS algorithm. © 2022, CC BY.

DOI
10.48550/arXiv.2206.02009
Publication Date
6-4-2022
Keywords
  • As numbers,
  • Central servers,
  • Communication cost,
  • Hessian-vector products,
  • Low dimensional,
  • Memory cost,
  • Memory requirements,
  • Newton-type methods,
  • Second orders,
  • Sketchings,
  • Learning systems,
  • Optimization and Control (math.OC)
Comments

Preprint: arXiv

Archived with thanks to arXiv

Preprint License: CC by 4.0

Uploaded 13 July 2022

Citation Information
A. Agafonov, D. Kamzolov, R. Tappenden, A. Gasnikov, and M. Takac, "FLECS: A Federated Learning Second-Order Framework via Compression and Sketching", 2022, arXiv:2206.02009