Article
On the Global Convergence of Continuous-Time Stochastic Heavy-Ball Method for Nonconvex Optimization
Proceedings of the 2019 IEEE International Conference on Big Data (2019, Los Angeles, CA)
Abstract
We study the convergence behavior of a stochastic heavy-ball method with a small stepsize. Under a change of time scale, we approximate the discrete scheme by a stochastic differential equation that models small random perturbations of a coupled system of nonlinear oscillators. We rigorously show that the perturbed system converges to a local minimum in a logarithmic time. This indicates that for the diffusion process that approximates the stochastic heavy-ball method, it takes (up to a logarithmic factor) only a linear time of the square root of the inverse stepsize to escape from all saddle points. This results may suggest a fast convergence of its discrete-time counterpart. Our theoretical results are validated by numerical experiments.
Meeting Name
2019 IEEE International Conference on Big Data, Big Data 2019 (2019: Dec. 9-12, Los Angeles, CA)
Department(s)
Mathematics and Statistics
Keywords and Phrases
- Dissipative Nonlinear Oscillator,
- Heavy-Ball Method,
- Non-Convex Optimization,
- Saddle Point,
- Small Random Perturbations of Hamiltonian Systems
International Standard Book Number (ISBN)
978-172810858-2
Document Type
Article - Conference proceedings
Document Version
Citation
File Type
text
Language(s)
English
Rights
© 2019 Institute of Electrical and Electronics Engineers (IEEE), All rights reserved.
Publication Date
12-1-2019
Publication Date
01 Dec 2019
Disciplines
Citation Information
Wenqing Hu, Chris Junchi Li and Xiang Zhou. "On the Global Convergence of Continuous-Time Stochastic Heavy-Ball Method for Nonconvex Optimization" Proceedings of the 2019 IEEE International Conference on Big Data (2019, Los Angeles, CA) (2019) p. 94 - 104 Available at: http://works.bepress.com/wenqing-hu/22/