We analyze the convergence of a multigrid algorithm for the Hybridizable Discontinuous Galerkin (HDG) method for diffusion problems. We prove that a non-nested multigrid V-cycle, with a single smoothing step per level, converges at a mesh independent rate. Along the way, we study conditioning of the HDG method, prove new error estimates for it, and identify an abstract class of problems for which a nonnested two-level multigrid cycle with one smoothing step converges even when the prolongation norm is greater than one. Numerical experiments verifying our theoretical results are presented.