In order to study the importance of surface phonons on the electron−phonon relaxation in plasmonic nanoparticles, the effect of size, shape, and materials have recently been studied. Gold and silver nanoparticles have shown no dependence on size and shape while copper nanoparticles have shown some size dependence. This suggests that the bulk phonons, which are sensitive to the bulk-phase structure, are solely responsible for the relaxation of the hot electron in gold and silver plasmonic nanoparticles. The importance of bulk phonons should depend on the degree of crystallinity. In the present study, we have found that the electron−phonon relaxation rate decreases greatly when polycrystalline prismatic gold nanoparticles are annealed and transformed into nearly single-crystalline nanospheres. The results are explained by the presence of high-density grain boundaries with dense, high-frequency molecular type vibrations which are effective in removing the energy of the excited electrons in the polycrystalline prismatic nanoparticles.