We investigate the interplay between coherent gap dynamics and damping in superconductors taken out of equilibrium by strong optical pulses with subgap terahertz frequencies. A semiphenomenological formalism is developed to include the damping within the electronic subsystem that arises from effects beyond Bardeen-Cooper-Schrieffer mean-field theory, such as interactions between Bogoliubov quasiparticles and decay of the Higgs mode. These processes, conveniently expressed as longitudinal T1 and transverse T2 relaxation times in the standard pseudospin language for superconductors, cause the gap amplitude to be suppressed after the pulse is turned off, but before the timescale where thermalization occurs due to coupling to the lattice. We show that our model quantitatively captures the experimental gap dynamics reported here of NbN and Nb3Sn through the picosecond timescale.