We analyze the impact of a proposed tidal instability coupling p modes and g modes within neutron stars on GW170817. This nonresonant instability transfers energy from the orbit of the binary to internal modes of the stars, accelerating the gravitational-wave driven inspiral. We model the impact of this instability on the phasing of the gravitational wave signal using three parameters per star: an overall amplitude, a saturation frequency, and a spectral index. Incorporating these additional parameters, we compute the Bayes factor (lnBpg!pg) comparing our p−g model to a standard one. We find that the observed signal is consistent with waveform models that neglect p−g effects, with lnBpg!pg=0.03+0.70−0.58 (maximum a posteriori and 90% credible region). By injecting simulated signals that do not include p−g effects and recovering them with the p−g model, we show that there is a ≃50% probability of obtaining similar lnBpg!pg even when p−g effects are absent. We find that the p−g amplitude for 1.4  M⊙ neutron stars is constrained to less than a few tenths of the theoretical maximum, with maxima a posteriori near one-tenth this maximum and p−g saturation frequency ∼70  Hz. This suggests that there are less than a few hundred excited modes, assuming they all saturate by wave breaking. For comparison, theoretical upper bounds suggest ≲103 modes saturate by wave breaking. Thus, the measured constraints only rule out extreme values of the p−g parameters. They also imply that the instability dissipates ≲1051  erg over the entire inspiral, i.e., less than a few percent of the energy radiated as gravitational waves.