We analyze the impact of a proposed tidal instability coupling p modes and g modes within neutron stars on GW170817. This nonresonant instability transfers energy from the orbit of the binary to internal modes of the stars, accelerating the gravitational-wave driven inspiral. We model the impact of this instability on the phasing of the gravitational wave signal using three parameters per star: An overall amplitude, a saturation frequency, and a spectral index. Incorporating these additional parameters, we compute the Bayes factor (ln Bpg!pg) comparing our p-g model to a standard one. We find that the observed signal is consistent with waveform models that neglect p-g effects, with ln Bpg!pg = 0.03+0.70-0.58 (maximum a posteriori and 90% credible region). By injecting simulated signals that do not include p-g effects and recovering them with the p-g model, we show that there is a ≃ 50% probability of obtaining similar ln Bpg!pg even when p-g effects are absent. We find that the p-g amplitude for 1.4 M⊙ neutron stars is constrained to less than a few tenths of the theoretical maximum, with maxima a posteriori near one-Tenth this maximum and p-g saturation frequency ∼70 Hz. This suggests that there are less than a few hundred excited modes, assuming they all saturate by wave breaking. For comparison, theoretical upper bounds suggest a ≲ 103 modes saturate by wave breaking. Thus, the measured constraints only rule out extreme values of the p-g parameters. They also imply that the instability dissipates a ≲ 1051 erg over the entire inspiral, i.e., less than a few percent of the energy radiated as gravitational waves.