Enhanced nonlinear optical response of a coherent atomic medium is the basis for many atomic sensors, and their performance is ultimately limited by the quantum fluctuations of the optical readout. Here we demonstrate that the off-resonant interactions, with the aid of the near-resonant process, can significantly modify the quantum noise of a coherent light field, even when its effect on the mean signal is negligible. The altered quantum optical noise distribution results in excess noise in the measurement quantity. We illustrate this concept by using an atomic magnetometer based on the nonlinear Faraday effect. These results show the existence of previously unnoticed factors in fundamental limitations in atomic magnetometry and could have impact in a wide range of atom-light-based precision measurements.