The joint impact of atmospheric refraction and turbulence effects on the modulation transfer function (MTF) of an incoherent imaging system is analyzed using numerical simulations based on the brightness function technique. Atmospheric refractivity is described by a combination of the standard MUSA76 and inverse temperature layer (ITL) models, and atmospheric turbulence effects are accounted for using the classical Kolmogorov turbulence framework. We show that ITL-induced refractivity located in the vicinity of an air volume essential for image formation can create strongly spatially anisotropic phase aberrations, which may significantly impact imaging system performance. Numerical analysis demonstrates that these phase aberrations may change the image size, and cause spatial frequency shifts in observed images of horizontally oriented sine-patterns. Numerical simulations show the that combined effects of atmospheric turbulence and ITL-induced refractivity cause the MTF to be a nonlinear function that is dependent on the imaging system characteristics, propagation path geometry, and atmospheric turbulence and ITL-induced refractivity parameters.