Environmentally mediated changes in body size often underlie population responses to environmental change, yet this is not a universal phenomenon. Understanding when phenotypic change underlies population responses to environmental change is important for obtaining insights and robust predictions of population dynamics in a changing world. We develop a dynamic integral projection model that mechanistically links environmental conditions to demographic rates and phenotypic traits (body size) via changes in resource availability and individual energetics. We apply the model to the northern Yellowstone elk population and explore population responses to changing patterns of seasonality, incorporating the interdependence of growth, demography, and density-dependent processes operating through population feedback on available resources. We found that small changes in body size distributions can have large impacts on population dynamics but need not cause population responses to environmental change. Environmental changes that altered demographic rates directly, via increasing or decreasing resource availability, led to large population impacts in the absence of substantial changes to body size distributions. In contrast, environmentally driven shifts in body size distributions could occur with little consequence for population dynamics when the effect of environmental change on resource availability was small and seasonally restricted and when strong density-dependent processes counteracted expected population responses. These findings highlight that a robust understanding of how associations between body size and demography influence population responses to environmental change will require knowledge of the shape of the relationship between phenotypic distributions and vital rates, the population status with regard to its carrying capacity, and importantly the nature of the environmentally driven change in body size and carrying capacity.