The present study aims at improving our understanding of the individual contribution of the components of total factor productivity (TFP) change to U.S. agricultural productivity. A novel sequential primal-dual estimation routine to calculate TFP change is proposed, using a multi-output input distance function in the first stage, followed by a cost minimization routine in the second stage. TFP change is estimated as the direct sum of the estimates of technical change, technical efficiency change, allocative efficiency change, input price effects, changes in output markup, and changes in returns to scale in each state. The validity of the proposed methodology is supported by the remarkable overlap and high correlation of our annual estimates of TFP change with the USDA’s measures of change in TFP by state. Although technical change tends to be the largest contributor to productivity change, it bears a low and statistically insignificant correlation with TFP change on an annual basis, whereas annual changes in the markup effect and returns to scale are highly and significantly correlated with TFP changes. This is the first study to find a slowdown of technical progress in the U.S. farm sector in the 1990s and 2000s, and technical regress during the farm crisis of the 1980s. While technical efficiency shows a positive overall trend, allocative efficiency shows a negative overall trend, and their combined effect (i.e., the overall cost efficiency) slows down TFP growth. The profound implications of these findings for agricultural policy design beg for a revision of the literature on the major drivers of TFP change.