Policy debates in education are often framed by using international test scores, such as the Programme for International Student Assessment (PISA). The obvious presumption is that observed differences in test scores within and across countries reflect differences in cognitive skills and general content knowledge, the things which achievement tests are designed to measure. We challenge this presumption, by studying how much of the within-country and between-country variation in PISA test scores is associated with student effort, rather than true academic content knowledge. Drawing heavily on recent literature, we posit that our measures of student effort are actually proxy measures of relevant non-cognitive skills related to conscientiousness. Completing surveys and tests takes effort and students may actually reveal something about their conscientiousness by the amount of effort they show during these tasks. Our previous work, and that of others validates this claim (e.g. Boe, May and Boruch, 2002; Borghans and Schils, 2012; Hitt, Trivitt and Cheng, 2016; Hitt, 2016; Zamarro et al., 2016). Using parametrizations of measures of survey and test effort we find that these measures help explain between 32 and 38 percent of the observed variation in test scores across countries, while explaining only a minor share of the observed variation within countries.