Studies such as PISA that attempt to compare outcomes across educational systems are expensive and difficult to implement. Further, the results of such studies are routinely criticized by educational commentators - particularly when the results are not consistent with their preconceived ideas about the relative merits and efficiencies of various educational practices and systems. This chapter discusses what is done to ameliorate the threats to the validity of PISA in five areas often targeted by commentators and reviewers as sources of invalidity in international comparisons. The five areas discussed are: (1) sampling - are the samples of students who undertake assessment equally representative of the full cohort of 15-year-old students in each country? (2) Item selection - is the range of items used in the assessment equally valid, or does it better match the goals of some education systems than others? (3) Translation - when the tests are translated into the various languages, does this lead to changes that systematically give an advantage to some countries over others? (4) Implementation fidelity - because the data are gathered by local authorities within each country, how can we be sure that the tests are administered, scored and processed using equivalent and objective measures? (5) Motivational issues - the PISA tests are low-stakes for the students. Are students in different countries equally motivated to perform well, and does this influence the outcomes?