Skip to main content
Article
The influence of equating methodology on reported trends in PISA
Journal of Applied Measurement (2007)
  • Eveline Gebhardt, ACER
  • Ray J Adams, ACER
Abstract

In 2005 PISA published trend indicators that compared the results of PISA 2000 and PISA 2003. This paper explores the extent to which the outcomes of these trend analyses are sensitive to the choice of test equating methodologies, the choice of regression models and the choice of linking items. To establish trends, PISA equated its 2000 and 2003 tests using a methodology based on Rasch Modelling that involved estimating linear transformations that mapped 2003 Rasch-scaled scores to the previously established PISA 2000 Rasch-scaled scores. This paper compares the outcomes of this approach with an alternative, which involves the joint Rasch scaling of the PISA 2000 and PISA 2003 data separately for each country. Note that under this approach the item parameters are estimated separately for each country, whereas the linear transformation approach used a common set of item parameter estimates for all countries. Further, as its primary trend indicators, PISA reported changes in mean scores between 2000 and 2003. These means are not adjusted for changes in the background characteristics of the PISA 2000 and PISA 2003 samples - that is, they are marginal rather than conditional means. The use of conditional rather than marginal means results in some differing conclusions regarding trends at both the country and within-country level.

Keywords
  • Assessment,
  • Education,
  • Educational assessment,
  • Testing,
  • Trends,
  • International programs,
  • Monitoring,
  • Rasch model,
  • Test reliability,
  • Trend analysis,
  • PISA
Publication Date
January 1, 2007
Citation Information
Eveline Gebhardt and Ray J Adams. "The influence of equating methodology on reported trends in PISA" Journal of Applied Measurement Vol. 8 Iss. 3 (2007)
Available at: http://works.bepress.com/ray_adams/7/