Article
Anchor rating in performance-based speaking tests
Working Papers on Language Acquisition and Education
(2002)
Abstract
With the increasing utilization of perfonnance-based assessments in English language programs,
procedures must be enhanced for making the judged perfonnances more reliable'. This study investigates
the potential of anchor rating through the utilization of many-facet Rasch measurement techniques in
order to make ratings more reliable and more consistent within a given test administration as well as from
one test administration to another. Five raters, two with a great deal of experience, one with some
experience, and two with no experience in rating tests of speaking ability, assigned ratings to sixty
graduate students ranging from intermediate to advanced levels of proficiency. Thetwo experienced raters
also re-rated videotaped tests of students that they had rated in two previous administrations of the test.
Results showed that the two highly experienced raters stayed generally consistent in their ratings over
time. A second finding of the study was that the pool of raters in the study varied significantly in their
relative severity, but became less diverse with experience and training. New raters also showed marked
changes in level of severity from one test administration to the next. It is argued that anchor ratingmay be
a viable means of improving the reliability and comparability of performance-based scores when anchor
raters are highly experienced and have a lot of stake in the test.
Keywords
- performance assessment,
- many-facet Rasch measurement
Disciplines
Publication Date
2002
Publisher Statement
Copyright 2002 International University of Japan
Citation Information
Gary Ockey. "Anchor rating in performance-based speaking tests" Working Papers on Language Acquisition and Education Vol. 12 (2002) p. 30 - 46 Available at: http://works.bepress.com/gary-ockey/11/