Presentation
Interpreting and using automated scoring in the clasroom
4th Annual International Conference of the Asian Association of Language Assessment
(2017)
Abstract
Automated scoring of student writing is increasingly widespread and used in a variety of high-stakes tests, including academic placement and college admissions tests. The widespread adoption of such systems has not allayed suspicion from some quarters about how Automated Essay Scoring systems mark, particularly given the inability of computer systems to read in the same way as humans. Particularly for classroom teachers, who may not necessarily understand how automated scores are generated for their students’ work, questions remain about what automatically assigned scores mean and how they can be interpreted and used.
This paper reports on the process of developing an argument-based approach (Kane, 2013) to validating the use of automatically-scored assessments by classroom teachers. The structure of the validity argument takes the form of the Toulmin model for reasoning (as cited in Kane, 2013; Williamson, 2013), which requires the development of warrants and backing to support inferences drawn from the students’ writing, including the scores assigned. The model also requires anticipating potential rebuttals to the argument and noting necessary qualifiers, so the paper addresses some of the limitations and boundaries of what automatically generated scores can tell teachers about their students’ writing. Some of the uses of AES-scored assessments in the classroom are highlighted, particularly in the capacity of additional confirmatory evidence of student performance, as justification for teacher professional judgments, and a basis for giving constructive feedback to students about areas for improvement.
The context of the research is an Australian online writing assessment (eWrite) developed by the Australian Council for Educational Research that is designed to provide classroom teachers with information about their students’ writing performances. eWrite uses an AES system to produce scores for several analytical writing criteria, as well as scale scores and ‘band’ scores. The writing is produced under classroom test conditions. The assessment is ‘low-stakes’, in that there are no systemically-imposed consequences for students based on their performance in the assessment, though several school systems have adopted eWrite as a common assessment to provide one measure of students writing ability in a region.
References
Kane, M. (2013). Validating the Interpretations and Uses of Test Scores. Journal Of Educational Measurement, 50(1), 1-73. doi:10.1111/jedm.12000
Williamson, D. M. (2013). Probable cause: Developing warrants for automated scoring of essays. In M. D. Shermis, J. Burstein, M. D. Shermis, J. Burstein (Eds.) , Handbook of automated essay evaluation: Current applications and new directions (pp. 153-180). New York, NY, US: Routledge/Taylor & Francis Group.
Keywords
- Automated Essay Scoring,
- AES,
- Writing Assessment
Disciplines
Publication Date
June, 2017
Location
Taipei, Taiwan
Citation Information
Nathanael Reinertsen. "Interpreting and using automated scoring in the clasroom" 4th Annual International Conference of the Asian Association of Language Assessment (2017) Available at: http://works.bepress.com/nathanael-reinertsen/3/