Presentation
A qualitative analysis of student writing rejected by an Automated Essay Scoring system
European Association for Language Testing and Assessment
(2018)
Abstract
The Australian Council for Educational Research’s online writing assessment, eWrite, uses Vantage Learning’s Intellimetric automated essay scoring (AES) system to score students’ writing. Approximately 5% of submissions are returned by the AES system unmarked and tagged with an error code, and are subsequently marked by a trained human rater. The raters provide numeric scores using an analytic rubric, and in some instances also provide written commentary about the scores.
This paper reports on a qualitative analysis of the commentaries of a sample of 86 scripts from 84 writers that were rejected by the AES and marked by trained human raters. Analysis of the commentaries was performed following the interactive model of qualitative data analysis, supplemented by descriptive statistical analysis of the accompanying scores.
The results of the analysis are that the comments reflect a wide-range of differing ability levels, and the scripts also received a wide range of scores. This suggests that rejection was not strongly associated with writing quality. However, some writing features were identified as occurring frequently in the sample, leading to the formulation of hypotheses about which writing features increase the probability of a script being rejected. These are presented as directions for future research.
The results of the study offer insight into some of the qualitative differences between human marking and AES. Broader implications for the use of automated scoring in writing assessments are considered.
Keywords
- AES; Automated Scoring; Writing Assessment
Disciplines
Publication Date
May, 2018
Location
Bochum, Germany
Citation Information
Nathanael Reinertsen, Toby Newton and Caithlin Power. "A qualitative analysis of student writing rejected by an Automated Essay Scoring system" European Association for Language Testing and Assessment (2018) Available at: http://works.bepress.com/nathanael-reinertsen/2/