![](https://d3ilqtpdwi981i.cloudfront.net/Cjryhx7_g0648d9m386A341VYSo=/0x28:469x636/425x550/smart/https://bepress-attached-resources.s3.amazonaws.com/uploads/6b/b6/4a/6bb64a9e-2d9e-4174-b8d0-6c187654444f/OECDAI.png)
Other
AI scoring for international large-scale assessments using a deep learning model and multilingual data
(2023)
Abstract
Artificial Intelligence (AI) scoring for constructed-response items, using recent advancements in multilingual, deep learning techniques utilising models pre-trained with a massive multilingual text corpus, is examined using international large-scale assessment data. Historical student responses to Reading and Science literacy cognitive items developed under the PISA analytical framework are used as training data for deep learning together with multilingual data to construct an AI model. The trained AI models are then used to score and the results compared with human-scored data. The score distributions estimated based on the AI-scored data and the human-scored data are highly consistent with each other; furthermore, even item-level psychometric properties of the majority of items showed high levels of agreement, although a few items showed discrepancies. This study demonstrates a practical procedure for using a multilingual data approach, and this new AI-scoring methodology reached a practical level of quality, even in the context of an international large-scale assessment.
Keywords
- Artificial intelligence,
- Computer assisted assessment,
- Computer assisted grading,
- Computer software,
- performance factors,
- Scoring,
- multiple choice tests,
- multilingual materials,
- Large scale assessment
Disciplines
Publication Date
February 21, 2023
DOI
https://doi.org/10.1787/19939019
Citation Information
Okubo, T., Houlden, W., Montuoro, P., Reinertsen, N., Tse, C. S., & Bastianianic, T. (2023). AI scoring for international large-scale assessments using a deep learning model and multilingual data (OECD Education Working Papers No. 287; OECD Education Working Papers, Vol. 287). https://doi.org/10.1787/9918e1fb-en