Presentation
Reduce randomly guessing effects in a university generic skills test
11th Conference of International Test Commission (ITC)
(2018)
Abstract
There frequent concern of guessing with multiple-choice questions (MCQ) in item response theory (IRT), particularly with the Rach model where a guessing parameter is not included. A practical solution is done post-hoc by removing responses to items too hard for a low ability examinee (see Andrich, et al., 2011; RUMM 2030, Andrich et al., 2012; Winsteps, Linacre, 2012). This study is used data from The Special Tertiary Admissions Test (STAT) to explore whether such responses from lower ability candidates to a hard item in a real test could be considered as randomly guessing and how Rasch item difficulty estimates improve after tailoring the response data. In this study we disregard responses when examinees encountered an item with a probability of 25% or less to get the correct answer to the item. Results showed that in the STAT data, responses from very lower ability candidates to hard items are fairly uniformly distributed across the four options of the test items. It supports for the assumption of random guessing in these responses and removing such responses could help to obtain more accurate item difficulty estimates. Additional simulation data sets with similar structure to STAT are generated. Results from parameter recovery show that Tailoring data could help to improve the Rasch item difficulty estimation. However, Tailoring responses those are not random guessing would lead to be more biased in item estimation. This suggests that more constraints rather than only difference between student ability and item difficulty are required for Tailoring data.
Keywords
- Test items,
- Multiple choice tests,
- Guessing (Tests),
- Item response theory,
- Rasch model,
- University entrance examinations,
- Difficulty level,
- Generic skills
Disciplines
Publication Date
July, 2018
Location
Montreal, Canada
Citation Information
Luc T Le. "Reduce randomly guessing effects in a university generic skills test" 11th Conference of International Test Commission (ITC) (2018) Available at: http://works.bepress.com/luc_le/34/