Article
Answering open ended questions: Does it matter if test takers have a choice of which questions to answer?
Testing International
(2018)
Abstract
Examinations remain one of the most popular methods for assessing knowledge. There are various questions formats used in exams and one of the most utilised formats includes short answer questions (SAQs) (Mullen, K & Schultz, M 2012). SAQs require examinees to provide a written answer of varying length to a prompt and depending on the exam, candidates can either be required to respond to all questions or select some questions to answer. In the instance where all questions are compulsory, it may be fair to compare the overall scores of the candidates directly. If however, candidates are allowed to answer different combinations of questions, the common practice of comparing total scores may in fact disadvantage or advantage particular candidates due to the varying difficulty of the questions selected and the level of knowledge required to obtain a certain score within a question. This study includes two types of analysis of the examination results of 118 candidates; classical analysis based on Classical Test Theory (CTT) and Rasch analysis. Candidates were required to answer any four of eight SAQs. When classical analysis was applied, some candidates were advantaged and some disadvantaged due to the varying difficulty of the questions they selected as well as the inconsistent levels of knowledge required to obtain different scores within questions themselves. It was concluded that applying Rasch analysis was the most appropriate methodology to obtain accurate results for candidates in order to compare performance across any combination of four questions from a total of eight SAQs.
Keywords
- Open ended questions,
- Tests,
- Examinations,
- Assessing knowledge,
- Test formats,
- Test analysis,
- Rasch
Disciplines
Publication Date
June, 2018
Citation Information
John J Barnard, L Davies, Neville Chiavaroli and M Trigg. "Answering open ended questions: Does it matter if test takers have a choice of which questions to answer?" Testing International Vol. 39 Iss. 2 (2018) p. 23 - 31 Available at: http://works.bepress.com/neville-chiavaroli/19/