Skip to main content
Contribution to Book
Factors that influence the difficulty of problem-solving items
The Nature of Problem Solving: Using Research to Inspire 21st Century Learning (2017)
  • Ray Philpot, Australian Council for Educational Research (ACER)
  • Dara Ramalingam, Australian Council for Educational Research (ACER)
  • John A Dossey, Illinois State University
  • Barry McCrae, Australian Council for Educational Research (ACER)
Abstract
This chapter presents a study undertaken by the authors using data from the PISA 2012 computer-based assessment of problem solving. It considered ten characteristics understood to influence the difficulty of items used in problem-solving assessment. Each item was rated to reflect the amount of each characteristic it possessed. The item responses from about 85 000 participants from 44 countries were analysed to obtain item response theory (IRT) estimates of item difficulties. The predictor characteristics were analysed in a number of ways, including a hierarchical cluster analysis, regression analysis with item difficulty as outcome variable and a principal component factor analysis. The main characteristics predicting difficulty seem to be: the complexity and type of reasoning skills involved in solving the problem; the amount of opportunity the solver is given to experiment or uncover hidden facets in a problem scenario (more opportunity to explore and experiment will make a problem easier); and the number and nature of constraints that a solution must satisfy (complex or conflicting constraints will make a problem more difficult).
Keywords
  • International comparisons,
  • Large scale assessment,
  • Problem solving,
  • Programme for International Student Assessment (PISA),
  • Secondary school students,
  • Academic achievement,
  • Achievement tests,
  • Test construction,
  • Scoring
Publication Date
2017
Editor
Ben Csapo and Joachim Funke
Publisher
OECD Publishing
ISBN
2076-9679
DOI
http://dx.doi.org/10.1787/9789264273955-11-en
Publisher Statement
Copyright OECD
Citation Information
Philpot, R., Ramalingam, D., Dossey, J. & Mccrae, B. (2017), "Factors that influence the difficulty of problem-solving items", in The Nature of Problem Solving: Using Research to Inspire 21st Century Learning, OECD Publishing, Paris. DOI: http://dx.doi.org/10.1787/9789264273955-11-en