Skip to main content
Factors that influence the difficulty of problem solving items
International Congress of Psychology (2012)
  • Dara Ramalingam, ACER
  • Ray Philpot, ACER

Computer-based assessment of problem solving allows problems of both static and interactive natures to be posed. Examples of static problems are scheduling and logic puzzles in which all relevant information is available to the solver at the outset. Problems of an interactive nature, on the other hand, require exploration of the situation to acquire additional knowledge needed to solve the problem. Examples include discovering how to use an unfamiliar mobile telephone or automatic vending machine. This study used data from the 2011 Field Trial of the PISA 2012 computer-based assessment of problem solving which comprised 34 static and 45 interactive items. Fifteen characteristics understood to influence item difficulty were posited. Each item was rated as to the amount of each characteristic it possessed, with weights assigned to the characteristics based on how important a part each was believed to play in problem solving. The items were ranked as to their potential difficulty according to the sum of the weighted ratings. The responses from approximately 39,500 field trial participants from 43 countries were analysed to obtain IRT estimates of item difficulties. Several related analyses involving the predictor characteristics were then conducted. These included a hierarchical cluster analysis of the characteristics, regression analysis with item difficulty as outcome variable, and factor analysis. A tentative conclusion is that the main characteristics predicting item difficulty are “Number of constraints to be satisfied”, “Distance to goal” and “Amount of reasoning required.

  • Assessment,
  • Problem solving,
  • Logic puzzles,
  • PISA,
  • Computer based assessment,
  • Testing,
  • Item difficulty
Publication Date
July, 2012
Citation Information
Dara Ramalingam and Ray Philpot. "Factors that influence the difficulty of problem solving items" International Congress of Psychology (2012)
Available at: