This paper presents the results from a tertiary entrance test that was delivered to two groups of candidates, one as a paper based test and the other as a computer based test. Item level differential reveals a pattern that appears related to item type: questions based on diagrammatic stimulus show a pattern of increased difficulty when delivered on computer. Differential in performance was not present in other sections of the test and it would appear unlikely to be explained by demographic differences between the groups. It is suggested this differential is due to the inability of the candidates to freely annotate on the stimulus when delivered on computer screen. More work needs to be done on considering the role of annotation as a problem solving strategy in high stakes testing, in particular with certain kinds of stimulus, such as diagrams.