The design and use of a valid and reliable instrument for the assignment of cognitive complexity ratings to chemistry exam items is described in this paper. Use of such an instrument provides a simple method to quantify the cognitive demands of chemistry exam items. Instrument validity was established in two different ways: statistically significant correlations between expert-based cognitive complexity ratings and student performance (as measured through statistical difficulty of items), and statistically significant correlations between expert-based cognitive complexity ratings and student mental effort ratings. Key benefits associated with instrument use include an enhanced understanding of the cognitive complexity of chemistry assessment tasks and as a means for characterizing exam content for the measurement of cognitive development.