Prof. Geoff Masters

Assessing Student Learning: Why Reform is Overdue

Geoff Masters — Wed, 19 Oct 2011 21:06:58 PDT

Advances in our understanding of human learning require new approaches to assessing and monitoring student learning. Much assessment thinking has changed little over the past fifty years. The field continues to be dominated by twentieth century introductory textbook concepts, including such dichotomies as formative versus summative assessment, criterion-referenced versus norm-referenced testing, quantitative versus qualitative assessment, informal versus formal assessment – distinctions that often hamper rather than promote clear thinking about assessment. Assessment practice also has changed little over this period. Traditional, high-stakes examinations continue to dominate what is taught and learnt in many of our schools and universities. Greater use is now being made of promising new technologies, including banks of online assessment tasks, computer adaptive tests and technology-based assessments of ‘new’ life skills and attributes. However, while emerging technologies are capable of providing more innovative and informative explorations of student learning, much electronic assessment remains pedestrian and underpinned by traditional assessment thinking. At the same time, progress in our understanding of learning itself is challenging long-held assumptions and pointing to the need for a paradigm shift in assessment theory and practice.

The Hard Work of Improvement

Geoff Masters — Wed, 19 Oct 2011 20:59:21 PDT

It is now widely recognised that, when performances are evaluated only in terms of measurable results, employees and organisations find ways to ‘game’ the system. Hospitals improve patient survival rates by taking fewer high risk patients; companies maximise short-term returns to shareholders by not investing in long-term growth strategies. And in extreme cases, a narrow focus on results produces corrupt behaviour – for example, manipulating a company’s financial results to make its performance look better than it is. There are obvious lessons in this experience for current efforts to improve educational outcomes. Following the model adopted in business, education systems in a number of countries are now attempting to drive improved performance by placing a strong focus on results such as student test scores, participation levels and school completion rates. These results metrics are being used to set targets for improvement and to hold teachers and schools accountable for producing better results, often with accompanying incentive schemes.

The Power of Expectation

Geoff Masters — Wed, 19 Oct 2011 20:46:35 PDT

Success in most fields of endeavour depends on an ability to visualise success. It has long been known that elite athletes mentally rehearse each performance prior to its execution. Advances in neuroscience show why this may be so important: the neurological processes involved in visualising a performance are almost identical to those involved in the performance itself. Indeed, simply watching somebody else perform activates ‘mirror’ neurons in the observer paralleling neuronal activity in the performer. The ability to visualise success and an accompanying belief that success is possible appear to be prerequisites for most forms of human achievement.

Progression and assessment; Developmental assessment

Margaret Forster et al. — Thu, 01 Jul 2010 23:45:04 PDT

Objective Measurement

Geoff Masters — Tue, 20 Apr 2010 18:46:45 PDT

The Rasch model is described as fundamental to objective measurement and this chapter examines key issues of conceptualising variable, especially in education. The notion of inventing units for objective measurement is discussed, and its importance in developmental assessment is highlighted.

Improving Literacy and Numeracy Outcomes

Geoff Masters — Mon, 05 Oct 2009 18:20:30 PDT

Understanding and Leading Learning

Geoff Masters — Mon, 05 Oct 2009 18:14:12 PDT

Item discrimination: when more is worse.

Geoff Masters — Mon, 05 Oct 2009 18:06:58 PDT

High item discrimination can be a symptom of a special kind of measurement disturbance introduced by an item that gives persons of high ability a special advantage over and above their higher abilities. This type of disturbance, which can be interpreted as a form of item bias, can be encouraged by methods that routinely interpret highly discriminating items as the best items on a test and may be compounded by procedures that weight items by their discrimination. The type of measurement disturbance described and illustrated in this paper occurs when an item is sensitive to individual differences on a second, undesired dimension that is positively correlated with the variable intended to be measured. Possible secondary influences of this type include opportunity to learn, opportunity to answer, and test wiseness.

Reporting and Comparing School Performances

Geoff N. Masters et al. — Mon, 05 Oct 2009 17:59:24 PDT

This report provides advice on the collection and reporting of information about the performances of Australian schools. The focus is on the collection of nationally comparable data. Two purposes are envisaged: use by education authorities and governments to monitor school performances and, in particular, to identify schools that are performing unusually well or unusually poorly given their circumstances; and use by parents/caregivers and the public to make informed judgements about, and meaningful comparisons of, schools and their offerings. Our advice is based on a review of recent Australian and international research and experience in reporting on the performances of schools. This is an area of educational practice in which there have been many recent developments, much debate and a growing body of relevant research.

Mapping student achievement

Geoff Masters et al. — Thu, 17 Sep 2009 17:57:31 PDT

The analyses, publications and reports of quantitative data in education and the social sciences usually omit basic information on the construct measured. Probabilistic models for test data make it possible to delineate coherent and richly described measurement continua that facilitate interpretation of student achievement. The potential of conjoint measurement to bring about fundamental advances in educational testing practice lies in part in the opportunities it provides to build useful maps of learning domains and to use those maps in communicating student achievements. This chapter presents two applications of conjoint measurement aimed at constructing and describing achievement variables, developing insights into the structure of learning domains, and providing descriptive interpretations of students' levels of achievement within those domains. Both applications are taken from work of the Australian Council for Educational Research in Melbourne.

A rasch model for partial credit scoring

Geoff Masters — Thu, 17 Sep 2009 17:23:19 PDT

A unidimensional latent trait model for responses scored in two or more ordered categories is developed. This “Partial Credit” model is a member of the family of latent trait models which share the property of parameter separability and so permit “specifically objective” comparisons of persons and items. The model can be viewed as an extension of Andrich's Rating Scale model to situations in which ordered response alternatives are free to vary in number and structure from item to item. The difference between the parameters in this model and the “category boundaries” in Samejima's Graded Response model is demonstrated. An unconditional maximum likelihood procedure for estimating the model parameters is developed.

New views of student learning: implications for educational measurement

Geoff Masters et al. — Thu, 17 Sep 2009 16:57:26 PDT

Recent research in cognitive psychology has drawn attention to the important role that students' personal understandings and representations of subject matter play in the learning process. This chapter briefly reviews some of this research, and contrasts the kind of learning that results in an individual's changed conception or view of a phenomenon with the more passive, additive kind of learning assessed by most traditional achievement tests. To be consistent with a view of learning as an active, constructive process, educational tests are required that focus on key concepts in an area of learning, and that take into account the variety of types and levels of understanding that students have of those concepts. In these tests, scoring responses right and wrong is likely to be less appropriate than using students' answers to infer their levels of understanding. This will require not only imaginative new types of test items, but statistical models that permit inferences about students' understandings once their responses have been observed. Psychometric approaches are sketched to construct measures of achievement from such tests.

The construction of tertiary entrance scores: principles and issues

D G. Beswick et al. — Thu, 17 Sep 2009 16:46:27 PDT

Restoring our edge in education : making Australia's education system its next competitive advantage

Geoff Masters — Thu, 17 Sep 2009 16:36:15 PDT

Future levels of educational attainment in Australia will be key determinants of individual, social and economic prosperity. Policies that deliver an increase in education and training levels will have wide-ranging benefits for individuals, standards of living and social cohesion. Over recent decades there have been steady increases in the education levels of young Australians. Future generations will be much more highly educated as current levels of educational achievement flow through the age structure over time. However, despite generally increasing education levels, Australia has one of the lowest secondary school completion rates among comparable countries. Australia's future economic competitiveness will depend on increasing the numbers of young people who complete 12 years of school or equivalent. And, while young Australians perform well on average, large numbers of young people leave school with unacceptably low levels of school achievement. This paper focuses on two specific concerns: first, the significant proportion of young people who become disengaged during their school years, achieve only minimal educational outcomes and have limited subsequent engagement in work or further learning; and second, the shortage of young people with the knowledge and skills required for effective participation in the Australian workforce. Five strategies are identified for addressing these concerns.

The Partial Credit Model

Geoff Masters — Thu, 10 Sep 2009 23:31:23 PDT

This comprehensive Handbook focuses on the most used polytomous item response theory (IRT) models. These models help us understand the interaction between examinees and test questions where the questions have various response categories. The book reviews all of the major models and includes discussions about how and where the models originated, conceptually and in practical terms. Diverse perspectives on how these models can best be evaluated are also provided. Practical applications provide a realistic account of the issues practitioners face using these models. Disparate elements of the book are linked through editorial sidebars that connect common ideas across chapters, compare and reconcile differences in terminology, and explain variations in mathematical notation. These sidebars help to demonstrate the commonalities that exist across the field. By assembling this critical information, the editors hope to inspire others to use polytomous IRT models in their own research so they too can achieve the type of improved measurement that such models can provide.

Part 1 examines the most commonly used polytomous IRT models, major issues that cut across these models, and a common notation for calculating functions for each model. An introduction to IRT software is also provided. Part 2 features distinct approaches to evaluating the effectiveness of polytomous IRT models in various measurement contexts. These chapters appraise evaluation procedures and fit tests and demonstrate how to implement these procedures using IRT software. The final section features groundbreaking applications.

Assessing achievement in Australian schools

Geoff Masters — Thu, 10 Sep 2009 23:16:45 PDT

This paper considers the educational and technical feasibility of introducing a comprehensive system of student performance measures which will allow valid and reliable national and international comparisons. The conclusion reached is that a useful system of student performance measures can be established in this country, and some specific recommendations are made for the implementation of such a system. But the discussion that follows also contains important caveats and warnings. Difficulties in establishing valid and reliable measures of student achievement are not to be underestimated. Many education systems, particularly large systems in Europe and the United States which have had performance measures in place for many years, are in the process of dismantling and re- building their assessment programs to make them more inclusive of the broad range of goals that we now have for students in our schools, and to make them more useful in day-to-day classroom learning. There are lessons to be learned in Australia from the experiences of education systems in other countries.

Competency based assessment in the professions

Geoff Masters et al. — Thu, 10 Sep 2009 23:13:15 PDT

The purpose of this paper is: to assist professions to develop, in conjunction with National Office of Overseas Skills Recognition, the most suitable methods of assessing individuals against national competency standards; to provide advice on assessment methods likely to be practical and useful for the professions generally; and to provide advice on assessment methods consistent with the Commonwealth government's principles for the reform of the overseas skills recognition process. The authors consider alternative ways of conceptualising occupational competence and show how different conceptualisations can have very different implications for assessment. They briefly review the notion of competency based assessment. What are its distinguishing characteristics? How is it different from current practice? The review of previous attempts to implement competency based assessment reveals that these attempts have often been associated with detailed check lists of occupational skills. The purpose of assessment under this approach is to establish whether or not each listed skill has been mastered. It is concluded that a narrow, behaviouristic approach of this kind is unlikely to be appropriate for defining and assessing professional competence, and an approach to competency based assessment in the professions based on realistic, complex workplace problems as recommended instead.

Rating scale analysis

Benjamin D. Wright et al. — Thu, 10 Sep 2009 22:29:26 PDT

Profiles of learning : the Basic Skills Testing Program in New South Wales : 1989

Geoff Masters — Thu, 10 Sep 2009 22:27:00 PDT

The 1989 Basic Skills Testing Program in New South Wales provides the most comprehensive picture yet compiled of literacy and numeracy learning in Australian primary schools. In 1989, some 53,800 Year 6 students in NSW government schools were tested in five aspects of literacy and numeracy. Another 2,300 Year 3 students took part in a pilot study. This book discusses the writing of the tests, the analysis of results, and the reporting of results to parents, teachers and schools. The aim of the basic skills tests is to describe, in positive terms, the skills that students have mastered, to identify areas in which students have special strengths and weaknesses, and to provide guides to further learning. The picture that emerges from this study is of widespread success in learning. The results point to much good teaching and a fine start in learning for most primary students. But this generally optimistic picture is over shadowed by the performances of some students who have not yet mastered essential Year 6 skills. These students need special help if they are not to fall further behind in their learning.

The essential process in a family of measurement models

Geoff Masters et al. — Thu, 10 Sep 2009 21:52:26 PDT

Five members of the Rasch family of latent trait models which have appeared more or less independently in the literature are brought together and identified as one model. In addition to sharing the distinguishing characteristic of the dichotomous Rasch model separable person and item parameters and hence sufficient statistics all five models share a common algebraic form and have as their basic element the fundamental process defined by Rasch's simple logistic expression. In these models, the sufficient statistics for person and item parameters are counts of events constructed to be indicative of the variable being measured, and the measures they enable are fundamental.