What Is Judicial Ideology, and How Should We Measure It?

David S. Law, Washington University in St Louis
Joshua B. Fischman, University of Virginia School of Law

Abstract

Scholars have for decades sought to measure judicial ideology and its impact on judicial behavior. They have not, however, taken care to identify the phenomenon that they are measuring in clear and convincing terms, or to evaluate different measurement approaches for their suitability to the task at hand. In this paper, we diagnose recurring weaknesses of the existing empirical literature, and we propose a measurement approach of our own that offers significant improvements over some of the most popular alternatives.

First, we identify the major conceptual and methodological obstacles to the empirical measurement of judicial ideology. Foremost among the conceptual challenges to be faced are the difficulty of defining ideology and the associated challenge of disentangling ideological and nonideological preferences. The failure of empirical researchers to confront this challenge is a fatal one because it is impossible to design and select appropriate tools for the measurement of a phenomenon that has not been properly specified. At the level of methodology, empirical researchers must also contend with the fact that ideology is a psychological phenomenon or latent trait that cannot be directly observed. From this fact stems the problem of observational equivalence: the observable behavior of judges is open to multiple interpretations and may be attributable to both ideological and nonideological motivations. Scholars must also contend with complications arising from the possibility that judicial ideology may be multidimensional, not only across areas of law but also within a given issue area.

Second, we discuss the strengths and weaknesses of different measurement approaches, and their suitability to different types of research questions. The two types of approaches discussed are those involving the use of a proxy for the judge's ideology, and those that rest upon an individualized assessment of a judge's ideology based upon his or her observable behavior. We also discuss issues surrounding the coding of cases for empirical analysis, with particular attention to the choice between unidimensional, multidimensional, and agnostic coding schemes.

Third, we perform a head-to-head competitive evaluation of two popular measurement approaches and a third approach of our own devising. Under our approach, we first analyze past voting behavior to arrive at an individualized estimate of each judge's ideology. We then estimate the impact of collegiality concerns and the cost of dissent on judicial voting. Finally, we use our estimates of both judicial ideology and the impact of collegiality to predict how each judge on a given panel will vote.

We test the predictive power of our approach against that of two popular proxy measures of judicial ideology-namely, party of appointing president and judicial "common space scores." For testing purposes, we use a data set that we coded consisting of approximately 1,900 asylum cases decided by the Ninth Circuit over a ten-year period. We find that common space scores perform only slightly better than party of appointing president at predicting actual judicial voting. Moreover, both proxy measures yield almost identical estimates of the impact of ideology on actual judicial voting. By contrast, our approach boasts greater predictive power than either of the proxy measures, and it further reveals that ideology has a much greater impact on judicial voting than can be discovered using either of the usual proxy-based measures.