14 December 2017


'Judging the Judiciary by the Numbers: Empirical Research on Judges' by  Jeffrey J. Rachlinski and Andrew J. Wistrich in (2017) 13 Annual Review of Law and Social Science asks
Do judges make decisions that are truly impartial? A wide range of experimental and field studies reveal that several extra-legal factors influence judicial decision making. Demographic characteristics of judges and litigants affect judges’ decisions. Judges also rely heavily on intuitive reasoning in deciding cases, making them vulnerable to the use of mental shortcuts that can lead to mistakes. Furthermore, judges sometimes rely on facts outside the record and rule more favorably towards litigants who are more sympathetic or with whom they share demographic characteristics. On the whole, judges are excellent decision makers, and sometimes resist common errors of judgment that influence ordinary adults. The weight of the evidence, however, suggests that judges are vulnerable to systematic deviations from the ideal of judicial impartiality.
The authors comment
Judges are the axle on which the wheels of justice turn. They manage pretrial proceedings, mediate settlement conferences, rule on motions, conduct bench trials, supervise jury trials, take guilty pleas, impose criminal sentences, and resolve appeals. In the process, they find facts, make or apply law, and exercise discretion. Judges wield enormous power and society therefore rightly expects much of them. Judges must be fair minded, impartial, patient, wise, efficient, and intelligent (Wistrich, 2010). They must set aside their politics and their prejudices, make rational decisions, and follow the law. (See, e.g., American Bar Association, Model Code of Judicial Conduct, 2011, Rules 1.1, 1.2, 2.2, 2.3, 2.4, 2.5, 2.8). But is it possible for judges to perform as we expect?
The answer to this question remains somewhat uncertain. Twenty years ago, Lawrence Baum (1997, p. 149) concluded, “Despite all the progress that scholars have made, progress that is accelerating today, we are a long way from achieving truly satisfying explanations of judicial behavior.” Much more research has been conducted since then, but judicial behavior still remains something of a mystery. Some scholars argue that judges behave rationally but make decisions that further their self-interest ( Epstein et al. 2013). That assertion, however, raises as many questions as it answers: What do judges see as their self-interest? Are fairness and impartiality their primary goals? What incentives do judges really face? After all, they rarely lose their positions and seldom get promoted. And even if judges primarily strive for fairness and impartiality, do they achieve these goals?
Research on human judgment and choice indicates that most people face cognitive limitations that lead them to make choices that do not consistently further their own ends (Ariely 2009). People commonly rely on intuition and simple shortcuts (or  heuristics) to make choices (Kahneman 2011). Heuristics can be effective and surprisingly accurate (Gigerenzer and Todd 1999), but can also lead to predictable mistakes when over-applied or misused. These problems plague professionals as well. Research on doctors, dentists, accountants, futures traders, and others shows that they all fail to live up to an idealized standard of judgment in many settings ( Ariely 2009). It would be surprising if judges are any different.
The available research on judges suggests that they sometimes f all short of the lofty ideal to which society holds them. A growing body of research supports the conclusion that although judges are often excellent decision makers, they have vulnerabilities. At the outset, we know that in some areas of law, judicial decisions are too chaotic. A study of immigration asylum decisions, for example, reveals that some judges grant asylum in a high percentage of cases while others almost never grant asylum (Ramji-Nogales et al. 2007). Asylum outcomes thus turn on the random assignment of a case to one judge or another. Decisions concerning whether to grant leave to appeal or to allow release on bond in immigration cases are similarly erratic ( Rehaag 2012; Ryo, 2016). Concerns about variation in conviction rates have also long haunted criminal law (Weisselberg and Dunworth, 1993). Even in criminal sentencing decisions in federal court, in which a highly structured set of guidelines cons trains judges, variation remains robust ( Scott 2011). Judges do not seem to decide as reliably as might be hoped or expected. Worse still, the variation does not just arise from chaos or a lack of meaningful standards, it arises from systematic vulnerabilities in how judges think.
This article surveys the empirical research that assesses whether judges live up to the standards of their profession. The evidence accumulated to date reveals that judges fall short in predictable ways. First, as the legal realists feared, judges’ personal characteristics influence their decision making. Specifically, the research indicates that when cases raise issues that are salient to judges’ personal characteristics, they do not consistently put their characteristics aside. Second, judges overreact to mechanisms of accountability, such as appellate review, retention, and promotion. Third, judges rely too heavily on intuitive ways of thinking that can be misleading. Fourth, in making decisions, judges sometimes rely on factors outside the record, including inadmissible evidence, their emotional reactions, and prejudices.
To be fair to judges, they labor under a great deal of academic scrutiny. The existing research on judicial decision making probably focuses too heavily on judicial failings. Scholars conduct their research with an eye towards showing that judges are politically motivated or biased. This is understandable, given the ideal of neutral judging that society expects from judges, but the emphasis on deviations likely makes judges seem worse than they are. The research includes several studies in which judges adhere to an ideal norm of neutrality, and we certainly include these in our review. No studies really provide usable estimates of how many cases are skewed by politics, prejudice, or other misjudgment, and the research does not support a means of making a reasonable estimate. The circumstances under which judges deviate from the norm are nevertheless worth exploring, not to make judges look bad, but to identify potential ways they might improve.
In reaching our conclusions, we review a diverse array of both experimental and field studies of judicial decision making. We set aside judges’ autobiographies and biographies, interviews of judges, careful parsing of individual opinions, and judges’ own accounts of how they make decisions. Such undertakings can provide valuable insights, but our focus lies on systematic empirical accounts of judicial decision making. These include archival studies of actual decisions and experiments or simulations using hypothetical cases. Although most research on judges emphasizes decisions of the US Supreme Court (especially since the Second World War), our focus lies with the state courts, lower federal courts, and a handful of international studies. Although the US Supreme Court is important, of course, it resolves few cases and represents only a tiny window into the judicial decision-making process. Each of the studies we incorporate into our analysis involves vastly more judges than the 39 people who have served on the Supreme Court in the last 70 years. The focus on the Supreme Court also tends to emphasize the role of politics in judging. Political influence is only one way judges can fail to meet the demands of their roles. We discuss this concern but expand upon it.