Decision Quality Blog

A colleague alerted me to what I think is a startling piece of research into decision making dynamics. The specific instance in mind was a landmark attempt to predict the outcome of the decisions facing the US Supreme Court prospectively. In other words, before the term starts, to take a shot at calling the outcome of all the cases the Court intended to hear. Think about the magnitude of that one for a minute. Here’s a snip from the findings which were published by the Columbia Law Review . . .

What is notable, in light of all the attention focused on the Court, is that few have tried to systematically predict its decisions prospectively. Given the high economic, social, and political importance of the Court’s decisions, a model that could prospectively forecast decisionmaking at a high rate of accuracy would be an invaluable tool to litigants and Court-watchers, even if the model itself were incompletely theorized. But prediction also has the potential to advance explanation by verifying, undermining, or modifying preexisting conceptions of the best ways to study the Court and understand how the Justices arrive at their decisions.

Our study compares two distinct methods of forecasting Supreme Court action, each drawing on the insights and strengths of a different discipline. Thus, the two prediction methods diverge dramatically in terms of methodology, and in this sense embody many of the differences between law and political science discussed above. The most notable distinction inheres in the level of generality the two methods employ. The statistical model looks at only a handful of case characteristics, each of them gross features easily observable without specialized legal expertise, and builds on general patterns ascertained from all 628 cases decided by the Rehnquist Court since 1994 and prior to the 2002 Term. The model is indifferent to many of the specific legal and factual aspects of the cases, instead predicting outcomes based on the same six (and only six) observable characteristics of each case.19 The legal experts, by contrast, utilized particularized knowledge, such as the specific facts of the case or statements by individual Justices in similar cases. We did not constrain the experts to consider only “legal” factors that might drive the Court’s decision. But although many considered nonlegal factors such as the Justices’ policy preferences, the experts, unlike the statistical model, could (and did) consider particular case law and specific constitutional or statutory texts and were thus able to particularize their analysis with regard to single cases in a way that the model was not.

So if you just skimmed the previous, here’s the deal . . .

• Before this, nobody had done a prospective study of the court. All study and commentary by both political science and legal experts was done historically.
• The contest pitted a statistical model against the best and the brightest.

And the results? The model won going away.

The basic result of our study is that the statistical model did better by a fair margin in forecasting the outcomes of last Term’s cases: The model predicted 75% of the Court’s affirm/reverse results correctly, while the experts collectively got 59.1% right.

This experiment captures only one specific Term and only one specific group of Justices, cases, and experts. The results might well be different in a different Term or with different experts. But for the 2002 Term, the model achieved notable success by utilizing a set of factors that appear to correlate with the Justices’ decisionmaking. That a forecasting machine that is indifferent to specific doctrine and text can predict cases so well is interesting, surprising, and worthy of further thought.

It’s a long report and worth reading if you’re into this sort of thing. It turns out that the model did especially well in cases with economic activity and experts excelled with “judicial power” cases. The model did better at predicting the swing voters (Kennedy and O’Connor) and the exprts did better with the more ideologically extreme Justices.

What’s more interesting, and why you should care beyond your interest in the doings of the High Court, is that the model ultimately won for a simple reason. It focused on a small number of variables that were highly predictive: the “experts” thought about many more factors and tended to weight them unreasonably and inappropriately in relationship to their predictive value. So, to pick a really specific variable, cases that came through the Seventh Circuit Court of Appeals in California got reversed. So, if the answer to the question, “Where did the case come from?” was, “San Francisco,” there was nothing else you needed to know. You were a Hall of Fame shoe-in if you predicted that Supreme Court would rule to reverse.

We see this same phenomena at work all the time . . . people trying to make decisions, and a prediction is a class of decision, get all balled up thinking about and debating points that, while interesting, don’t really matter when it comes to driving the quality of the final outcome. The lesson, “Focus on what matters” is obvious, but hard to do. Even experts miss it. They outsmart themselves with all their knowledge. A “dumb” model built by smart people will reliably beat them assuming they are designed to seek out the levers and uncertainties that account for all the action (or variability) which this one did.

Tags: 2002Supreme Court, Predicting, Outcomes, Columbia Law Review, decision making, uncertainty, decision modeling