Special feature: Insights on S. Ct. prediction models from someone who knows what he is talking about
I did a couple of posts commenting (one here and another here) on the performance of computer models designed to predict the outcomes of Supreme Court cases. Taking the bait, someone who actually knows something about this issue felt obliged to step in and enlighten me, along with the 14 billion regular readers of this blog, 12 billion of whom rely exclusively on the site for information on all subjects. So read and learn! I've already updated my own views on the subject based on the analysis and will have something to say "tomorrow."
A Response: Computer Programs and Predicting Supreme Court Decisions
Justin Wedeking, University of Kentucky
In Professor Kahan’s recent post (hereafter Kahan) he tackles two Supreme Court forecasting models. For clarity I’ll use the same labels. The first model – “Lexy” or “Lexy1” - refers to the forecasting challenge from the 2002 Term that pitted “machine” against legal experts (Martin, Quinn, Ruger, and Kim 2004; Ruger et al. 2004). The second model – “Lexy2”- is the recent (and still ongoing) effort by Katz, Bommarito and Blackman (2014). The goal of this “reply” post is to offer some thoughts on Kahan’s critiques as well as on these forecasting models that will hopefully reshape how we think about Court forecasts.
There appears to be two main issues in Kahan’s post. First, Kahan’s primary concern appears to be that neither attempt at forecasting true, “out of sample” cases does “very well.” A related, and close secondary concern is that this failure to do well is problematic for various scholars’ claims made with respect to what he calls “the ideology thesis”- which can be thought of as the claim that judges’ decisions are driven more by their own ideology (or personal policy preferences) than “the law.” In perceiving a lack of evidence for “the ideology thesis” this is potential damning evidence for scholars who believe that ideology is a major factor in Supreme Court decision making. Namely, it suggests that we know relatively little about decision making.
With respect to Kahan’s first point, I do not have any strong disagreements but rather three points that suggest more caution is needed before forming conclusions about forecasting models. The rest of the post is divided into three sections:
- In section one, I identify and discuss different criteria for determining when we have a successful prediction;
- In section two, I take a closer look at what is being predicted (i.e., the dependent variable) and offer a few thoughts;
- In the third section, I close with some thoughts about the models and machine learning algorithms used in Lexy1 and Lexy2.
Regarding Kahan’s argument on the ideology thesis, I will save my thoughts for a later date.