California Supreme Court by the numbers: oral arguments

Although political science professors began applying data analytic techniques to the study of appellate decision-making in the late 1940s, analysts largely disregarded appellate oral arguments until the past 15 years. Before that time, many appellate lawyers insisted that oral argument gave no real indication of how a case was likely to come out, and just because any particular justice's questions seemed to indicate hostility (or favor) towards one advocate's case didn't mean that's how the justice will ultimately vote.

In this two-part series, we will look at the 2017 data from California Supreme Court to see what we can learn. In part one, we will look at the court as a whole. In part two, we will zoom in to the individual justices to see what we can learn. But first, let's look at a few of the oral argument studies that have been done regarding other courts, including the U.S. Supreme Court.

Oral Argument Studies

The earliest oral argument analytics study is Sarah Levien Shullman's 2004 article for the Journal of Appellate Practice and Process. Shullman attended 10 oral arguments at the U.S. Supreme Court during the October 2002 term. Shullman counted the questions asked by each Justice during oral argument, assigning a score between one and five to each depending on how helpful or hostile the question appeared to be. For the first seven cases, she compared the number and helpfulness score for each justice's questions to his or her ultimate vote in the case. She then used the resulting insights to predict results and votes in the remaining three cases. She showed that oral argument can in fact show an advocate a great deal about the likely result -- (1) the party asked the most questions generally lost; (2) justices are likely to ask more questions of the side they're voting against; (3) and if a particular justice is especially active in a case, asking significantly more questions than he or she typically does, that justice is probably writing an opinion.

Chief Justice John Roberts addressed oral argument analytics in an article published just before his nomination to the Supreme Court. Roberts counted the questions asked in the first and last cases for each argument session in 1980 and the first and last cases in each argument session for the October 2003 term. He found that the party asked the most questions lost 24 of 28 cases. Then-Judge Roberts told the Supreme Court Historical Society facetiously that it turned out that the key to appellate advocacy was "simply to get the court to ask your opponent more questions."

In 2009, Professor Timothy R. Johnson and three other professors reviewed transcripts from every Supreme Court Case decided between 1979 and 1995. The professors' study reached the same conclusion as the earlier work -- the side asked the most questions is far more likely to lose.

Professors Lee Epstein and William M. Landes and then-Judge Richard A. Posner published a separate study the same year. Because oral argument transcripts at the Supreme Court did not identify which justice asked a question until 2004, Professors Epstein and Landes and Judge Posner tracked the total questions asked by the Supreme Court for the years 1997 through 2004, and questions asked justice-by-justice for the years 2004 through 2007. Across 2,886 cases, they showed that (1) the side asked more questions is likely to lose; (2) the side asked longer questions is likely to lose; and (3) cases where total questions are relatively evenly distributed are likely to be decided by a close vote.

I have believed for years that oral argument is a reliable indication of what the result is likely to be in an appellate case and of the justices' likely votes. So two years ago, I decided to conduct the first large-scale oral argument analytics study I'm aware of for a lower court. I reviewed and charted every oral argument at the Illinois Supreme Court between 2008 and 2016. My conclusions were quite similar to those reached by the earlier scholars in regard to U.S. Supreme Court arguments. In both civil and criminal cases, the side asked the most questions was more likely to lose. The greater the difference between one side and the other in total questions, the less that side's chances of winning. Most justices more heavily questioned the side they would ultimately vote against. When a justice asked significantly more questions than his or her usual pattern, he or she was generally writing an opinion.

California Supreme Court

So what can we learn from the oral arguments in the 84 cases decided by the California Supreme Court in 2017? The court was slightly more active in civil than criminal cases, asking 2,572 questions in civil appeals and 2,473 in criminal cases.

Who's Asking?

In civil cases, the most active justice for appellants was Justice Mariano-Florentino Cuéllar, who asked 231 questions -- an average of 5.5 per case. Justice Carol Corrigan was the most active justice for criminal appellants, asking 306 questions, or 7.46 per case.

Justice Goodwin Liu was the most active questioner for respondents in both civil (296 questions) and criminal cases (292 questions), averaging 7.05 questions to civil respondents and 7.12 to criminal respondents. Justice Corrigan was the most active questioner in rebuttal arguments, asking 54 questions in civil cases on rebuttal (an average of 1.32 per case) and 49 in civil cases, or an average of 1.2 per case.

Overall, the most active questioners in civil cases were Justice Liu (495), Justice Cuéllar (491), Chief Justice Tani Cantil-Sakuaye (339), Justice Corrigan (334), Justice Leondra Kruger (311), Justice Ming Chin (308) and Justice Kathryn Werdegar (252 questions prior to her August 31 retirement). Pro-tem justices sitting in Justice Werdegar's place asked 42 questions.

On the criminal side, Justice Corrigan led (589), followed by Justice Liu (494), Chief Justice Cantil-Sakauye and Justice Werdegar (347 apiece), Justice Chin (286), Justice Kruger (204) and Justice Cuéllar (191). Pro-tems asked only 15 questions in criminal cases.

Who's Answering?

The average appellant in a civil case received 28.49 questions last year. The respondent received 31.81, and 8.61 questions were asked in the average rebuttal.

Criminal appellants averaged 28.96 questions in their initial presentation. Respondents received 29.14 questions and 4.41 questions were asked in criminal rebuttals.

Who's Winning?

The California Supreme Court is an interesting case study for oral argument analytics because the decision process is front-loaded. Although it's been debated just how much difference oral argument makes, everyone seems to agree that the court has what amounts to a tentative opinion by the time argument is scheduled. One might speculate that this would make justices less likely to use oral argument as an opportunity to debate with their colleagues.

In most types of cases last year, the party who got the most questions at oral argument typically lost. In civil affirmances, losing parties last year averaged 34.59 questions while winners received an average of 30.06 questions. In criminal affirmances, losing parties averaged 28.82 questions while winners averaged 24.95 questions.

The numbers were somewhat more equivocal in mixed-result cases where the court affirmed in part and reversed in part. In civil cases, appellants in partial reversals averaged 32.6 questions to 34.6 for respondents. In criminal cases, appellants in partial reversals averaged 31.5 questions to 26.75 questions for respondents.

The numbers are particularly intriguing when the court reversed. In civil cases, losing parties in reversals averaged 28.22 questions to 28.01 for winners. Criminal reversals were the only scenario in which losing parties averaged fewer questions -- 31.47 for losing parties, 34.13 for winners.

Tomorrow, in part two of this two-part series, we will dive into the data for the individual justices and take a look at what lessons we can learn from each justice's participation during oral arguments.