Analysis of LA County pretrial risk assessment tools raises questions

An analysis of pretrial risk assessment scores given to defendants in Los Angeles County over the course of a year reveals a faulty system that could lead to more incarceration and more crime, according to experts in the field of pretrial detention.

As part of its ongoing investigation into algorithms being used in pretrial proceedings, a practice that will be codified under SB 10 should it be enacted in October 2019, the Daily Journal reviewed nearly 300 pages of records obtained through a California Public Records Act request showing the risk assessment scores for defendants arraigned in 2017.

Of the 11,240 scores assigned by the probation department to defendants who were arraigned that year, 73 percent fall into the high-risk category, a number some experts say shows the algorithm is not operating properly. In fact, they say, it is possible the county is using a post-conviction model in a pretrial setting, though the department denies this.

According to the probation department, which is tasked with the county's pretrial services, the current tool separates defendants into three categories in terms of their risk of committing a new crime, of being rearrested and/or of failing to appear in court.

Scores are based on a questionnaire arrestees are given by the department's pretrial services unit. Defendants with a score between zero and seven are considered low risk, between eight and 15 medium risk, and above 16 high risk.

Dan Dawson, head of communications for the probation department, said the county uses a "modified Wisconsin model" and it does not use any post-conviction tools "because Pre-Trial only serves pre-sentenced clients."

However, most counties in Wisconsin used a risk assessment tool designed by the Laura and John Arnold Foundation known as the Public Safety Assessment.

According to Marie VanNostrand, director of data analytics at Luminosity Inc. who works on the development and validation of pretrial risk assessment tools across the nation, the only state-developed risk assessment tool in Wisconsin was created by the Department of Corrections and was designed to predict the likelihood of post-conviction recidivism.

In addition, the three-tiered scale Los Angeles County uses and the scale defining those three categories is almost a carbon copy of the Wisconsin prison department's model, which varies by a single digit when defining its categories. (Low risk is scored between zero and eight, et cetera.)

The oversight could account for the high percentage of arrestees who received high-risk designations since a 2009 validity study of the Wisconsin model conducted by the Council of State Governments Justice Center found an "over-classification" of inmates who were considered high risk, specifically 93 percent.

Dawson said the Los Angeles County department hasn't completed any recent validation studies of its current tool. He wouldn't comment on the accuracy of the percentage determined by the Daily Journal's analysis but said that since the passage of Proposition 47, which changed drug possession from a felony to a misdemeanor, "there are now more people in custody who have committed more significant and violent crimes."

"The classification system is faulty based on what you're trying to accomplish," VanNostrand said. "The goal pretrial generally is to identify the highest risk defendants -- which should be reasonably small -- that are so high risk there's no legal standard to ensure court appearance or community safety."

"Seventy-three percent by any measure is unacceptable," she continued.

According to VanNostrand, most validated algorithms will categorize between 5 and 15 percent of defendants as high risk. Under the new law, SB 10, the county would be required to use a validated assessment tool. The state's Judicial Council has said it will compile a list of validated tools counties can choose from, though it has not released any details regarding which algorithms, if any, it has decided to include.

Superior Court Judge Lewis A. Davis of Contra Costa County, who offered a statement in his capacity as a board member for the Alliance of California Judges, said judges are aware many of the tools being used have not been adequately validated and, therefore, are hesitant to rely on them.

"The Alliance of California Judges does not take issue with the use of a score from a validated risk assessment tool as a factor in deciding whether to allow the pretrial release of an arrestee," Lewis wrote in an email. "We do, however, caution against basing release decisions exclusively, or even primarily, on these tools."

According to the language of SB 10, pretrial services would be authorized to release low-risk defendants and some medium-risk defendants with the intention of reducing the number of defendants detained before trial. When the algorithm skews high, however, fewer defendants are eligible for immediate release, which could mean detaining miscategorized defendants until a pre-arraignment or arraignment hearing, depending on the circumstances.

According to a 2013 Arnold Foundation-funded study, there are serious consequences in doing that. Looking at data from Kentucky, the study found in addition to the fiscal cost of keeping a defendant in custody, communities suffered a societal cost: Low-risk defendants, when detained for two to three days, were 40 percent more likely to commit new crimes before trial than equivalent defendants held no more than a day.

"It's a destabilization that has harmful effects in the short term and long term," said VanNostrand, who was one of the researchers for the study.

Recent legislative actions have also indicated a wariness about pretrial assessment algorithms. After being sworn in for a second term Monday, state Sen. Bob Hertzberg, D-Van Nuys, introduced a bill that would require the state to collect data from pretrial risk assessment tools in an effort to "prevent racial, gender, and economic disparities in pretrial release practices," according to a news release.

The Senate agreed Tuesday to move the bill forward.

Lewis said he hopes the state's newly elected governor, Gavin Newsom, would push for more comprehensive tools.

"We all want to strike the right balance between protecting the public and allowing arrestees to live and work in the community while awaiting trial," he wrote.

What the data shows:

A California Public Records Act request by the Daily Journal yielded the final disposition codes -- where they were applied -- for defendants arraigned in 2017 in Los Angeles County. An analysis of this subgroup, which consisted of nearly 700 entries, showed a judicial branch largely indifferent to the algorithmic suggestions.

Los Angeles County Superior Court public information officer Mary Hearn said the court does not comment on the rulings of its judicial officers. But, as indicated by the chart above, judges in Los Angeles County treat high-, medium- and low-risk defendants the same when it comes to sentencing, denying release or deferring judgment.

Superior Court Judge Lewis A. Davis of Contra Costa County, who offered a statement in his capacity as a board member for the Alliance of California Judges, said in addition to concerns about the use of invalidated tools, many judges in his organization are critical of significant information missing from many assessments. For example, many tools omit information about an arrestee's probation or parole status, which Davis said "could seriously understate the risk an arrestee poses to public safety."

He also said that unlike judges, algorithms inevitably fail to see the whole person.

"We don't believe that a limited check-the-box approach to risk assessment is an adequate substitute for in-person interviews of arrestees and consideration of their community ties -- a factor many assessment tools ignore or understate," Davis wrote in an email.

Under SB 10, judicial discretion is still the overriding factor in determining pretrial detention.