For my Criminologists - RCT shows that almost no interventions have lasting benefits

[WIA20XX]

Here's the paper

https://www.bu.edu/bulawreview/files/2023/12/STEVENSON.pdf

Here's the author's tweets

This paper surveys 50+ years of randomized control trials in criminal justice and shows that almost no interventions have lasting benefit -- and the ones that do don't replicate in other settings.

While this might be disappointing from the perspective of trying to engineer change, I argue it teaches us something important about the structure of the social world

At least when it comes to the type of limited-scope interventions evaluated via RCT, the social world is full of stabilizing forces that resist change. And causal processes are complex and context dependent, meaning success in one place will be hard to replicate in another

This has important implications for social change.

Systems-conserving interventions (those that change one factor while leaving the remaining social structure intact) are unlikely to bring about meaningful benefits

Those who seek social change have three options. 1. Focus on interventions with direct effects (i.e. feeding people) 2. Continue with systems-conserving change but give up expectations of large effects or the ability to identify effective interventions in advance

Embrace systemic reform with all it's uncertainty.

This paper is evidence-based; it is based on hundreds of randomized controlled trials. But the evidence rejects some of the central tenets of the evidence-based reform movement.

We are highly unlikely to identify effective interventions via RCT. We will not be able to pilot test before scaling up. There is not a set of proven interventions that can be widely adopted

Of course there are a lot of nuances about the extent and details of my claim. They are addressed in the paper. I hope you read it! It is the paper I am most proud of

Given that I found this on The Marginal Revolution, he's only posting it because he's a biased libertarian/conservative, but waiting for others to critique.

 

[Pressure]

On the surface the idea that criminals aren’t a monolith it makes sense.

Granted it’s only one part of criminal justice. That being removing bad folks from society is considered desirable even if there’s no reform.

 

[Professor Emeritus]

Like she says in the paper, one of the main things this shows is that the problems and their solutions are multifaceted and complex enough that you're not going to see massive (statistically) change reproducible across multiple populations with a single small intervention. Everyone in your sample is still stuck in the same fukked-up society even if you change one minor aspect of their experience.


I haven't read the whole article, but was skimming through and ran into this interesting paragraph.

One of the few interventions in which success has been replicated is city-sponsored summer job programs for teens. Numerous RCTs have found that summer employment reduces criminal justice involvement.145 Some of the studies suggest that effects persist past the summer of employment, implying that the effect is not entirely caused by keeping kids busy.146 This is an important finding: summer jobs programs are relatively easy to implement and scale, and reductions in crime and criminal justice involvement are meaningful benefits. But note that there is no evidence that this intervention leads to wholesale change in youth trajectories. Summer jobs do not appear to increase average wages or employment after completion of the program, nor do they increase educational outcomes.147 While some studies find impacts on arrests for violent crime, others find impacts only on drug or other minor offenses.148

That struck me because it seems like the sort of intervention that certain people here mock (I'm not thinking of a particular person, just the posts that mock community programs meant to keep kids in high-risk areas busy). But giving young people something productive and structured to do with their time at that critical age does appear to make a difference.

 

[Professor Emeritus]

Wow, read the whole paper and the conclusions are more radical than I expected.


She's saying straight up that it looks impossible for "moderate" policy to bring about change. Anyone advocating for policy that generally keeps the system in place while hunting for incremental "improvements" is working against the evidence. She's a little cautious about saying it outright, because it's impossible to test scientifically with RCT the same way incremental differences can be tested, but she very, very strongly implies that the only way to change anything for the better is via large-scale, systemic change.