Yes, I’m not disparaging the practice (though I personally think those who do it are deceived). I’m simply making the point that it’s not a real factor in societal change (which is along the lines of the point the article was trying to make).
In most of the examples given the aim is to be seen to be doing something. It is cheaper and/or avoids unwanted change than doing something that actually works.
For example, lots of organisations will have diversity officers and training. Very few do things such as taking names, languages spoken (obviously you would leave any that are relevant to the job) and other clues to sex and race from job applications before shortlisting people for interview.
In the security realm we have security theater. In the social realm, we have compassion theater. "We are spending money on problem X" shows that the supporters of the spending care about people with problem X. Any proposed cut to that spending shows that the opponents to the spending hate people with with problem X and don't want to help them. Meanwhile, the money actually supports those in charge of the spending and various suppliers of goods and services.
I've seen this mentioned in the context of leadership. Someone said the best leaders get people to do things they wanted to do anyway. They just act as a kind of meeting point (or 'lightning rod', take your pick) that senses the general mood, and figures out a way forward that most of the group can agree with. Repeated (sometimes continuous) back-and-forth is important here.
In politics it's often the other way around: many people are unhappy, politician comes up with a plan to do something, and tries to push that through. Such action often fails as it -mostly- ignores feedback from the people the changes are imposed upon.
Acknowledging a problem is important. But more important is understanding the problem through the eyes of those affected by it, understanding how an action would change things from that point of view, and adjusting control knobs as you go. Few leaders are good at ALL those things.
I think giving all schoolchildren free food is self-evidently a good thing and "verifiable measure of success" requirements are just a way to manufacture BS justifications for taking food away from children from famously unreliable policy outcome studies.
One point raised in the article is that things that are easy (or possible) to measure aren't necessarily a good success criteria. And we'll choose an intervention based on affecting something that we suspect is easy to measure instead of what we think creates the actual outcomes we want. I would argue most of the important outcomes in life have no objective measure or proxy measure. I don't have a solution to that, just an observation.
> Plain packaging — where you strip cigarette boxes of their attractive branding — seems like a no brainer, but in Australia, it looks like the policy caused smokers to switch to cheaper brands and to smoke more
This example provides an easily understood illustration of why so many social programs fail. Simply put, the people proposing and creating the programs have zero understanding of the people whose lives they are trying to influence and control. Like generals ensconced in their offices located not even on the same continent as the front line they study maps and tables and hear the advice of experts with even less direct experience than themselves and then dispatch orders that arrive too late and attempt to command a reality that does not exist. The reports they receive back show nothing has changed or the situation has got worse and yet they never adapt or see their own fault. Why should they? They've suffered no consequence.
Maybe we should save the anger for the rich and powerful people selling addictive cancer rather than the ones who need to find some way of reducing their harm without getting sued or voted out.
But this policy has been very successful ... Every policy has some unwanted side effects but smoking overall has seen large reductions since its introduction.
If you read the report instead of a blog post with an agenda you will find out that the plain packaging accelerated decline in smoking by about 25% (p<0.01)
This really is not the message though. From the post:
“That’s why the worst possible reaction to a failed intervention is resignation. We may not know enough to improve our lot right now, but perhaps we could figure it out, if we tried.”
From reading the whole post, I gather that he’s truly not advocating some libertarian angle, he really is writing on the observation that policy is hard, and it’s nearly impossible to actually predict all the effects of societal intervention.
From my reading, I’m not sure even sure that he’s advocating caution or reluctance to social policy. Just goat sacrifices!
I think neither you nor the comment you are replying to are understanding what the article is trying to say.
It is saying, acts must be done to reaffirm the commitment to the goal, the goat is a litmus test as to whether the goal is worth it. Not to abandon the goal, some things are worth the sacrifice, but to be aware of it.
Wow. So we must not ask if any govt interventions work for fear that the wrong answer will empower political groups disfavoured by the govt and govt staff?
We should not cherry pick interventions that don't work and misleadingly show that interventions meant for one purpose did not work for another purpose in order to advance a narrative that no interventions work.
If you read more of the author's stuff it'll be obvious to you why this is wrong.
His whole thing is "we don't understand the world remotely as well as we think we do, and we need to be trying more diverse things (and reliably measuring the outcomes) so that we can understand it better"
The entire post seems to be along the lines of ‘no laws ever work!’ and then you scratch the surface and they do work.
There was a post here about Mississippi miracle https://en.wikipedia.org/wiki/Mississippi_miracle recently which was due to one of the programs mentioned in the original post claiming it produced no improvements.
As in the original post is flat put lying to push an agenda. You can google for like 2mins on many of the points and see counter evidence.
This seems like a super important corrective, but I don't see the agenda you're seeing. The article concludes by arguing we need to do lots more stuff, and suggests that people whose takeaway from the article is "stop doing social science interventions" should take the place of the goats in the sacrifices.
If true, it would somewhat undermine another point of the article, which is that since almost all of our interventions are ineffective we need to invest more in understanding which ones actually work.
I'm sure that many other examples could be found to replace this one, but if a lot of the "ineffective interventions" turned out to be effective under closer scrutiny then it would be a blow to the thesis for sure.
But the paper cited by the blog post also covers the smoking prevalence data and finds it to not be statistically significant. Now you can look at their model and decide if you find that conclusion convincing, but I don't think just pulling a number from a different report (especially one from the Australian government, which has its own agenda in motivating its own policy) is really such a gotcha here.
To be clear, I don't want to play the game of paper vs. paper. I simply think there is more depth of detail here than can be nicely summarised in a comment.
Imagine if they had been able to take proper measures decades ago, bankrupting the tobacco companies and imprisoning their executives for covering up the damaging effects of their products and engaging in a massive campaign of false advertising. Maybe that would have moved the needle.
It is not a surprise that when people are given virtually no power to do anything, they have a hard time accomplishing very much.
The irony here of course is that most comments on HN - including this one and the one it responds to - are arm-chair commentary on generals' actions, even less informed, even more disconnected.
Unintended consequences are inevitable. The danger isn't that they'll exist - it's that they'll be ignored entirely or ignored until they cannot be, that the desire for panacea produces a willful blindness to their existence or myopic fixation on their effects to the point where the value of the intended consequences are ignored.
Another relevant example here is vaping: an amazing win for harm reduction in existing smokers, but an attractive on-ramp to the addiction for some who would have avoided it otherwise. No panacea.
Smoking cigarettes has an incredibly massive public health cost. Addiction to nicotine makes people wring their hands, but population wide doesn't have a significant cost and might even be a net benefit. Vaping likewise has an incomparably minimal public health cost, though at the margins there's probably more injury than nicotine use alone.
Yes, usually there are unintended consequences. The most important unintended consequence here was the anti-vaping sentiment engendered by the anti-smoking campaign choosing to conflate nicotine with the harms from inhaling combustion byproducts and tacitly embracing a cateforical, puritanical morality regarding substance use.
"Findings were mixed amongst the four studies assessing consumption, with some studies finding no difference and some studies finding evidence of a decrease."
This also ignores that the intervention was intended to reduce the number of people taking up smoking and to encourage current smokers to quit, which appears to have happened according to that same review.
> In response to the policy, smokers switched from more expensive to cheaper cigarettes and reduced their overall tobacco expenditure and expenditure intensity. However, as smoking became less costly, smokers consumed more cigarettes.
And yes, even if that's true, it's separate from the question of how effective it was as stopping nonsmokers from starting.
Software engineering, for all its faults as a field, figured this out a long time ago. The essence of agile (the good parts) is: the people doing the work understand it best, and the plan is wrong (and making the plan is a waste of time).
Most implementations of Agile that I've seen do keep the plan, but the plan is allowed and even encouraged to change. High level planning still provides valuable insight, especially from a business perspective which is the whole point of the majority of systems.
>Simply put, the people proposing and creating the programs have zero understanding of the people whose lives they are trying to influence and control.
I mean I think that's definitely true in some cases but it's a wide wide world out there, and the universe of things that count as "social programs" is impossibly vast, and some of them are quite well researched and simply work when rolled out. Congestion pricing in NYC for instance was the product of policy nerds and seems like it simply worked.
I completely agree there's unintended consequences and regulation can make things worse but not to the point of a universal anti government ethos.
I think some "interventions" may also be the result of political compromises or "deals" where one side might want the intervention to be ineffective.
E.g. if the tobacco industry is forced to choose between either hard bans on smoking, or a highly visible but largely symbolic measure like changing the packaging, they will probably choose the latter.
Want to cut a social program because you don't want to pay for it, but don't want to look overtly evil? First, shoot out the kneecaps. You're making it more efficient by cutting weight! Second, notice that it can't walk anymore and cut it entirely. We need something better! Optional third step: give a high-minded speech about how attempts to solve problems are doomed so we shouldn't even try, or shouldn't try without an infinite amount of meetings first.
Take the taxes off cigarettes. Attach them to a tax stamp which has to be purchased separately and redeemed upon price of cigarettes. Make it a hassle on top of more expensive.
WASHINGTON (AP) Recent reports indicate the federal goat sacrifice program has failed to produce a meaningful improvement in the success rate of public policy over the past five years.
The program, which costs taxpayers over $1 billion annually at a minimum of $1 million per goat, has also had little impact on out of control spending in unrelated government programs, a finding that reportedly surprised the bill's original cosponsors.
Advocacy groups have condemned the practice as inhumane. Proponents, meanwhile, blame rising costs on the $300 million spent each year on therapy for all individuals involved. Critics are now questioning how much of that funding actually goes toward mental health care, and whether more attention should be paid to the care of the goat prior to slaughter.
> One of my favorite genres of psychology experiment is called something like: “the study is probably bullshit but the effect is probably real”. At the top of that heap is The Illusion of Control, a classic from 1975 and a favorite of Psych 101 classes everywhere. Usually, this study gets summarized as “people value their lottery tickets more when you let them pick the numbers vs. when you pick the numbers for them.” It’s as if people think that the mere act of picking imbues their ticket with a bit of juju, which makes them less willing to part with it.
> That would be a beautiful study, but it wasn’t the study that was actually conducted. The “lottery tickets” in question were in fact football cards, each one with a picture of a player, their team, etc. Some participants were given the opportunity to choose their football card and some had a card handed to them. Later, just before the drawing, all of the participants were asked how much it would take to buy their card back. On average, the pickers asked for $8.67, while the not-pickers said $1.86.4
> I’m sure you can see the obvious and less exciting explanation for these results: when people were given the chance, they picked their favorite players. The people who didn’t get to pick ended up with a card they liked less. This “illusion of control” effect is merely a “reality of people having preferences and acting on them” effect.5
Alternatively, people who are knowledgeable of a subject will have selfishly biased opinions about it, which they value more than unknowledgeable people will value an arbitrary stance on the subject. From experience, this seems to affect things beyond situations of chance like lotteries, and infests collaborations like businesses in ways such as Not Invented Here Syndrome or generally poor management choices that fail to consider situational factors (e.g. overarching decrees that are ineffective in certain situations/locations/applications).
>Alternatively, people who are knowledgeable of a subject will have selfishly biased opinions about it, which they value more than unknowledgeable people will value an arbitrary stance on the subject. From experience, this seems to affect things beyond situations of chance like lotteries, and infests collaborations like businesses in ways such as Not Invented Here Syndrome or generally poor management choices that fail to consider situational factors (e.g. overarching decrees that are ineffective in certain situations/locations/applications).
And the ignorant are always the loudest.
The fresh minted safety guy with his shiny new clipboard and vest is a fool running around the plant spewing rabid idiocy straight out of the internet comment section.
His boss who's been doing things for a decade is more measured, knows you can't tell a guy how to use a screwdriver if you also want to tell him how to use a ladder
A lot of lottery players will say use a birthday or something. Kind of interesting to think about this as deciding on pure random search or setting a seed and sampling that seed over the space. The behavior becomes much less about juju and maybe more an actual functional one with some biological utility, this ability to sample a defined input across a function set. The lottery aspect is almost blinding the point of why it might be useful to think like that.
About (dear lord) 30 years ago when the national lottery came to the shores of Britain, my family had the neighbours over and there was some faux hand holding "seance/trance" make-pretend, with us spitting out numbers for the very first lottery ticket. We won a tenner!
Silliness of the highest order. I still have those numbers written down. They've got the same chance as any other set - but this set are _mine_. I might just be the only person alive who was there, now I reflect upon it. Huh
> A lot of lottery players will say use a birthday or something.
Ironically, this makes the ticket less valuable.
You can't do anything to change the probability of winning, but you can change the probability of winning at the same time as other players (which affects the size of the winnings). By picking birthdays (i.e. forcing the numbers to be in the first 30 or so), your prize, if you do win, is likely to be smaller than a really random selection.
> There ought to be an entire liturgy of grotesqueries, in fact—pricked fingers, names written in blood, robes, candles, chanting, obsidian blades, still-beating hearts, the whole shebang.
The idea that such ceremony induces humility may be true in a limited individual sense, in that a person may come to believe that they are nothing but their deity is everything. But at the same time it elevates the status of whoever is the current puppet master of their deity. The end result is net uptick in hubris, on behalf of their lord, whose relayed word is sufficient without any need to resort to such crudities as science or debate.
> The point of this ritual is not to signal our obeisance to the gods (although if any of them want to stop by and help us out, that would be cool), but to remind ourselves that we know nothing. This is procedurally-induced humility, which is desperately necessary, because an alarming number of our best-intentioned interventions do not work.
> For example:
> In 2009, Congress spent $3 billion on “School Improvement Grants” for the worst-performing schools in the country. As far as we can tell, those grants did nothing.
> Universities that hired a Chief Diversity Officer between 2001 and 2019 did not end up with more diverse students, faculties, or administrators.
> The American government spent $280 million to help Afghan women get jobs, and now they can’t really say whether any women got any jobs.1
> Remember 20 years ago, when giving laptops to kids in developing countries seemed like the most important thing to do? Well, a 10-year RCT of 531 rural schools in Peru found zero effects of “One Laptop Per Child” on academic performance.2
> Researchers developed four different messages that were supposed to convince parents to vaccinate their kids. None of the messages worked, and one of them accidentally made parents more likely to believe that vaccines cause autism.
Something that needs addressing on all but the laptop one: there are parties who had a vested interest in seeing these example programs fail. In order:
* people who think public schools exist only as instruments of mass indoctrination for secular values and who would rather not have their money go to that, and/or would like to be the investors in "market-based" solutions to America's supposed education problems
* people who think that anyone and everyone who isn't white, male, middle-class-or-above, and Judeo-Christian gets into college programs because the government forced them in there, and find other ways to get those people to leave, because they see them as a threat to their freedom to do and say whatever they want in the spaces of higher education, and a threat to their sense of intelligence and superiority
* the Taliban. I hope I don't need to explain that one further.
* medical grifters who can't push you things like bleach enemas and colloidal silver if you aren't sick because you got vaccinated, among others
I can't speak to what happened in Peru, but these are all examples where you can easily point to people in American or Afghan society participating in the efforts in bad faith, and thus sabotaging them.
Make graffiti classes mandatory in school. Ensure they are taught by gym teachers with no discernible artistic interest or talent. Subsidize a Graffiti text book industry. Create schools of approved graffiti. Launch a "America's Got Graffiti" television show with celebrity randos.
I had a related idea about vaccination back during COVID. Have Hillary Clinton be quoted saying nobody should be vaccinated and the "Trump vaccine" is bogus. Then have Trump be quoted saying nobody should be vaccinated and the "fake Democrat vaccine" is bogus. Result: everyone gets vaccinated.
A negative version of this, I think, is mandatory diversity education and DEI, which had the effect of making racism and fascism cool and countercultural among the youth. Deplatforming had the same effect. "It's being banned which means it's what THEY don't want you to know!" The big wave of deplatforming in 2019 and 2020 probably helped Trump win in 2024 by way of the Streisand effect.
I think with vaccines it's more that certain people already don't like the idea and are looking for any hint of legitimacy to justify their decision not to vaccinate. Regular people use data to justify decisions they've already made, not the other way around. On the "party I don't support" angle: if I think it's 3 PM and look at a broken clock and see that it says 3 PM, that doesn't move me away from my suspicion that it's 3 PM.
> It’s inevitable that some of our interventions are going to fail. (If they don’t, we should either be suspicious of the results, or we should be more ambitious with our programs.) What we really want to know is: when we try to improve the world through some policy or program, how often do we succeed?
Looking at policy-making it seems we routinely mistake having a good reason to want something to happen for having good evidence that our proposed intervention will actually make it happen.
Perhaps the crucial missing ingredient is a requirement that policies contain their own mechanisms for being proved wrong?
Interestingly, we do have quite a lot of material and studies on how to change human behavior and a whole lot of data supporting it: it's called marketing. I think this goes to show that when the entire economy is working on a problem we actually can make an impact, it's just unfortunately all going towards projects that make people poorer and more depressed.
There are limits to that as well though. The USSR put an enormous amount of resources towards creating the “New Soviet Man”, including education, “re-education”, films, literature, psychology, and even biology (eugenics).
You have to make sure your end goal is realistic and attainable. The Soviet example is extreme, but sometimes even changes that look deceptively simple aren’t realistic due to deeply embedded social issues or plain human nature.
I think it's quite funny how the "unintended consequences" people on the one hand love to highlight every unintended consequence or ineffective intervention, but also have zero interest in finding out why the interventions did not work out as planned. Wouldn't this be a great starting point for scientific research?
If I'm investigating a bug and find that the program is still misbehaving after my fix, my usual reaction is to dig deeper, find out what I missed or how my fix changed the program's execution. My reaction would not be "eh, better not fix anything at all".
> Most of those are left-coded failures, so here are some right-coded failures:
"I'm neither left nor right, I'm just an unpolitical neoliberal..."
Isn't trying a plausible sociological intervention then measuring its effectiveness a win regardless of how effective it ended up being? Either way, we know more about what's likely to work for future interventions. It's the scientific method.
These interventions target very complex systems. It's not unexpected that we need to try a bunch of things that don't work before we find something that does, to the extent that most of the things we try are ineffective, right?
No man, don't you know if you can't guarantee success or if you ever swing and miss you should just give up. If any single one of your plans fail, seppuku and self immolation are the only option. Successes must be ignored and not even mentioned.
I read the whole article, and in the beginning part the only thing I felt the author wanting to change was my own mind. He is drilling in the idea that common sense interventions more often than not do not work. (maybe even almost always don't work).
His proposed change at the end is that we must always guess and check (emphasis on check) for now, and we may eventually get into a "Psychohistory" level of understanding in the future, but we are far from it now.
> Isn't trying a plausible sociological intervention then measuring its effectiveness a win regardless of how effective it ended up being?
Only if the value of whatever smidgen of knowledge you gain (assuming you do actually gain any) outweighs the much-more-than-smidgen cost of the intervention to all the people who weren't helped, and in a significant number of cases were harmed, by it.
As you can see, I don't think whatever benefits are gained actually do outweigh the costs. Which means that no, continuing to try such interventions is not a win.
The lesson we should be learning from all this is to stop trying to do so many things top down. That whenever you see some kind of large scale statistic that you think is not what it should be, the last thing you should do is concoct a large scale intervention to try to change it. Or even concoct a small scale experiment to "test" the idea, with the intention of scaling it up if it seems to work. Still less should you sacrifice a poor goat who did nothing to deserve it, just to underscore the point that your intervention won't work.
That has to carry with it the assumption that the response to similar interventions is long lasting over time. I'm shooting from the hip here, but I'd expect culture, life experiences, etc to play a role in how interventions would play out.
If my hunch were to price right, anything we'd learn there would have a short half live and may not hold out past, say, a generation or two.
I can say anecdotally that my grandparents would respond differently to many situations and interventions than I would. They grew up during the Great Depression and he flew in the war, those two events alone very much impacted how they viewed things for the rest of their life.
Looking at those only a generation behind me and even they seem to respond very differently than I'd expect those in my generation would.
In this context that would mean running the same intervention at scale 30 times without each run impacting the next. Maybe that is possible, but it seems unlikely to me especially if the intervention has a half life on the order of a single generation as life experience and social norms change.
The question I was raising was locality though, it was temporal. Such experiments take time to conduct, and running a couple dozen or more times takes even more time.
The question I was raising was whether said results would be useful beyond a decade or two into the future, not across different cities today.
Then you're still misreading my point. Such tests take a recent amount of time to design, get approved, execute, and analyze.
Say it takes a year even when 30 runs are done in parallel while keeping proper control over population distributions, variables and controls, etc. If the results are only useful on the timeline of a generation or two, it would only make sense to run such studies on events pretty likely to happen.
Testing interventions of a hundred year pandemic event wouldn't be useful if the results became invalid after a generation, for example. Unless you're willing to run such studies every generation knowing that roughly 1:5 would be useful. That'd depend on how expensive and invasive the intervention study is in the first place.
I think so. But a big part of the article's complaint is that we usually don't even bother checking.
The reason the article resonated with me is because policies and interventions are often sold as "this thing is self-evidently great, and if you oppose it you're an idiot/jerk", rather than "this seems promising, so let's give it a try and scale it up if it works well". When I look at proposed measures at my local government level (for example) almost all of the supporting literature focuses on why the problem is important, rather than why the proposed interventions are likely to succeed. Being "effective" as a legislator means enacting policy, so if you want to be effective you try to keep the conversation on the first part, where it's easier to find agreement.
It's quite interesting to appreciate that these issues were observed, and written about, some 2400 years ago. Times when ostensibly everything would have been practically alien. Yet it emphasizes that the thing that's really changing is not humanity, so much as it is technology. We're still trying, and failing, to deal with the exact same issues as we were thousands of years ago.
And mind you that's just the first time it was written down. I wouldn't doubt that these observations and issues long predate even Plato. Because I think one thing that's become clear is that he wasn't necessarily unimaginably brilliant so much as he was simply capable of objectively seeing the patterns in society and recording them. Though perhaps that is some form of brilliance in its own right.
I agree, but we should be more systematic about it. Instead of one intervention, we should be asking what are our ten best ideas for interventions and try all of them on a small scale in different places.
I think most sociological interventions don't follow the scientific method because you need a control group. You can't just use random people you haven't helped elsewhere as a control, because you need to identify all of the external variables that you can so as to properly assess the effectiveness of your method (ruling out changes external to your intervention).
There aren't too many people comfortable with intentionally letting others suffer for the sake of the scientific method, and those who do have a hard time getting past ethics review boards.
Instead we have this reality where we throw money at problems and claim credit whenever something good happens, and claim we did good work when nothign changes.
That's a good point even thought it would be so much more efficient.
> There aren't too many people comfortable with intentionally letting others suffer for the sake of the scientific method, and those who do have a hard time getting past ethics review boards.
The way I see it, it is a mindset problem rather than an ethical one.
Because policies are considered de facto as "improving things", it makes sense that not applying it would be unethical. However, the article makes a good point that this assumption is clearly flawed.
If the mindset would be more along the lines of "there is a bad situation, we think we know how to improve it but we are not 100% sure", then it would not sound so unethical to form control groups.
For instance, using the "laptop for every child" case the governement could have instead picked a few schools that would receive school laptops with the intention to measure the effectiveness of the policy.
If it had a positive impact, then you can proceed with it and buy a laptop for every student. It seems to me more logical and also less wasteful to proceed that way.
I agree, personally- I have a gut rejection of the attitude of "we have to do something so let's do anything just in case it helps".
In the case of laptops for every child, though, people argued that they were redressing an existing imbalance between kids whose parent could afford Internet and home laptops and those who couldn't.
They really should have trialled introducing them to classrooms and observing the impacts and so on- as you say, this is low stakes enough to not be an ethical issue.
The problem in this specific case is the outcome was entirely secondary to two external objectives; selling laptops and a misguided sense of social justice.
Lots of examples in this article, but none of them go deep enough to matter. Just like people bringing up the cigarette issue, I see other factors in sone of the other topics, such as medical school (residency slots weren't mentioned, default assumption was motivators were solely economic, etc).
I'm not sure what to even discuss about this article due to the low quality. I guess there overall point is that policies and interventions don't necessarily scale, or perform the same across cultures/cohorts, or even accomplish their intended goals. I would largely agree with those points, but not due to the way the article got there.
ive never had a presentation go well if i didnt place a "sacrificial slide" in the deck, or plan an "oops here is a bug, let me recover, and we're back on track" during the demo
One very successful social intervention in the UK was the push to reduce drink-driving. Starting around the 80s I think, maybe earlier. There was a big media campaign with TV ads as well as implementing and enforcing stricter laws. What I find interesting is that many in the older generation still found it more acceptable to have a few drinks and drive, whereas those growing up during this period largely find it very socially unacceptable.
One feature of this intervention was that it directly coerced the behavior that was their desired outcome, with the threat of stricter enforcement.
Whereas many of the interventions mentioned in the article were seemingly logical but indirect: If we do A, it will cause people to behave B, resulting in C.
The article talks about measurement of impact as though it's a totally value-neutral idea, but it's absolutely not:
* Measurement requires substantial logistical overhead; both up-front in creating a plan for measurement, and during intervention due to the extra record keeping involved. This is a (probably frequently large) drain on program efficacy. Just listen to any schoolteacher's take on standardized testing, evaluation rubrics, etc.
* Some contexts, and some types of intervention are more amenable to measurement than others. Insisting that nothing which is not measured can be valuable biases our decision-making towards the most measurable actions, which are likely not the best. Strong correlation with simplistic actions, short timelines, and lack of responsiveness to individual context.
* Goodhart's law.
Author (and everyone) would benefit from the book Seeing Like a State.
tough article. 90% is a tongue in cheek ridicule, with lots of data points, and then the bait and switch at the end, with far less argumentation.
mainly, interventions are experiments, and therefore we need more of them, you never know. but they are not experiments, otherwise we wouldn't repeat failed ones over and over, or let existing failing ones run on and on.
maybe the answer really is more simple, that most programs are indeed politically motivated endeavors, indeed largely signals of a worldview, less programs to bring about material improvement.
fwiw - i dont know the answer, but having your cake and eat it, being perceptive of the absurdity, yet, not daring to call it out, is an author's disposition, less a truth. so maybe, if we do the one hand other hand game, accept the bleak truth glimpsed in the facts, but also accept that there's nothing to be done about it within existing frameworks, and not point at it as something desirable.
130 comments
[ 3.0 ms ] story [ 54.4 ms ] threadFor example, lots of organisations will have diversity officers and training. Very few do things such as taking names, languages spoken (obviously you would leave any that are relevant to the job) and other clues to sex and race from job applications before shortlisting people for interview.
A signal of an act. Not the act itself. Post l-modernisms ugly head rears itself once again.
In the security realm we have security theater. In the social realm, we have compassion theater. "We are spending money on problem X" shows that the supporters of the spending care about people with problem X. Any proposed cut to that spending shows that the opponents to the spending hate people with with problem X and don't want to help them. Meanwhile, the money actually supports those in charge of the spending and various suppliers of goods and services.
I've seen this mentioned in the context of leadership. Someone said the best leaders get people to do things they wanted to do anyway. They just act as a kind of meeting point (or 'lightning rod', take your pick) that senses the general mood, and figures out a way forward that most of the group can agree with. Repeated (sometimes continuous) back-and-forth is important here.
In politics it's often the other way around: many people are unhappy, politician comes up with a plan to do something, and tries to push that through. Such action often fails as it -mostly- ignores feedback from the people the changes are imposed upon.
Acknowledging a problem is important. But more important is understanding the problem through the eyes of those affected by it, understanding how an action would change things from that point of view, and adjusting control knobs as you go. Few leaders are good at ALL those things.
But as software people we all know that turning a measure into a target yields problems.
This example provides an easily understood illustration of why so many social programs fail. Simply put, the people proposing and creating the programs have zero understanding of the people whose lives they are trying to influence and control. Like generals ensconced in their offices located not even on the same continent as the front line they study maps and tables and hear the advice of experts with even less direct experience than themselves and then dispatch orders that arrive too late and attempt to command a reality that does not exist. The reports they receive back show nothing has changed or the situation has got worse and yet they never adapt or see their own fault. Why should they? They've suffered no consequence.
https://www.health.gov.au/sites/default/files/study-of-the-i... (page 12)
They also tried quitting more often (p=0.013)
https://pmc.ncbi.nlm.nih.gov/articles/PMC3710988/
“That’s why the worst possible reaction to a failed intervention is resignation. We may not know enough to improve our lot right now, but perhaps we could figure it out, if we tried.”
From reading the whole post, I gather that he’s truly not advocating some libertarian angle, he really is writing on the observation that policy is hard, and it’s nearly impossible to actually predict all the effects of societal intervention.
From my reading, I’m not sure even sure that he’s advocating caution or reluctance to social policy. Just goat sacrifices!
It is saying, acts must be done to reaffirm the commitment to the goal, the goat is a litmus test as to whether the goal is worth it. Not to abandon the goal, some things are worth the sacrifice, but to be aware of it.
His whole thing is "we don't understand the world remotely as well as we think we do, and we need to be trying more diverse things (and reliably measuring the outcomes) so that we can understand it better"
There was a post here about Mississippi miracle https://en.wikipedia.org/wiki/Mississippi_miracle recently which was due to one of the programs mentioned in the original post claiming it produced no improvements.
As in the original post is flat put lying to push an agenda. You can google for like 2mins on many of the points and see counter evidence.
I'm sure that many other examples could be found to replace this one, but if a lot of the "ineffective interventions" turned out to be effective under closer scrutiny then it would be a blow to the thesis for sure.
To be clear, I don't want to play the game of paper vs. paper. I simply think there is more depth of detail here than can be nicely summarised in a comment.
You can still try and revert back if it fails.
It is not a surprise that when people are given virtually no power to do anything, they have a hard time accomplishing very much.
Unintended consequences are inevitable. The danger isn't that they'll exist - it's that they'll be ignored entirely or ignored until they cannot be, that the desire for panacea produces a willful blindness to their existence or myopic fixation on their effects to the point where the value of the intended consequences are ignored.
Another relevant example here is vaping: an amazing win for harm reduction in existing smokers, but an attractive on-ramp to the addiction for some who would have avoided it otherwise. No panacea.
Yes, usually there are unintended consequences. The most important unintended consequence here was the anti-vaping sentiment engendered by the anti-smoking campaign choosing to conflate nicotine with the harms from inhaling combustion byproducts and tacitly embracing a cateforical, puritanical morality regarding substance use.
This is documented, but it is not clear how much this is due to packaging vs. Australia having the highest tobacco tax in the world.
> and to smoke more
{{Citation needed}}
This claim is directly contradicted by this Cochrane review. https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD...
"Findings were mixed amongst the four studies assessing consumption, with some studies finding no difference and some studies finding evidence of a decrease."
This also ignores that the intervention was intended to reduce the number of people taking up smoking and to encourage current smokers to quit, which appears to have happened according to that same review.
> In response to the policy, smokers switched from more expensive to cheaper cigarettes and reduced their overall tobacco expenditure and expenditure intensity. However, as smoking became less costly, smokers consumed more cigarettes.
And yes, even if that's true, it's separate from the question of how effective it was as stopping nonsmokers from starting.
I mean I think that's definitely true in some cases but it's a wide wide world out there, and the universe of things that count as "social programs" is impossibly vast, and some of them are quite well researched and simply work when rolled out. Congestion pricing in NYC for instance was the product of policy nerds and seems like it simply worked.
I completely agree there's unintended consequences and regulation can make things worse but not to the point of a universal anti government ethos.
E.g. if the tobacco industry is forced to choose between either hard bans on smoking, or a highly visible but largely symbolic measure like changing the packaging, they will probably choose the latter.
Want to cut a social program because you don't want to pay for it, but don't want to look overtly evil? First, shoot out the kneecaps. You're making it more efficient by cutting weight! Second, notice that it can't walk anymore and cut it entirely. We need something better! Optional third step: give a high-minded speech about how attempts to solve problems are doomed so we shouldn't even try, or shouldn't try without an infinite amount of meetings first.
This happens a lot.
WASHINGTON (AP) Recent reports indicate the federal goat sacrifice program has failed to produce a meaningful improvement in the success rate of public policy over the past five years.
The program, which costs taxpayers over $1 billion annually at a minimum of $1 million per goat, has also had little impact on out of control spending in unrelated government programs, a finding that reportedly surprised the bill's original cosponsors.
Advocacy groups have condemned the practice as inhumane. Proponents, meanwhile, blame rising costs on the $300 million spent each year on therapy for all individuals involved. Critics are now questioning how much of that funding actually goes toward mental health care, and whether more attention should be paid to the care of the goat prior to slaughter.
> That would be a beautiful study, but it wasn’t the study that was actually conducted. The “lottery tickets” in question were in fact football cards, each one with a picture of a player, their team, etc. Some participants were given the opportunity to choose their football card and some had a card handed to them. Later, just before the drawing, all of the participants were asked how much it would take to buy their card back. On average, the pickers asked for $8.67, while the not-pickers said $1.86.4
> I’m sure you can see the obvious and less exciting explanation for these results: when people were given the chance, they picked their favorite players. The people who didn’t get to pick ended up with a card they liked less. This “illusion of control” effect is merely a “reality of people having preferences and acting on them” effect.5
Alternatively, people who are knowledgeable of a subject will have selfishly biased opinions about it, which they value more than unknowledgeable people will value an arbitrary stance on the subject. From experience, this seems to affect things beyond situations of chance like lotteries, and infests collaborations like businesses in ways such as Not Invented Here Syndrome or generally poor management choices that fail to consider situational factors (e.g. overarching decrees that are ineffective in certain situations/locations/applications).
And the ignorant are always the loudest.
The fresh minted safety guy with his shiny new clipboard and vest is a fool running around the plant spewing rabid idiocy straight out of the internet comment section.
His boss who's been doing things for a decade is more measured, knows you can't tell a guy how to use a screwdriver if you also want to tell him how to use a ladder
Silliness of the highest order. I still have those numbers written down. They've got the same chance as any other set - but this set are _mine_. I might just be the only person alive who was there, now I reflect upon it. Huh
Ironically, this makes the ticket less valuable.
You can't do anything to change the probability of winning, but you can change the probability of winning at the same time as other players (which affects the size of the winnings). By picking birthdays (i.e. forcing the numbers to be in the first 30 or so), your prize, if you do win, is likely to be smaller than a really random selection.
The idea that such ceremony induces humility may be true in a limited individual sense, in that a person may come to believe that they are nothing but their deity is everything. But at the same time it elevates the status of whoever is the current puppet master of their deity. The end result is net uptick in hubris, on behalf of their lord, whose relayed word is sufficient without any need to resort to such crudities as science or debate.
> For example:
> In 2009, Congress spent $3 billion on “School Improvement Grants” for the worst-performing schools in the country. As far as we can tell, those grants did nothing.
> Universities that hired a Chief Diversity Officer between 2001 and 2019 did not end up with more diverse students, faculties, or administrators.
> The American government spent $280 million to help Afghan women get jobs, and now they can’t really say whether any women got any jobs.1
> Remember 20 years ago, when giving laptops to kids in developing countries seemed like the most important thing to do? Well, a 10-year RCT of 531 rural schools in Peru found zero effects of “One Laptop Per Child” on academic performance.2
> Researchers developed four different messages that were supposed to convince parents to vaccinate their kids. None of the messages worked, and one of them accidentally made parents more likely to believe that vaccines cause autism.
Something that needs addressing on all but the laptop one: there are parties who had a vested interest in seeing these example programs fail. In order:
* people who think public schools exist only as instruments of mass indoctrination for secular values and who would rather not have their money go to that, and/or would like to be the investors in "market-based" solutions to America's supposed education problems
* people who think that anyone and everyone who isn't white, male, middle-class-or-above, and Judeo-Christian gets into college programs because the government forced them in there, and find other ways to get those people to leave, because they see them as a threat to their freedom to do and say whatever they want in the spaces of higher education, and a threat to their sense of intelligence and superiority
* the Taliban. I hope I don't need to explain that one further.
* medical grifters who can't push you things like bleach enemas and colloidal silver if you aren't sick because you got vaccinated, among others
I can't speak to what happened in Peru, but these are all examples where you can easily point to people in American or Afghan society participating in the efforts in bad faith, and thus sabotaging them.
Solution:
Make graffiti classes mandatory in school. Ensure they are taught by gym teachers with no discernible artistic interest or talent. Subsidize a Graffiti text book industry. Create schools of approved graffiti. Launch a "America's Got Graffiti" television show with celebrity randos.
Graffiti will be dead within ten years. :-)
A negative version of this, I think, is mandatory diversity education and DEI, which had the effect of making racism and fascism cool and countercultural among the youth. Deplatforming had the same effect. "It's being banned which means it's what THEY don't want you to know!" The big wave of deplatforming in 2019 and 2020 probably helped Trump win in 2024 by way of the Streisand effect.
Looking at policy-making it seems we routinely mistake having a good reason to want something to happen for having good evidence that our proposed intervention will actually make it happen.
Perhaps the crucial missing ingredient is a requirement that policies contain their own mechanisms for being proved wrong?
You have to make sure your end goal is realistic and attainable. The Soviet example is extreme, but sometimes even changes that look deceptively simple aren’t realistic due to deeply embedded social issues or plain human nature.
If I'm investigating a bug and find that the program is still misbehaving after my fix, my usual reaction is to dig deeper, find out what I missed or how my fix changed the program's execution. My reaction would not be "eh, better not fix anything at all".
> Most of those are left-coded failures, so here are some right-coded failures:
"I'm neither left nor right, I'm just an unpolitical neoliberal..."
These interventions target very complex systems. It's not unexpected that we need to try a bunch of things that don't work before we find something that does, to the extent that most of the things we try are ineffective, right?
His proposed change at the end is that we must always guess and check (emphasis on check) for now, and we may eventually get into a "Psychohistory" level of understanding in the future, but we are far from it now.
Only if the value of whatever smidgen of knowledge you gain (assuming you do actually gain any) outweighs the much-more-than-smidgen cost of the intervention to all the people who weren't helped, and in a significant number of cases were harmed, by it.
As you can see, I don't think whatever benefits are gained actually do outweigh the costs. Which means that no, continuing to try such interventions is not a win.
The lesson we should be learning from all this is to stop trying to do so many things top down. That whenever you see some kind of large scale statistic that you think is not what it should be, the last thing you should do is concoct a large scale intervention to try to change it. Or even concoct a small scale experiment to "test" the idea, with the intention of scaling it up if it seems to work. Still less should you sacrifice a poor goat who did nothing to deserve it, just to underscore the point that your intervention won't work.
If my hunch were to price right, anything we'd learn there would have a short half live and may not hold out past, say, a generation or two.
I can say anecdotally that my grandparents would respond differently to many situations and interventions than I would. They grew up during the Great Depression and he flew in the war, those two events alone very much impacted how they viewed things for the rest of their life.
Looking at those only a generation behind me and even they seem to respond very differently than I'd expect those in my generation would.
The question I was raising was whether said results would be useful beyond a decade or two into the future, not across different cities today.
Locality means that 30 runs does not require 30 sequential runs.
Say it takes a year even when 30 runs are done in parallel while keeping proper control over population distributions, variables and controls, etc. If the results are only useful on the timeline of a generation or two, it would only make sense to run such studies on events pretty likely to happen.
Testing interventions of a hundred year pandemic event wouldn't be useful if the results became invalid after a generation, for example. Unless you're willing to run such studies every generation knowing that roughly 1:5 would be useful. That'd depend on how expensive and invasive the intervention study is in the first place.
No, paying $3B for information of dubious value and which was pretty obvious ahead of time is not a win.
The reason the article resonated with me is because policies and interventions are often sold as "this thing is self-evidently great, and if you oppose it you're an idiot/jerk", rather than "this seems promising, so let's give it a try and scale it up if it works well". When I look at proposed measures at my local government level (for example) almost all of the supporting literature focuses on why the problem is important, rather than why the proposed interventions are likely to succeed. Being "effective" as a legislator means enacting policy, so if you want to be effective you try to keep the conversation on the first part, where it's easier to find agreement.
It's quite interesting to appreciate that these issues were observed, and written about, some 2400 years ago. Times when ostensibly everything would have been practically alien. Yet it emphasizes that the thing that's really changing is not humanity, so much as it is technology. We're still trying, and failing, to deal with the exact same issues as we were thousands of years ago.
And mind you that's just the first time it was written down. I wouldn't doubt that these observations and issues long predate even Plato. Because I think one thing that's become clear is that he wasn't necessarily unimaginably brilliant so much as he was simply capable of objectively seeing the patterns in society and recording them. Though perhaps that is some form of brilliance in its own right.
[1] - https://en.wikipedia.org/wiki/Ship_of_State
There aren't too many people comfortable with intentionally letting others suffer for the sake of the scientific method, and those who do have a hard time getting past ethics review boards.
Instead we have this reality where we throw money at problems and claim credit whenever something good happens, and claim we did good work when nothign changes.
> There aren't too many people comfortable with intentionally letting others suffer for the sake of the scientific method, and those who do have a hard time getting past ethics review boards.
The way I see it, it is a mindset problem rather than an ethical one. Because policies are considered de facto as "improving things", it makes sense that not applying it would be unethical. However, the article makes a good point that this assumption is clearly flawed. If the mindset would be more along the lines of "there is a bad situation, we think we know how to improve it but we are not 100% sure", then it would not sound so unethical to form control groups.
For instance, using the "laptop for every child" case the governement could have instead picked a few schools that would receive school laptops with the intention to measure the effectiveness of the policy.
If it had a positive impact, then you can proceed with it and buy a laptop for every student. It seems to me more logical and also less wasteful to proceed that way.
In the case of laptops for every child, though, people argued that they were redressing an existing imbalance between kids whose parent could afford Internet and home laptops and those who couldn't.
They really should have trialled introducing them to classrooms and observing the impacts and so on- as you say, this is low stakes enough to not be an ethical issue.
The problem in this specific case is the outcome was entirely secondary to two external objectives; selling laptops and a misguided sense of social justice.
I'm not sure what to even discuss about this article due to the low quality. I guess there overall point is that policies and interventions don't necessarily scale, or perform the same across cultures/cohorts, or even accomplish their intended goals. I would largely agree with those points, but not due to the way the article got there.
it's the surprising surprises that get you
That... that's not a failure, that's exactly the point of this requirement
Lots of social programs are not working to improve lives of adults that are already messed up.
Aiming at kids is the way to go.
That’s how the YouTube influencers got big, they basically groomed generations of followers.
Whereas many of the interventions mentioned in the article were seemingly logical but indirect: If we do A, it will cause people to behave B, resulting in C.
In places where you can take the metro to the bar, or walk to the bar, it's a lot different.
https://www.experimental-history.com/p/the-one-science-refor...
mainly, interventions are experiments, and therefore we need more of them, you never know. but they are not experiments, otherwise we wouldn't repeat failed ones over and over, or let existing failing ones run on and on.
maybe the answer really is more simple, that most programs are indeed politically motivated endeavors, indeed largely signals of a worldview, less programs to bring about material improvement.
fwiw - i dont know the answer, but having your cake and eat it, being perceptive of the absurdity, yet, not daring to call it out, is an author's disposition, less a truth. so maybe, if we do the one hand other hand game, accept the bleak truth glimpsed in the facts, but also accept that there's nothing to be done about it within existing frameworks, and not point at it as something desirable.
The fact, it costs taxpayers' money is bad enough. It often reduces freedom of choice, introduces bad second order consequences, etc.
Generally, it is not a government's job to do social engineering.