What happens when the public lose confidence in academic findings? (thomasprosser.substack.com)

https://www.wired.com/story/social-science-reproducibility/

https://www.psychologytoday.com/us/basics/replication-crisis

I didn't learn a damn thing from the process.

In grad school I was told to generally assume that the tool with the second highest line in the key graph was the best tool. The top line was invariably the author's tool and its results rarely translated to other experiments. But the tool with the second line was usable by somebody other than its original author on a different problem space.

It will not happen, though.

Unfortunately I have worked with people with a master’s in computer science from an american university who couldn’t write a coherent sentence in English. Let alone write code or design a system.

I’ve also interviewed people with illustrious careers who just weren’t a fit for the role they applied to.

Credentialing is in poor shape in our industry.

instead of IQ test -> competence signal

you have:

SAT/ACT = IQ Test -> university acceptance criteria -> degree/credentials -> competence signal

As it became clear the people had lost belief in god kings, aristocrats began paying the church to educate their kids and confer nonsense degrees the proles could not falsify.

It was not hard learning guitar on my own. We create a lot of carbon mess for junior to get to an official guitar degree.

There is a whole lot of unnecessary LARPing deference to wealthy people who don’t do much of the real work keeping themselves alive.

It’s not a sustainable social meme to tighten the screws on the masses while handing their own kids the keys to the castle. 800,000 sworn LEO in the US, beholden to a paycheck and retirement account. None of the people at my local bank own that bank anymore. Why is my agency propping up someone else’s grandpa?

All of this was absolutely relevant in 1950 for working at a large corporation, in a white-collar job.

The more recent job market doesn't look much like this, but then good training for the current job market would probably not call for an institution that looks much like a university. It would probably call for something more like trade schools and apprenticeships, with guilds to protect the apprentices from poor treatment (something that, for example, Ph.D. students are sorely lacking). It seems unlikely that universities can adapt to a situation that primarily requires that they go away or at least become much smaller, to be replaced by a different system.

Therefore, they mainly ensured companies hired people from similar university programs that the test givers also attended.

It seems no matter how many great projects I make, or quality papers I write, that's second best to 10 minute logic puzzles for a growing amount of companies.

As with most development team leaders, I have interviewed and employed CS graduates who are woefully unprepared for the reality of a job writing production-level code.

I don't see this as a condemnation of education per se. More that the common concept of education (that it prepares young people for their careers) is completely misaligned with academia's concept of education (that it trains young people in how to think, and how to conduct research).

[0] attribution for this is interesting and varied: https://quoteinvestigator.com/2021/04/02/computer-science/

I'm only in my forties but it's incredible how much this has changed in my lifetime.

I remember talking to an intern a few years ago and they made a comment about what a joke Columbia is (in regards to master's students). I was shocked because when I was a teen Columbia was seen as incredibly prestigious. I argued with them that Columbia is a really prestigious, well regarded school, but they and a few other interns just laughed and rolled their eyes.

Looking into it more it became obvious how this view had changed. It turns out Columbia, like many top tier schools from when I was a kid, has basically turned into get fancy version of a degree mill for master's students. Even though the school boasts some impressive faculty in many departments, it's also clear, based on the grads from there I have worked with after this conversation, that very little of that prestige is passed on.

As someone who used work in academia, it's wild how rapidly the entire system has decayed. From publisher or perish culture creating a mountain of non-reproducible work, to schools across the spectrum rushing to exchange credibility for cash. There have been so many systemic errors made that I don't believe academia will recover.

I suspect we'll see a fairly massive contraction in higher ed in the next decade or so.

In many other fields that have advanced degree requirements, companies care far more about where you went to school and who you studied under.

For example - I have a friend that is an occupational therapist. They've had 5 different jobs in the last 10 years. Every single interview they've done has been mostly just a meet and greet. No real difficult questions, no tests, no trial periods. They just applied, the company saw their degree and experience, and the interview was simply there to make sure they could work together.

It turns out that there’s very high correlation between the characteristics that make successful students and those that make successful workers. So a degree is still a useful signal during hiring, but it doesn’t certify a specific level of competency or skill set.

No university has any incentive to say no to someone who wants to pay 200k to have you grade some homework for a couple of years, as long as they aren't at capacity, and moreover very small incentives to fail them.

The irrelevance of the paper as a signal is the solution to this problem.

That says more to me about the companies practicing this style of interview than it does your degree or the institution granting it.

I think this makes a lot of sense and we have a system that allows you to get a well paying middle class job by completing a vocational education and offers paths to get higher education from there or even switch to the academic track. This always made sense to me and I think it's crazy how this is in other countries where the choice is going to University or learning your profession on the job.

I had a group project in 4th year university, in a very difficult math-heavy advanced graphics course, and my partner provided me with his half of the project and it was all global variables named A through Z, with a few A1,A2,AA,AB thrown in, and reused extensively for different purposes.

His code actually worked but it surprised me that someone could do 4 years of university (in a good CS school) but have almost no talent for actual coding.

In the same way that height among professional basketball players may not be a great predictor of their performance relative to other players, but you don't see too many 5ft players eithers.

I think if you’re working with really advanced topics, the credentials start to be meaningless, because the important fact is whether a candidate can truly work with complex concepts/technologies. You expect the candidate to come in already knowing what they need, and you don’t need any assurance that they’re “trainable.”

But for most of us here in the middle of the pack, the degree is more meaningful because it shows that at the very least, a candidate has been exposed to slightly higher-level concepts. As an example, when you’re trying to get someone to step up out of a help desk role into an analyst/developer role, that’s a lot easier to achieve with someone who’s had at least a degree’s worth of exposure to concepts of strategic thinking, lifecycle management, development methodologies, etc. Most of us don’t need to be passionate geniuses to do good work. For us non-geniuses, the conceptual “scaffolding” provided by academia opens up a lot of potential. I definitely believe that some people don’t need the degree, but they probably aren’t the average Joe.

we need nothing short of a cultural shift toward incentivizing real progress and productivity, and valuing real knowledge rather than education, which we only had for a fleeting moment in the US (middle of last century), that only comes from really constraining markets toward both fairness and risk tolerance. we lost our way when we loosened our belt in that regard (for instance, we now instead reward to act of having capital rather than deploying it strategically).

In Europe this became known as the "Bologna process" in which established "academic degrees" literally - in a EU-bureaucratic hunch and self-aggrandizement - got thrown away overnight (in Germany: Diplom, Magister) and then modernized, accordingly rated and standardized in a modular fashion, repurposed for the international/European (labour) market. The faculties and universities themselves from then on were run in a more and more managerial fashion with "operating numbers" and by winning state-funded budgets ("exellency" programs) and by efficiently recruiting third-party sponsoring.

For a typical (german) undergraduate student there isn't much a difference between school and university, anymore. Most Gen X'ers (and Baby-Boomers for that matter) will readily tell you how they for the first time had to learn to resort to self-reliant learning in want of detailed instructions (like in school) but also began to appreciate the less strict scheduling not only for partying but for developing their critical thinking in heated arguments with peers or when confronted with challenging ideas. This is mostly gone, now, a big part of that former public space has vanished.

For attaining craftmanship a particular skillset for a particular field, universities were never a good place to begin with, that's what internships and later your chosen vocations are for. But if they don't prepare you for the labour market and don't provide you with some kind of Humboldtian ideal - a universal starting point with sufficent leisure in pondering about the right/critical questions to ask in a nourishing environment - so what are they actually for?

It also says:

1. We don’t trust your former employer to restrict promotion to quality employees only. After all, people have to do the leetcode interviews all over again when applying for their next job.

2. If you’re new to the industry and weren’t enough of a chump to take out student loans, you’re welcome to try your luck here

The funny thing is, all those leetcode questions are just the kind of thing that's taught at CS courses (algorithms and datastructures) and the kind of material that CS grads are likely to know well. And the people who interview you are more often than not graduates themselves.

Because, duh, tech companies are part of the same establishment as academic institutions. But don't tell them that because they all want to be disruptive, instead.

What's difficult is that it's hard to communicate the nuance of what a study actually studied. How many people were involved? What were the methods used? What was the effect size? What were the confounding factors that couldn't be controlled for? None of this can be effectively communicated in a world that reads headlines and probably not much beyond that.

And of course, the media does sensationalize because they're incentivized to. You'll click a headline that says "alcohol is good for you, says science" but probably won't click a headline that tries to communicate the nuance.

Further, the criticism of academia as being full of egg-headed elites who are completely disconnected from the public need has been around for decades - far longer than the current systemic funding and tenure problems.

http://www.bbc.co.uk/news/av/world-41521671/100-women-amy-cu...

Very doubtful. You just need common sense. Like, if this actually worked then you'd be seeing it spread very fast everywhere, but in reality it apparently needs a constant flow of TED Talks to get people to take it seriously. Ditto for priming studies, etc.

More recently there's been COVID. The problems here aren't exactly the same as the replication crisis, partly because nobody seems to try and replicate anything in the first place. Really IMHO the problems are much worse. Nothing could possibly destroy confidence in academic and public sector expertise faster than being told masks don't work, followed by they work so well they're now mandatory, followed in turn by seeing case graphs where switching mandates on/off very clearly has no effect whatsoever.

The constant flow of bad models that were never acknowledged is particularly destructive. Most people can't articulate why the models always seemed to be wrong except through vague generalities like "bad assumptions", but it doesn't matter. Epidemiological modelling is toast in the public mind, along with the lockdowns they led to. The understanding that they're unreliable is fact based, not emotional, even if the details aren't there.

At any rate, having done deep dives into too many trash COVID papers in the past two years, I can't say knowing the exact details of why they're wrong makes much difference. The important thing is actually the social problems surrounding them, in particular the lack of any institutional acknowledgement or response that there's an issue. Observe how discussion of the replication crisis started with and still mostly comes from individual researchers, not institutional leaders. We can see there is no psychological acceptance of how badly they've been getting it wrong because, for example, Imperial College London (home of Prof Lockdown himself) just banned parents from attending their children's graduation ceremony due to COVID:

https://dailysceptic.org/2022/03/31/imperial-college-london-...

This is at a time of record high cases and after the Queen got it at 95. Nobody cared, it was in the news for barely a day. ICL comes across as a delusional institution with this sort of act, and combined with all the stories of activist craziness on campus, it's getting widely noticed by the general public.

While papers remain the main standard by which science is communicated maybe there is a place for those in the scientific community that do know to be more vocal about which papers are worth reading.

https://fantasticanachronism.com/2020/09/11/whats-wrong-with...

"studies that replicate are cited at the same rate as studies that do not"

Confidence is eroded in all papers by this sort of thing, because academic structures are the same across fields. Whilst individual fields or labs may be more rigorous than others, people can't be expected to keep a massive database in their heads of which fields are good and bad. To be able to generalize about quality requires institutions that create quality, but universities don't do this. And because universities dominate "science" or at least the public discussion of it, that lack of quality control reflects on all science.

So what does this mean? The professors pushing the papers out are not doing their due diligence and there is no one else doing public checking. The loop is open. It is not science. That's the crisis.

Just last week or so there was a paper in Nature about how fMRI studies -- a modern backbone of neuroscience work -- have also been mostly unreplicable. This follows a litany of other papers showing the same thing. That Nature paper made transparent allusions to genetics which invested tons of money in studies which were also largely unreplicable, with fairly obvious power issues that were ignored. When the replicability crisis started getting attention, there were interviews with pharmacology corporation scientists saying that the pharm industry tends to internally assume that 2/3 of published pharmacology studies will not replicate when applied to the real world. There were studies showing that oncology had similar rates of problems, and so forth and so on.

The social sciences is fuzzy as you say but they also tend to turn the microscope on science itself first. Meta-analysis in its modern form (as opposed to early statistical methodology papers) comes from psychology; this is just more of the same. There probably are differences across fields but I think few fields are immune.

Also, I would push back on the idea that the general public's mistrust is limited to the social sciences. I would argue that issues with climate change skepticism are driven in part by a general distrust of academic findings and scientists, and many of the problems during the pandemic with skepticism about vaccination, masking policies, and so forth were driven by skepticism about biomedical research. Possner said "such challenges do not affect academic confidence in the reality of (say) climate change and Coronavirus"; I was puzzled by this precisely because I couldn't tell if he was being ironic, or pointing these out as areas where there is a contrast between academic confidence and public confidence in those findings, as if to say "well look at how much the public distrusts that; now imagine when academic confidence is eroded further?"

My biggest problem with the linked piece, in fact, is that it is framed around the question "what happens when the public loses confidence in academic findings" when that problem is already here to a big extent. We saw what happened during the pandemic. That's what happens.

Academics to some extent is falling apart, and it's not limited to the social sciences. The problems with misttrust are running deep and I think academic and political communities have their head in the sand, rather than taking their structural problems seriously. It's always the messenger: the irrational public, the muddle-headed social scientists who are exposing the problems, it's never the corruption in academics itself.

For some time, Kale was faddy, every study was recommending eating it. REcently, I read somewhere it gives you cancer.

groan

https://www.theonion.com/eggs-good-for-you-this-week-1819565...

I would argue instead that you have demonstrated the importance of peer review (a cornerstone of academic research). The problem is not really academic research, but that we do too little of both academic research and peer review.

First, as you found, there was no correlation between deaths and descent. Presumably there weren't statistically anomalously fewer Gomezes than Smiths.

This doesn't eliminate the fact that people wrote down fewer dead as Hispanic than as white. So:

Second, there actually were fewer dead Hispanics than dead whites, understanding "Hispanic" as "a person who looks Hispanic" and "white" as "a person who looks white".

Third, or alternatively there is no statistical difference between dead Hispanics and dead whites, but somehow fewer dead Hispanics ended up at the morgues to be written down as dead.

“Now faith is the assurance of things hoped for, the conviction of things not seen.” - Hebrews 11:1 (ESV)

> I have the same opinion of academics - show me what you can light on fire.

I don't know if you're going to be welcome in a lot of liberal art colleges, since this is the ending(SPOILERS!):

> And Elijah said to them, “Seize the prophets of Baal; let not one of them escape.” And they seized them. And Elijah brought them down to the brook Kishon and slaughtered them there.

There's a huge power in knowledge consensus. It's got all kinds of flaws, but it meanders generally in a growth direction, and that's better than any other prediction system I'm familiar with.

This also has a reasonable take-away: sometimes lighting an altar on fire can be faked even when it looks real. Or, "beware the flashy demo". Fusion and especially some of the more shameless corners of AI come to mind.

My apologies if this stretches the analogy too far.

Alexa, set room temperature to 28°C!

No. We shouldn't be putting so much weight on the title of "doctor" for physics PhD's either; several physics PhD's I know personally are absolutely barking mad, and there's a long history of physics PhD's getting over their skis on other fields like medicine.

Meanwhile, the usage of "doctor" that you're decrying here is the more historically well-established one.