J.M. Ziman wrote in 1962: "It is typical of modern physicists that they will erect skyscrapers of theory upon the slender foundations of outrageously simplified models."
I guess it is only a matter of time before the proverbial skyscraper collapses. I have had first hand experience with publishing papers of decreasing relevance. It's almost always due to professors being unwilling to give up their mental model of reality. I had one professor tell me he would not let me publish my result because it contradicted a paper he wrote in the late 70s. It was the grossest violation of the spirit and purpose of science I've encountered, and it quickly turned me off from the whole institution. Egos in academia are often unchecked.
Is there any advice onto how one deals with research misconduct in the industry? That allows to avoid expensive pitfalls like Theranos? Is leaving the place the best option? Simply leaving the place doesn't seem optimal. It feels like, it just leaves an opening for a recent H1 from a poor country, with a different tolerance for dishonesty. And the end-result is continuing misconduct.
I think you should speak up with the expectation that whistleblowers don't do well, but with the knowledge that science corrects only if scientists correct it. It's unlikely a single person can take down the culture of an industry, but one person can certainly start a movement.
You're definitely well justified in feeling torn about this. I agree that leaving in silence is not the best idea, but I certainly wouldn't blame you if you did that. It's hard to take action.
"This problem cannot be solved by appointing non-experts to review panels – that merely creates incentives for research that’s easy to comprehend."
That doesn't sound SO bad. Selection criteria seems like a major issue here, and the author only talks about the validity of the results.
Why do we care about physics? Or psychology? There are answers to these simple questions, but I don't think the answers inform the selection criteria. We don't care about physics because academic researchers need to publish papers or get funding.
Having any old non-expert on the panel could be problematic. Especially if they become political tools, and right now they'd definitely become political tools. But it would seem reasonable to me if there were adjacent experts. What do engineers or chemists want to see in physics? These aren't hostile interlopers. But if nobody outside a field cares about a line of research, or trusts a line of research, then maybe that's a bad line of research.
Agreed. The article also said that having outsiders on the panel would just be an incentive to write easy to understand research proposals, and I was reminded of the idea that you need to understand something really well to be able to explain it clearly. If a physicist cannot explain their research proposal to an engineer or physicist, perhaps they need to go off and think about it some more.
You’re right. What we should also do is that all critical code that may significantly impact lives and people’s wealth and security should be written in a manner that can be understood by laymen.
So functional programming is right out. Anything more complex than BASIC is probably not a solution. Probably only literate programming is allowed.
There was some short Ted Nelson video I watched a while ago (I'm afraid I'll never find it) where he made this claim: they needed SQL and a separation of concerns (i.e., DBA) to build up a financial system where it was hard to embezzle money. And it's not JUST ACID and transactions that you need (though you definitely need those things!) but also the ability to audit access to the database-of-record, which SQL gives you.
Maybe the corollary is that you should build up an appropriate domain DSL so that you can express your important claims or transactions in a comprehensible way. It's not to say that you can't misimplement the DSL to your advantage, but there's an implicit good-faith claim you are making that is different from raw vertically-integrated code.
> But it would seem reasonable to me if there were adjacent experts. What do engineers or chemists want to see in physics?
A natural experiment in this already exists: there are interdisciplinary journals and conferences with heterogeneous editorial boards. From the outside looking in, I suspect mostly what happens is that either people stick to reviewing the subset of papers from their own community or else do a poor job of reviewing papers from outside their own community.
> But if nobody outside a field cares about a line of research, or trusts a line of research, then maybe that's a bad line of research.
I'm skeptical; a lot of research is useless until it's groundbreaking, and the phase shift is rather late in the development.
I like that this article includes suggested changes:
> For starters, every scientist should know how being part of a group can affect their opinion. Grants should not be awarded based on popularity. Researchers who leave fields of declining promise need encouragement, not punishment because their productivity may dwindle while they retrain. And we should generally require scientists to name both advantages and shortcomings of their hypotheses.
The first and last points look good to me. The middle two, not so much. They certainly sound reasonable, but what does a grant system not based on "popularity" mean? How is it different from what we have now, where we find some (putative) experts in the area, put it in front of them, and ask what they think? We need some way of deciding what to fund and what not to fund, and who exactly is better equipped to make those decisions than existing researchers?
Similarly, I like the idea that people should be able to switch fields. But the implicit argument seems to be that the emphasis on publications makes this difficult, so publications should be de-emphasized in assessing researcher quality. But what is better? Asking a researcher's peers? Then it is a popularity contest again. Weighting publications differently, i.e. "what are your 5 best papers?" rather than "how many papers do you have?"? Then who judges the 5?
For all the criticism heaped upon bibliometrics: they get used because assessing the quality of most research is difficult, so even noisy filters like conferences (in CS) or journals are taken as a useful signal. It's not clear to me what better options exist, especially when trying to choose between many options. It's just not possible for one person or even a small group of people to truly familiarize themselves with the bodies of work of all 100 candidates for a faculty position/grant/fellowship/etc.
Agreed, and I would add that suggestions 1 and 4, although fine, will not be enough to make a substantive difference. To be honest, Sen. Paul's suggestion (if implemented properly, i.e. add experts from other fields rather than political hacks) sounds more likely to help.
A graduating PhD student in theoretical physics from a Top 5 school said something similar to me last year. Roughly, he said it was very easy to slip into bullshitty fooling yourself when doing theoretical physics research, and it took vigilance from within to do truly good research, because external bodies were unlikely to really punish you as long as the math worked out.
> it took vigilance from within to do truly good research
Well said. I've been mulling on the idea that a scientific mindset must include a deeply embedded skepticism that isn't afraid to analyze and question even the most pervasive and mundane aspects of one's own being.
Of course, one has to get through the day and take care of life, in a practical fashion. But I think too often we assume our actions to be much more intentional than they actually are.
I think this article is looking at the effect rather than the cause. Why might it be that so much of science has started to edge away from well... science. Today nearly 1 in 8 people have a masters degree or higher. And in one in 33 have a doctorate or professional degree. [1] And these numbers seem to be increasing. That's really insane if you think about it. We now have more people with postgraduate degrees than had any degree in 1970, and postgraduate education generally implies a necessity to publish. But discovering genuinely new and novel information is not easy. So what do people publish instead? Enter the current state of science and academia. I think the most informative paragraph in this article was:
"Or look at Brian Wansink, the Cornell Professor with the bottomless soup bowl experiment. He recently drew unwanted attention to himself with a blogpost in which he advised a student to try harder getting results out of data because it “cost us a lot of time and our own money to collect.” Had Wansink been aware that massaging data until it delivers is not sound statistical procedure, he’d probably not have blogged about it."
Physics is one of the least susceptible to this issue, but no field is immune. The issue is that 'science' has become more about publishing than actual discovery. The two should be synonymous, but they are not.
The statistics about masters and professional degrees are mostly irrelevant; those aren't research degrees and, although there are some exceptions, people with those degrees don't publish science.
1.77% of the 25+ population has a doctorate degree. Even among those, a huge amount are not PhDs and are not really research-focused. E.g., go read the CV of the typical University of Phoenix doctorate holder.
Furthermore, it's worth noting that the world got a lot more complex since the mid-20th century. The entire field of Computer Science happened, for example.
Definitely great points. On the other hand, even when you really lowball the numbers - I do think the point remains. 1.77% is 3.5 million people with doctorates in the USA alone. Perhaps we can see this issue manifest is something more directly quantifiable. Postdoctoral researchers tend to be some of the most highly educated and ostensibly skilled individuals in our population. Yet an average postdoc [1] will tend to earn [much] less than a mailman [2]. It seems difficult to explain this outside of a severe oversupply. And should the rate of people pursuing doctorates be increasing, which seems to be the case, I think we can expect things to get even more silly both in terms of compensation as well as output quality.
First, I'm generally skeptical that "the market" is accurately pricing the value of work produced by either postdocs or federal mail carriers.
Second, a postdoc is still a training position. Medical residents are also not paid much better than mail carriers either.
Third, postdocs select for two types of people: those who want R1 professorships and those who need a position to stay in the country. Both of those groups drive down wages. I.e., it's sort of meaningless to compare a group of people who are working for the money to a group of people who are working for a shot at a seat in the upper tier of a tournament-style profession.
Fourth, both annual and lifetime earnings of PhD holders (at least in STEM) continue to outperform high school graduates (including mail carriers).
There's an oversupply of people interested in R1 faculty positions, but that doesn't mean there's a oversupply of phds per se. The labor market seems to absorb these people quite readily and at apparently high salaries as soon as they decide to quit academia.
The market is not created by some third party. It simply represents the meeting point between what people are willing to buy for and what people are willing to sell for. That postdocs are willing to work for scraps relative to their skill level is indicative that supply has gone way beyond demand. For instance software developers also face similar issues of what you're mentioning with lower wage foreign workers willing to accept much less thus driving wages down, yet nonetheless high demand keeps wages high.
This is not particularly controversial and some have even suggested that starting to increasingly restrict the number of graduate students admitted could be the right path forward, as mentioned in this article [1] from John Hopkins: These and other suggestions for career preparation make one implicit assumption: that the nation needs the current number of biomedical scientists. A radically different approach to balancing the biomedical workforce equation would be to simply reduce the “supply” — that is, admit fewer students into science graduate programs. The article ends on what was apparently supposed to be an up beat note suggesting that instead of such draconian measures, perhaps we can look to people who are doing things like aiming to go teach high school science, with a doctorate.
Just like there's a glut of wordpress plugin authors and a shortage of ML experts, different fields of academia have different levels of supply/demand.
Also, USPS really is not a prototypical example of free labor market: those salaries/pensions/benefits will be a lot smaller if/when usps privatizes...
I never understand why there's this thinking that more education "should" lead to a higher salary. It's not borne out by either theory or practice. I've heard several PhD's complain that they don't earn as much as someone with not degree at all. Is it something that PhD candidates tell themselves to get through the misery?
It comes down to supply versus demand. If what somebody learns in increasingly advanced education has any practical benefit then it would imply that they would have a practical skill that few other people would have.
Of course this is premised on the rarity of supply, which is no longer really true with millions of phds and growing out there. And it's also premised on the existence of a practical and meaningfully valuable skillset gained in the process of obtaining the doctorate which is I think reasonable, but still certainly debatable -- and at the minimum going to be contingent on what/where somebody studies.
28 comments
[ 0.26 ms ] story [ 52.0 ms ] threadI guess it is only a matter of time before the proverbial skyscraper collapses. I have had first hand experience with publishing papers of decreasing relevance. It's almost always due to professors being unwilling to give up their mental model of reality. I had one professor tell me he would not let me publish my result because it contradicted a paper he wrote in the late 70s. It was the grossest violation of the spirit and purpose of science I've encountered, and it quickly turned me off from the whole institution. Egos in academia are often unchecked.
An MBA friend’s professor told him that economics is just like gravity. It’s not worth wasting his time on Earth thinking about it.
You're definitely well justified in feeling torn about this. I agree that leaving in silence is not the best idea, but I certainly wouldn't blame you if you did that. It's hard to take action.
That doesn't sound SO bad. Selection criteria seems like a major issue here, and the author only talks about the validity of the results.
Why do we care about physics? Or psychology? There are answers to these simple questions, but I don't think the answers inform the selection criteria. We don't care about physics because academic researchers need to publish papers or get funding.
Having any old non-expert on the panel could be problematic. Especially if they become political tools, and right now they'd definitely become political tools. But it would seem reasonable to me if there were adjacent experts. What do engineers or chemists want to see in physics? These aren't hostile interlopers. But if nobody outside a field cares about a line of research, or trusts a line of research, then maybe that's a bad line of research.
Well, that's exactly what they're asking for funding for.
The entire crux of the issue is that these are not complete ideas, and need to be developed further; which takes time, and therefore, money.
So functional programming is right out. Anything more complex than BASIC is probably not a solution. Probably only literate programming is allowed.
Maybe the corollary is that you should build up an appropriate domain DSL so that you can express your important claims or transactions in a comprehensible way. It's not to say that you can't misimplement the DSL to your advantage, but there's an implicit good-faith claim you are making that is different from raw vertically-integrated code.
There are still billions of lines of production COBOL code out there, about as complex as BASIC, running all our financial and infrastructure systems.
Simple is sometimes much, much, better than performant or elegant.
A natural experiment in this already exists: there are interdisciplinary journals and conferences with heterogeneous editorial boards. From the outside looking in, I suspect mostly what happens is that either people stick to reviewing the subset of papers from their own community or else do a poor job of reviewing papers from outside their own community.
> But if nobody outside a field cares about a line of research, or trusts a line of research, then maybe that's a bad line of research.
I'm skeptical; a lot of research is useless until it's groundbreaking, and the phase shift is rather late in the development.
> For starters, every scientist should know how being part of a group can affect their opinion. Grants should not be awarded based on popularity. Researchers who leave fields of declining promise need encouragement, not punishment because their productivity may dwindle while they retrain. And we should generally require scientists to name both advantages and shortcomings of their hypotheses.
The first and last points look good to me. The middle two, not so much. They certainly sound reasonable, but what does a grant system not based on "popularity" mean? How is it different from what we have now, where we find some (putative) experts in the area, put it in front of them, and ask what they think? We need some way of deciding what to fund and what not to fund, and who exactly is better equipped to make those decisions than existing researchers?
Similarly, I like the idea that people should be able to switch fields. But the implicit argument seems to be that the emphasis on publications makes this difficult, so publications should be de-emphasized in assessing researcher quality. But what is better? Asking a researcher's peers? Then it is a popularity contest again. Weighting publications differently, i.e. "what are your 5 best papers?" rather than "how many papers do you have?"? Then who judges the 5?
For all the criticism heaped upon bibliometrics: they get used because assessing the quality of most research is difficult, so even noisy filters like conferences (in CS) or journals are taken as a useful signal. It's not clear to me what better options exist, especially when trying to choose between many options. It's just not possible for one person or even a small group of people to truly familiarize themselves with the bodies of work of all 100 candidates for a faculty position/grant/fellowship/etc.
Well said. I've been mulling on the idea that a scientific mindset must include a deeply embedded skepticism that isn't afraid to analyze and question even the most pervasive and mundane aspects of one's own being.
Of course, one has to get through the day and take care of life, in a practical fashion. But I think too often we assume our actions to be much more intentional than they actually are.
"Or look at Brian Wansink, the Cornell Professor with the bottomless soup bowl experiment. He recently drew unwanted attention to himself with a blogpost in which he advised a student to try harder getting results out of data because it “cost us a lot of time and our own money to collect.” Had Wansink been aware that massaging data until it delivers is not sound statistical procedure, he’d probably not have blogged about it."
Physics is one of the least susceptible to this issue, but no field is immune. The issue is that 'science' has become more about publishing than actual discovery. The two should be synonymous, but they are not.
[1] - https://en.wikipedia.org/wiki/Educational_attainment_in_the_...
1.77% of the 25+ population has a doctorate degree. Even among those, a huge amount are not PhDs and are not really research-focused. E.g., go read the CV of the typical University of Phoenix doctorate holder.
Furthermore, it's worth noting that the world got a lot more complex since the mid-20th century. The entire field of Computer Science happened, for example.
[1] - https://grants.nih.gov/grants/guide/notice-files/NOT-OD-15-0...
[2] - https://www.bls.gov/oes/current/oes435052.htm
Second, a postdoc is still a training position. Medical residents are also not paid much better than mail carriers either.
Third, postdocs select for two types of people: those who want R1 professorships and those who need a position to stay in the country. Both of those groups drive down wages. I.e., it's sort of meaningless to compare a group of people who are working for the money to a group of people who are working for a shot at a seat in the upper tier of a tournament-style profession.
Fourth, both annual and lifetime earnings of PhD holders (at least in STEM) continue to outperform high school graduates (including mail carriers).
There's an oversupply of people interested in R1 faculty positions, but that doesn't mean there's a oversupply of phds per se. The labor market seems to absorb these people quite readily and at apparently high salaries as soon as they decide to quit academia.
This is not particularly controversial and some have even suggested that starting to increasingly restrict the number of graduate students admitted could be the right path forward, as mentioned in this article [1] from John Hopkins: These and other suggestions for career preparation make one implicit assumption: that the nation needs the current number of biomedical scientists. A radically different approach to balancing the biomedical workforce equation would be to simply reduce the “supply” — that is, admit fewer students into science graduate programs. The article ends on what was apparently supposed to be an up beat note suggesting that instead of such draconian measures, perhaps we can look to people who are doing things like aiming to go teach high school science, with a doctorate.
Just like there's a glut of wordpress plugin authors and a shortage of ML experts, different fields of academia have different levels of supply/demand.
Also, USPS really is not a prototypical example of free labor market: those salaries/pensions/benefits will be a lot smaller if/when usps privatizes...
Of course this is premised on the rarity of supply, which is no longer really true with millions of phds and growing out there. And it's also premised on the existence of a practical and meaningfully valuable skillset gained in the process of obtaining the doctorate which is I think reasonable, but still certainly debatable -- and at the minimum going to be contingent on what/where somebody studies.