Could this just be selection bias? Clearly the older works that people are still talking about are pretty special. This is sort of addressed in the "top cited papers" section. But I think a better metric would be something like "are papers that we'll still be discussing in 20 years still being published." Hard to say until 20 years goes by.
I am talking about old papers that might not be widely known (i.e., have few citations) but present some useful results that I can use in my research.
In my field, the latest papers are a bit "click-baity". They have interesting titles with many trending words but when you read them it is hard to find either a coherent vision or a single result that one can build on top of.
Of course, I am generalizing. There is great research being done today and low-quality papers in the past. But it is completely true that the need to publish has distorted the goal of research communication.
> A basket of indicators all seem to document a trend similar to what we see with technology. Even as the number of scientists and publications rises substantially, we do not appear to be seeing a concomitant rise in new discoveries that supplant older ones. Science is getting harder.
Nothing worth doing is easy. We have cleared the low hanging fruit and can now do the actual work. This is so exciting :)
(also I wonder sometimes if as a society we are teaching people less to be critical thinkers, who are not afraid to disagree with the mainstream)
But more seriously, I think what you said only applies in iterations of revolutions. A new breakthrough (e.g. new physics, new invention) could create a lot more low hanging fruit.
> I think what you said only applies in iterations of revolutions
I think a major problem is ever since general relativity, all of our (existing and potential) new revolutions are very abstract and hard to relate back to human perception.
Take dark matter/energy for example - if we get a better model to understand that, it will revolutionize how we think about the universe, but it will (hopefully) have less impact on day-to-day society than the nuclear model of atoms did.
Hmm, I don't think that's something we can predict. Someone could have said the same thing before general relativity... in fact, most people did because they thought we were almost done.
I would argue that general relativity barely affects humans day to day lives, whereas the nuclear model of the atom completely changed the global political system.
>Take dark matter/energy for example - if we get a better model to understand that, it will revolutionize how we think about the universe, but it will (hopefully) have less impact on day-to-day society than the nuclear model of atoms did.
Is unrelated to:
>I would argue that general relativity barely affects humans day to day lives, whereas the nuclear model of the atom completely changed the global political system.
Maybe to help you, I was using General relativity as a time marker (i.e. it was the first sea-change scientific discovery that had limited impact on day-to-day human life)
You then argued that my premise was incorrect, because before general relativity was formalized, people could have believed that all future discoveries were too abstract to affect day-to-day human life.
I then argued that you were correct in that general relativity in fact was too abstract to affect humans' day-to-day life.
I think we need to figure out how to extend human lifespan to allow for experts to continue to grow their knowledge/skills past the 40 year retirement mark to have a greater chance of making more advanced breakthroughs.
I think the (hopefully) more realistic action plan would be to change the academic system so that the average academic does not need to spend what amounts to the full official working hours on non-research tasks, latest after their first Post Doc, i.e., ~2 years after reaching academic maturity...
Also, many major historical scientific breakthroughs have been just a smart (usually rich) dude/dudette with a lot of free time and not a member of the conventional 'academic' system.
So somehow figure out a way for public to get in on the research instead of government restrictions on who can purchase supplies which limits research to the .0001% who have PhD's.
This opens the door to a lot more mind power going into research.
At this point, I think “a lot of free time” is a more limited resource than supplies and lab equipment. Though I may be biased as I come from an extremely non-capital-intensive field (pure math).
I see you've not met my advisors. Please no. Old scientists sometimes contribute to push the younger generation forward. In my experience though, most of the time they're a liability, due to their incredibly overdeveloped ego and their deep involvement into politics. I'm an academic MD, so this phenomenon might be more marked than in other fields.
See the book "the structure of scientific revolutions".
There's no private industry competition in academia, which allows those old scientists to become old slow codgers.
Most scientific breakthroughs through out history were just private citizens outside of the conventional academic establishment.
We need to put an end to these government restrictions on buying supplies and equipment which keeps the research down to just a tiny fraction of the population.
Allowing private citizens to do research would put a fire under those old codgers asses.
>> Allowing private citizens to do research would put a fire under those old codgers asses.
I have a few published papers in peer-reviewed journals with 30+ citations and I'm a college dropout, but I run a company that can fund/produce this kind of research as part of our mission statement. It's not really restrictions, but rather money/funding, and willingness of private companies to publish publicly (which is a big blocker for most).
it's not the government restriction, but wage slaving that keeps private individuals from making scientific breakthroughs. Perhaps UBI could be the answer.
I'm in a field that requires no more advanced equipment than a decent laptop, and there's not a vast array of private citizens "putting a fire" under anyone.
One common thing seen when looking back on discoveries is many times the 'same' discovery was made independently by many people. The bottleneck was more in 'publishing', that is getting other people to know about what you figured out.
If I, with my current education, was sent back 200 years... I could probably do OK with inventing stuff (engineering education, so I have some good answers backed by a nice model, much of which I'd have to work backwards to derive because who remembers 200 level engineering classes?). It took Marconi a while to realize that he had to ground the antennas, so there's a pretty good discovery for free.
On the other hand, if I was actually born then... I think I'm a pretty clever guy, but not a genius. Someone like me 200 years ago would probably... I dunno, be trusted to fix mill equipment. Or maybe work on watches, depending on where I was born. Maybe I'd be a town pharmacist or something.
Up until about 300 years ago it was possible for a wealthy person with plenty of free time to learn literally all of the scientific knowledge in western civilization.
Sorry if this sounds elitist to anyone but (I guess it is):
When the R&D workforce expands at the rate it has over the last few decades you will invariably see less talented and less talented people entering the field. Also, various forms of overhead will grow with the size of the workforce.
As an example about 20 people attended the 1911 Solvay Conference on Physics. How many attend a conference today?
With the growing size of the R&D workforce also comes the most destructive force in any human endeavor: politics.
IMVHO less talents are there because less public research where researchers manage themselves is there. R&D on-purpose for making money, product, in a publish-or-perish aka short-time-to-market move simply do not work.
Weelll ... the world population in 1911 was approx. 1,8 billion people, growing fourfold since then. One would expect some extra talents to emerge from the extra population.
Also, an average Earthling is now much richer than before. A random Korean or Turk of 1911 had much smaller chance to study physics, much less money to even buy a ticket to travel to the Solvay Conference. Both is now easier and cheaper.
On the downside, there is a lot more prestigious and well paying jobs today. Instead of tackling physics, best minds of today may be sitting in Facebook, trying to discover yet more ingenious ways to push ads on the rest of us.
> ... the most destructive force in any human endeavor: politics.
I imagine you were being tongue-in-cheek, but just to unreasonably latch onto that statement a bit: politics is certainly infuriating, and it's easy to treat it as some unalloyed Bad Thing that we wish would just go away, but politics is essentially communication, and so it can be a good thing, too. The bigger the group, the more communication you need to keep everything (somewhat) together.
For instance, on a larger "human endeavor" scale, I am grateful that WW3 has not happened yet — thank you politics (:
As with many things, it's a tool that can be used poorly or used well. But the tool itself is not inherently bad or destructive, I don't think. Or maybe it is both inherently good *and* bad, and you can't entirely separate the two.
The first comment was clearly talking about the internal, institutional politics of things like professional societies, academic fields, etc, while if we look at the other branch of this particular subthread we have a pretty long post about things like the media and international politics.
Perhaps it is impossible to nip talk of the latter type of politics in the bud, though.
I'm not especially convinced that the two are entirely separate. It feels to me that they sprout from the same source, more or less. I suppose it depends on where you want to draw a line around a very fuzzy word ("politics").
My intent in bringing up the issue originally was that when you have larger groups of people interacting, you get larger-scale patterns of communication. People form groups and sub-groups; people give support to groups based on trust of that group or one of its representatives, rather than getting to know everyone individually; higher-level conglomerations of ideals form because tracking every issue in detail is too much, etc.
There's a lot of gunk that comes along with that. But it is not all bad, or destructive. It can even be _con_structive.
I think that generally applies to institutional politics as well as national/world politics.
OK, but it's a particular kind of resource allocation.
Economics is about resource allocation using money and markets. Politics is about resource allocation using... influence? Connections? Coercion?
If we had an open, transparent market where sponsors could bid on research, it might work better. (Hey, HN: Anybody want to set up such a market, funding it by taking a small cut of the bid?)
Imagine we somehow were able to purge all political and media influence, past and present, from the minds of each person today. And then had them rank the issues they find most relevant in the world. Where do you think the things people today spend all their time politicking on would rank? Similarly imagine they wrote out their 'oughts' of what the world ought look like in their eyes. How much overlap do you think there would be with the political platforms of today?
Of course we can only speculate, but to me it seems self evident that there would be effectively 0 overlap. The issue is that politics invariably turns into a viral team sport where people pick some side, adopt that side's views wholesale, convince themselves they're the most important thing ever, and then set out to convert everybody else to their team, beat the other teams in any way possible, and of course always fanatically cheer on their own team regardless of whether or not its deserved.
And then when things get hot enough, you get violence. Politics is precisely what started WW1. An otherwise irrelevant Archduke was assassinated and then that politically snowballed into everybody killing everybody everywhere, because politics. Then we chose to engage in extreme political myopia imposing absolutely harsh punishments on the losers of the war which, shockingly enough, didn't really lead to them rejoining the "world order" but instead declaring war on it again, and nearly winning.
And the only reason we haven't yet had WW3 is nuclear weapons. War doesn't really work when you can guarantee that your country (and you, for that matter) will most likely not exist at the end, "win" or lose. But now political frenzies are overriding even that most basic aspect of self preservation and inching us closer to WW3 than we've ever been. So no, I do not think he was being tongue in cheek.
> Where do you think the things people today spend all their time politicking on would rank?
I realize "politicking" is often used in a derogatory sense, and perhaps that's how you mean it here, but there is a whole lot of plain-old-politics that takes place on a regular basis which has good overlap with those issues you hinted at. Things like: planning and building infrastructure, making sure people do not go hungry or thirsty and that they have places to live, having common laws, etc. A list kind of like "what have the Romans ever done for us?"[1].
Certainly, politics/politicking can also have an amplifying effect on the bad stuff too, as you mention.
I suppose my main issue with what you say is the bit starting: "politics invariably turns into a viral team sport ..."
I don't see it as invariable. The pendulum has been swinging that direction recently, and I don't like it either, but I don't see it as a one-way road to doom.
Though I respect if you do feel that way, of course!
----
Aside:
Your comment led me to read a bit about history, and I found on Wikipedia[2]:
"According to a 2021 study, Franz Ferdinand's absence was key to the breakdown of diplomacy and escalation into war, as Ferdinand had been the most powerful and effective proponent for peace in Vienna."
Like you, I had the "an otherwise irrelevant Archduke" idea in my head, but I guess now I'm not so sure. So, thanks for the indirect new info (:
But it seems like if you're making the argument that science is getting harder, then raw numbers should matter more than percentages. From the article:
"I’m claiming that science is getting harder, in the sense that it is increasingly challenging to make discoveries that have comparable impact to the ones in the past".
So, unless you're arguing that there aren't an equivalent 20 people alive today as prestigious as the 20 people that went to the 1911 Solvay conference. Then the fact that there are greater percentage of less talented people now, shouldn't make it harder for the greater number of talented people to make "discoveries <of> comparable impact". (With some caveats of course.)
I think discoveries of equivalent impact are being made, but not in physics laboratories. Looking back at the current century I think it will be clear that the most impactful discoveries / inventions were related to information processing and that they were made in the private sector.
Yea I know. But I figured the point would stand anyway so used the example. Which 20 people in physics would you even invite today? (Rhetorical question.)
So acedemical research isn't relevant to new technologies anymore? Then one might conclude that most technology driven research is done in private companies rather than in public academia.
Private companies are often quite happy to fund an academic research group for a specific project. They also sometimes use this as a source of pre-vetted, competent future hires for R&D positions.
The focus is quite different in private companies and research institutions.
Yes, but why? How can taxpayer-funded research institutions morally justify doing research that doesn’t increase our collective wealth or at least lead to some useful application in the short term?
Because aprior it's not really clear what does increase our collective wealth. For example, the whole quantum age (and hence the silicon age), came about from people trying to understand the spectrum of radiation emitted from a heated box. If you spoke to someone at the end of the 19th century they'd probably think this was a waste of time, and tell you that people should be trying to make steam engines more efficient.
> How can taxpayer-funded research institutions morally justify doing research that doesn’t increase our collective wealth or at least lead to some useful application in the short term?
It is not obvious that fundamental research does not “increase our collective wealth of lead to some useful application”. Just not in the short term, usually. Despite a lot of economics theories, companies being directed by their self-interest or that of their shareholders does not lead to good outcomes over the long term. Someone needs to do it, who’s not going to be limited by ROI over the next 5 years. In practical terms, this can only be public funding.
Taxpayer-funded institutions justify it by pointing out that if there are short-term applications, companies tend to already do the R&D themselves, therefore public money is not needed to have good outcomes. The alternatives are:
- research in private institutions with public money (the big pharma model), which has obvious ethical issues when said private institutions make billions off of publicly-funded research.
- public institutions funding short-term research through things like patent licensing, which is quite impopular (there is a strong argument that public institutions’ output should be public; governments are not corporations).
- governments not funding research at all, which quickly leads to countries falling hopelessly behind as you cannot count on corporations to give a fuck about public good.
Overall, it seems like a good idea to let everyone do what they do best: corporations focus on bringing stuff to market and applications developments, whilst institutions with a longer outlook focus on things that could be used in applications 20 to 100 years from now.
By definition the new is not always obviously the next, so "not relevant" requires a crystal ball for anything other than basically already established industries.
Deep learning is beyond what VLSI was for the transistor, for example.
Not as much as it used to it would seem. And that has got to do with some major misalignment in academic institutions. I don’t even want to sound off on all the BS “research” that’s going on in humanities (and which worryingly spills into STEM at an alarming rate).
But even STEM research itself wastes too much time with costly self-serving objectives, rather than shooting for breakthroughs that lead to actual applications. Take CERNs large hadron collider, which has produced preciously few new insights, despite costing a fortune in taxpayer money. The Higgs boson is all well and good, but it’s hardly a new finding.
Or take mathematics’ famous millennium problems, where only a subset (eg. P vs NP) would lead to practically useful new insights. By contrast, solving some obscure numbers theory problem would benefit humanity how exactly?
I think it's the opposite. Science stagnates because the system focuses too much on justifications and practically useful results and too little on curiosity.
For a few decades, the prevailing school of thought in public administration has focused on justifications. You must always justify how the public will benefit from the proposed use of money. Then you must report that you used the money for justified reasons. The administrators will also audit you to ensure that you didn't use the money for frivolous purposes.
As a consequence, you can't get funding for studying something you find interesting. Instead, you must always justify how your research will benefit the public. If you manage to get funding, you must spend a lot of time reporting how you used that money and what it did enable you to do. Those reporting requirements are also one of the major reasons for the administrative bloat in the academia.
The general public does not want to fund BS research, and science stagnates largely as an unintended consequence of that.
I would say for cs this is largely true now? At least for deep learning stuff my impression is pretty much the most impactful stuff are coming from the big tech companies. Academics sort of pick the scraps around them. I mean Google (deep mind) were the ones who figured out protein folding and for a while quite a lot of academics were jittery that they'd keep that knowledge to themselves.
When was it easy? So much early discovery has been binned or replaced; easy answers were wrong. It took hundreds of years of effort to bubble up relativity and information theory.
When was this golden age in the past when the world was magically easier and “better”? I see complaints about social media brainwashing people but what is religion and nation state populism?
I have a hard time taking contemporary opinions on “life is getting harder” seriously or sincerely. It’s just another clickbait trope.
It was definitely easier in the past. I look at a lot of past scientific discoveries and I think I could have easily come up with them if I was in the time period and was educated. Could have made light bulbs, automobiles, telephones, computing machines, etc.
Life is getting harder in some fairly concrete and objective ways. Most noticably the number of hours the average person needs to work to be able to buy a house has gone up a lot.
That’s a political problem not a scientific problem.
No one wants to tax rich people who are of the age to have benefited from an era of high taxes. We let them pull the ladder up behind them.
There is no real reason to follow their orders or coddle their sensibilities. What are they going to do? Instigate a civil war from their Lay Z Boy? 60-70% of the population want to reverse their political policies across the spectrum of contexts. Just do it.
I do think it is getting harder, but that is not necessarily bad at all. One big bottleneck in science is the technology to do experiments. A lot of the discoveries that required lower tech are for the most part over, now we need some serious technology to make discoveries. Hubble has its limits, so we need the James Webb Telescope. Einstein predicted gravitational waves, but it took almost 100 years to develop the technology of LIGO and get it up and running to run the experiments. Also, the technology needed is also insanely more expensive. Newton needed time, pen, paper to writes hit laws of motion. Now we need satellites out at the L2 point that can be put in place remotely since it may be awhile before people can swing by and make repairs or troubleshoot physically.
Maybe just spit balling. Maybe it isn't that science has gotten harder, but science needs more engineering support than what it currently has to develop the technologies to allow more a more rapid discovery.
The Scientific Method, as classically defined, is a process of reducing a potential problem space to a single variable and testing it six ways to Sunday to prove to yourself and others that you aren't just imagining things. I don't think it's an accident that physics and chemistry are far ahead of psychology and biology. Both are emergent behaviors and emergent behaviors are notoriously difficult to lock down to a single variable. I've been wondering for a while now if we are just running out of 'simple' problems to test and running out of techniques for simplifying problems. That at some point there is only hard stuff left, even by a contemporary definition of "hard".
I just want to state this as context, not as an invitation to tangent into an argument: I think general purpose AI is going to fade from consumer software again, as it has so many times before. But I suspect that some of the tools may find a home in areas where all of the problems are multivariate, and things like advanced techniques in linear algebra can help find signals in the noise when you can't control an environment.
I recall years ago someone discovering that chemo works better on an empty stomach, and not just for the obvious reason of not having anything to throw up. Normal cells in "starvation mode" absorb toxins slower, while many tumors ignore this signal. If we can nail down things like "If you have these genes and your serum vitamin D is > 120 and you fast for >8 hours and ingest 20-40 mg of caffeine an hour before infusion, tumor shrinking is increased by 40%" by mining through mountains of telemetry and then working backward from there to find the causal link.
Some of these measures are not really convincing, most obvious Nobel prices/paper, the rate of Nobel prices/year is fixed so when the number of papers goes up, the rate of publications that get a Nobel goes down.
Yeah, some of the measures were curious, such as unique keywords per 10,000 papers. It isn't clear to me what we should expect if science was doing fine -- more unique keywords total, yes, but per 10,000 papers? Why?
In general, I think the paper needs a stronger argument about what a null hypothesis should be and why that is violated.
This is especially true in math and physics. You will find that no matter what problem you can think of, either it has already been solved to the highest level of abstraction, or it's an unsolved and famous problem, or not worthwhile/trivial. Like, what about analogous of elliptic functions for non-elliptic integrals? already been done. The past century has seen a huge explosion of research into STEM subjects, from math, to physics, to biology, to computer science, etc.. Tens of billions of people people ever lived since the 1800s, and even just a tiny, tiny fraction of them are doing research, is still a huge amount of output. There just isn't much new ground to break, so this means discoveries will either be much more incremental or require considerably more mental horsepower.
Psychology is sorta the opposite: there is no limit to the number of experiments you can run on people or possible associations between causes and effects. It's not like psychology , literature, philosophy, or history has gotten harder over the past century, unlike math, physics, or economics. Sure, there are more advanced statistical methods, but running experiments hasn't gotten harder. This is also why the vast majority of physicists and mathematicians are teachers rather than researchers, and why econ papers have gotten much longer and are full of dense stats methods. There are always going to be new discoveries in biology and medicine, and same for applied math and applied physics, such as engineering or astronomy, but theoretical math and probably also theoretical physics are as saturated as can be.
> This is especially true in math and physics. You will find that no matter what problem you can think of, either it has already been solved to the highest level of abstraction, or it's an unsolved and famous problem, or not worthwhile/trivial.
This hasn't been my experience in fluid dynamics. There are so many places where one could apply a standard approach that people haven't. It takes some effort and experience to recognize that, but it's still true in my experience.
I think what you described is probably true for sexier areas like high-energy physics, but not true for more "boring" areas like turbulent flows. Yes, turbulence is "unsolved", but that doesn't mean that useful information couldn't come from applying standard approaches to particular turbulent flows. The variety of industrially-relevant flows is quite large.
I agree with you there are so many areas still vastly unexplored, fluid dynamics or the whole area of complex systems is definitely a prime example. Stephen Hawkins called his inaugural lecture "Is the End in Sight for Theoretical Physics?" suggesting that we are very close to the end of theoretical discoveries.
Although less famous I would argue that prediction was just as wrong as "640kB should be enough for everyone", lots of extremely fundamental discoveries had been made since then, and we don't seem to significantly closer to a theory of everything.
He discusses in (if memory serves) "A Brief History of Time" whether we will keep on discovering ever-more accurate physical theories, or if there's a theoretical limit. The Planck scale offers a theoretical limit, so there's hope.
At any rate, that is how I understood him. Not that we're nearly there, but that our current path of discovery seems finite.
To be honest I have not read his books, but I always had the impression from his talks that he was quite narrow minded in that he largely only meant astro/particle physics when talking about theoretical physics, ignoring many areas in the process. Also worth noting that the book came out 10 years after his lecture, so he probably revised his views during that time.
I think GP is conflating Maths with Pure Maths. While I’m sure there is new ground to break in Pure Maths, I would agree that it’s getting harder and harder.
However, as you note, in Applied Maths there are more unsolved problems than you could address with a million researchers.
A physicist or pure Mather social looks at Navier-Stokes and says, “Oh, we know how that works, nothing to do there I guess”, whereas the applied mathematician looks at it and goes, “holy cow, this could keep my entire department busy for the rest of our natural lives”.
> A physicist or pure Mather social looks at Navier-Stokes and says, “Oh, we know how that works, nothing to do there I guess”
I don't know if that's the best example, it's one of the Millennium Prize problems to prove that smooth solutions always exist to the Navier-Stokes equations. Pure mathematicians do, by and large, still consider proving the existence of things to be "something".
Maybe I'm misunderstanding what you're saying here.
He was simplifying for clarify, but what he said was just another form of what the person above him said which was:
>> no matter what problem you can think of, either it has already been solved to the highest level of abstraction, or it's an unsolved and famous problem, or not worthwhile/trivial. <<
So, Millennium Prize is an unsolved and famous problem like Reimann Hypothesis, etc..
Well. I wonder. You generalize a bit from maths and physics to all of STEM there, and throw computer science in there too saying there's not too much new ground to break. But that isn't what I see when I survey the CS literature. Instead it feels like important areas get neglected and ignored, whilst enormous herds thunder towards fashionable topics.
My guess is this slowdown is happening for a few reasons:
1. Increased number of researchers = increased team size = less innovation. The article shows increase in team size but doesn't ponder the implications. Teams shy away from bold ideas, in my experience. If you want innovation it has to come from individuals empowered to work alone and recruit slowly. The moment you're put in a team situation you are suddenly expected to pitch and convince others to take a risk on an idea that perhaps you aren't even sure about yourself yet, which is a high bar to meet. And the team won't want to try it because if it works the glory will associate with the individual who came up with the idea and drove it forwards, leaving the others in the shade. So teamwork puts pressure on people to propose 'safe' ideas that were found outside the group, which nobody will object to and which everyone can share equally.
NB: non tech firms struggle to create new tech partly for this reason. They have a culture of creating so-called innovation teams. This practice is rampant in finance for example. I never saw an innovation team do anything truly surprising. You could always guess up front what topics they'd be "researching" before learning anything about them because the range of topics was so narrow.
2. State subsidies. We know these kill worker efficiency. If that weren't true the USSR would never have fallen behind the USA in terms of wealth. What the article refers to as science is really academia, and academia is dominated by ever increasing amounts of government money. Whilst the article phrases this as science getting "harder" it can also be seen as researchers simply becoming less efficient than they were in the past, which is exactly what we'd expect to happen given that academia is a parallel planned economy. Efficient here means in terms of discovery production not paper production, of course.
3. Falling paper quality. Another way to view (2). I feel like half my HN comments are about this problem these days but the quality of papers in some research fields is staggeringly low, sometimes junk quality. As in, you could throw out 90%+ of the papers and the field would get better not worse. In a few fields like "social bot" research or epidemiology I'd struggle to name any recent papers that weren't intellectually fraudulent in some way. If you join a field as a researcher because you feel like it's an important topic, and then discover that the papers published in the last 10-20 years are much more likely to be non-replicable or have nonsense methodologies than those published 50 years ago, then you'll probably end up reading older papers because you feel you get more out of them. Then you'll end up citing them more often as a result.
I definitely feel I saw this when reading the epidemiology literature. Papers from the 1950-1990 period were quite different to modern papers. Way less fancy maths, much easier to read, more obvious and logical questions being asked and no WTF moments. You definitely got a feeling that the authors were intellectually curious and wanted to understand epidemics. From 2000 onwards the papers became nearly always useless.
A big part of this is the post-2000s era explosion in the use of advanced statistical methods and, especially, the acceptance of unvalidated models as "science". The creation of free tools like R and STAN made it much easier and so many papers now are just people playing around with R and random arbitrary datasets. They plot some regressions and publish a paper. Unvalidated modelling seems to have destroye...
"2. State subsidies. We know these kill worker efficiency. If that weren't true the USSR would never have fallen behind the USA in terms of wealth. What the article refers to as science is really academia, and academia is dominated by ever increasing amounts of government money. Whilst the article phrases this as science getting "harder" it can also be seen as researchers simply becoming less efficient than they were in the past, which is exactly what we'd expect to happen given that academia is a parallel planned economy. Efficient here means in terms of discovery production not paper production, of course."
This isn't actually true.
"For the first time in the post–World War II era, the federal government no longer funds a majority of the basic research carried out in the United States. Data from ongoing surveys by the National Science Foundation (NSF) show that federal agencies provided only 44% of the $86 billion spent on basic research in 2015. The federal share, which topped 70% throughout the 1960s and '70s, stood at 61% as recently as 2004 before falling below 50% in 2013."
But we're not talking about percentage shares, we're talking about relative changes and budgets rose significantly over the past 50 years. The increase in federal spending in the USA stopped between 2005-2015, but "university" and "philanthropy" continued rising (same problems more or less), and of course spending in other countries continued to rise. Also the rise restarted:
"Small boost this year completes 4 years of substantial growth despite Trump ... NIH's budget now stands at $42.9 billion, a 33% rise over its 2016 level of $32.3 billion. Similarly, spending by DOE science tops $7 billion, compared with $5.4 billion in 2017, a boost of 30%. NASA science programs rose by 8% and 11% in 2018 and 2019, respectively, before slowing in 2020 and 2021. NSF's budget, now nearly $8.5 billion, has grown the least among the four biggest federal science agencies. But even so, a 14% rise since 2017 compares favorably with an overall increase of only 4% during the second term of former President Barack Obama."
So these are massive spending increases, regardless of what industry is doing.
There's something else we need to watch out for here. Corporate research isn't famous for its high production of papers, but the evidence that "science" in general is getting harder is based on paper-related metrics. It's possible that science hasn't really been getting harder, but that corporate R&D has been sucking the best science out of the academic realm into a place where fewer papers are published overall, so there's less to cite. The fact that the most cited paper of all time is an obscure biology paper and that pharma spending is such a large component of "basic science", is suggestive of this.
> State subsidies. We know these kill worker efficiency. If that weren't true the USSR would never have fallen behind the USA in terms of wealth.
The USSR never did fall behind the USA in terms of wealth; it started very far behind when it came into existence in the ashes of czarist Russia, and caught up quite a bit through one of the most rapid industrializations the world has seen. Even after the initial phase of industrialization, from the 1950s throughout early 1980s, the Soviet economy grew faster than the US economy.
But other countries during the same period caught up and became first tier competitors, also rising from the ashes. Japan was nuked twice and by 40 years later was out-competing US industries. As the story goes, the USSR fell partly because Boris Yeltsin did a random 'spot check' of a US supermarket on a trip to visit NASA and couldn't believe what he saw.
At any rate, a full discussion of this might be off-topic, but let's say that it's a pretty uncontroversial idea in economics that a state run economy (like science) tends to be less efficient and produce lower quality outputs
It may be uncontroversial in economics, but it politics (at least in the US) central economic planning is always proposed as the solution for every problem.
For example, the current infant formula crisis was entirely the result of government management of production, distribution, and pricing of it. But the suggested fix is to have the government launch a criminal investigation of Abbot.
If biology is a science, some part of psychology is science. But I agree with the idea that many things that are called psychology have little to do with science. The first coming to mind is psychoanalysis. As a psychology student, I'm absolutely shocked at how much credit is given to charlatans.
> This is especially true in math and physics. You will find that no matter what problem you can think of, either it has already been solved to the highest level of abstraction, or it's an unsolved and famous problem, or not worthwhile/trivial.
I did not experience this in my 8 years in academic math (PhD + postdoc). Maybe it differs a lot from field to field, but mine was positively teeming with unsolved problems. More importantly, it's teeming with paths that will clearly lead to more unsolved problems (of all difficulty and importance levels).
In fact, the thing I miss the most from academia is probably helping new students blaze some of those new paths. Both for their sake, and as a way for me to discover new problems for myself.
In maths, this does seem very dependent on what work your peers and community recognize as progress. Maybe problems that are more and more trivial should be considered acceptable to publish, simply as being a contribution in filling in gaps. Or maybe the works of many more mathematicians should be tied together in larger contributions.
More scientist will mean that overall we have more time to spend in the dusty cobweb filled corners that have yet to be studied in fine detail. And while many of those corners contain nothing interesting. If even one of them contains something truly undiscovered it's worth it right?
Having been long in academia, what has increased is commercialization of all kinds of science, number of science managers (like modern day billionaires, taking credit for the work actually done by the graduate students and postdocs, growing to be huge), number of politicians putting their names in various papers (I see managers co-authoring on average a paper per week), brutal competition for status and citations, excellent writing and formalism with little or no substance, rise of administrators with large incomes, gangs and tribes accepting each other grants and papers, intensified politics on awards/recognitions/invited talks and control of main publications, people constantly chasing the same topical subjects for grants and industry relevance, and similar activities. The science component has actually decreased in my view. The system and practices sometimes remind me of the Wall Street, with the currency being fame.
I don’t think this system will last for too long. The environment is increasingly filled with status hungry people. Balaji has a good prediction on that.
I agree, modern academia seems to be more about meta-skills of gaming the system and playing politics rather than who is the best at actual research. Probably why so many potential academics just go into private industry
>I don’t think this system will last for too long. Balaji has a good prediction on that.
He addressed this topic in a podcast around two years ago (he has too many podcasts to find it).
He was asked about his views on academia vs industry, having been in both himself. He said capable people in academia will gradually realize that it’s better to start companies or do the same work in industry, rather than writing these grants.
I think we underestimate the amount of politics that was going on in science previously.
Really compared to the shenanigans with Newton, Leibnitz or Hooke and Newton, much of what we see today is quite harmless. Or take for a more recent example Teller and Oppenheimer. I think in science (but not only in science) we tend to idealise the old days and scientists, because we remember the discoveries not the politics and their personal flaws. I would even argue that this greatly demonstrates the success of the scientific process. In the end the best science won, despite all the political infighting.
I often wonder about this, and have talked about this all with older colleagues. My sense is some things were the same, but others were really different.
Even looking at the peer review process: it was much more informal in the distant past, and maybe more akin to what happens today with invited papers in my experience, or with small open source software projects.
There were certaintly very petty but intense personal squabbles in the past but that's a bit different from the pervasive structural issues in science today.
At some level too, I don't care what it was like 100 years ago. I can still see the problems today.
Peer review really was different in the past. Hell, Nature even used to occasionally accept papers without any review at all until the mid-1970s, if the editor thought they would provoke good discussions.
Watson and Crick's famous 1953 paper where they presented the double helix of DNA is one such example.
Yes I didn't mean to say that there are no issues in science. I disagree that politics and personalities are a big part though. I think we have huge structural issues, requirements and pressures are increasing constantly, the way into science takes much longer and much talent is burnt out in the process, current funding favors short term "low risk" research and disincentives trying out new things or changing direction/field...
My sense is the structural issues amplify the political and personality issues, in the sense that when those things arise they can maybe have bigger effects than in the past. But that's just speculation.
I wish things like the research in this blog post got more attention, because it's important in understanding how things have and have not changed.
Science continues to work, but the work of being a "scientist" really sucks because of this stuff.
I personally left science because I could see it had 98% overlap with what I'd do as an engineer (or entrepreneur, in the case of a PI), with a tiny fraction of the monetary benefit. Being a winner in that system is a great life, but it rests upon a huge pyramid of people who are working their butts off at low pay and lower prestige, for a brass ring that few will ever grasp.
I agree with you. Being a scientist has become worse (although I would argue that is true for many other jobs as well). A PI nowadays really works like the leader of a small startup constantly looking for money to keep the ship afloat without the prospect of a big payout at the end.
> A PI nowadays really works like the leader of a small startup constantly looking for money to keep the ship afloat without the prospect of a big payout at the end.
I said exactly the same thing when I first left academia for tech, and people would look at me suspiciously and ask why I'd ever do such a thing. I looked at every PI I knew, and saw a task essentially identical to running a startup, but with no financial upside -- and the downside risk of being jobless and adrift if you don't get tenure.
What I spent my day doing was dealing with managerial tasks, fiddly stuff for funding applications, etc.
I got to actually code for a whole day last week, and it was marvelous. It was also only possible because it was an all-hands-on-deck, everything else can wait situation.
I spent like 10 total hours writing analytical code in 2020; maybe another 100 hours of analytic design. The rest was meetings, politics, coordination. I got a national award and millions in funding. I would trade it for more time playing with the puzzles. Do I feel good that my people got published and covered rent, hit major life goals (marriages, home purchases)? Yes. But, man, do I wish I got to do more science…
Weren't scientists of old after endowments from kings and royalty, too? Further, I seem to recall a Vertiasium about the guy who solved the cubic, fighting for a position at a college by dethroning another professor through an academic duel.
Teller touches upon this subject. In an interview published on YouTube, he says, increasing scholarships and science funding further doesn’t help; because money doesn’t buy science, money buys technology; we actually don’t have enough scientists on whom to spend the money. And that, most of the advances in quantum mechanics and atomic physics were made by individuals genuinely interested in problems in these domains, and worked in modest conditions. Quantum mechanics theory took practically nothing to develop.
Politics, of course, is part of the humanity, but it manifests differently in different environments (academia vs industry, or then vs now). There is undoubtedly politics in industry, however, there are also products and reality checks. The companies won’t survive if they operate entirely based on politics. In academia, on the other hand, the environmental feedback can be weaker depending on the subject; there, politics could come to the fore. Amusingly, Kissinger said, politics in academia is specially vicious precisely because the stakes are so low.
As for then versus now, there used to be a lot of low-hanging fruit that could be taken until 1980s, and there weren’t many researchers. But now there are fewer accessible opportunities, with far more researchers. If you consider that the size of the cake has not increased proportionally (due to factors mentioned in the article), while the number of players has increased substantially, you may conclude that the environment has become much more competitive. Increased competition indicates increased politics, emphasis on selling, networking, and other non-scientific factors relevant to industry, except there you actually get paid. The conditions are nowadays totally different. There all sorts of performance metrics (H-index, citations, number of papers etc), that have become target, in place of good science, which are gamed.
I can't decide whether Teller's head is stuck up the rafters of the ivory tower he lives in or whether it is held up where the sun does not shine. There are so many students in academia searching for the magic path through graduate school where you can get a permanent role either in teaching or tenureship (research) that it is becoming a crisis for both STEM fields and otherwise. It is easy for him to preach when he is one of the few that have "made it" through a combination of intellectual pedigree, geopolitical necessity, and the plucking of low hanging fruits in an emerging field.
Yeah I really don't understand what he's saying there - where does he think his salary comes from? Money buys those interested individuals time to pursue that interest, and the more of it we have the more people we can put in the right time and place to find the next big thing.
I think the key point is the "and worked in modest conditions", emphasis on modest.
Injecting a ton of money creates perverse incentives and attracts people who are not interested pursuing something out of curiosity and interest, but for the money.
I think it probably creates a hostile environment for people who are there out of pure interest, because if there is a way to earn more money/status/power/whatever by doing something other than good work people will do it. So you effectively get punished for doing what you're supposed to instead of playing the game.
Academia is a career, not some sort of side project for bored scientists. We don't measure the competence of developers by the time they spend on side projects outside of work. You don't have to be full of visionary ideas to do research, and talent is everywhere. Teller would't be where he was had he been born in South America or Africa.
If you only give money and positions to people with "passion" when there is a clear demand for more, that's called exploitation. Maybe consider becoming an aerospace investor instead, that field is full of talent chasing after lost dreams for pennies. If you are a software engineer on this forum, you know damn well what your net worth is in the current North American market compared to the other legacy engineering fields. It is patronizing to expect the other scientific fields to be worth "less" than CS.
> Injecting a ton of money creates perverse incentives and attracts people who are not interested pursuing something out of curiosity and interest, but for the money.
That's just an excuse to exploit people. The same nonsense as with teachers. "We want teachers that are passionate, not there just for the money". Bah.
In the end we're trying to get stuff done, and it doesn't really matter whether the pipes are kept clean by somebody really passionate for sanitation, or somebody who just thinks the pay is good enough to tolerate the smell.
The argument does not hold any sort of deeper inspection. The greats of quantum mechanics, people like Bohr, Heisenberg, Schrödinger etc. we're all professors, which at the time put you up in the highest ranks of the middle class or the lower upper class + the status of the title it implied in Germany for example. They were amongst the highest paid salaried employees. In comparison now a typical professor salary is lower than what many junior software developers get straight out of undergrad. And that is after years of job uncertainty through PhD, postdocs and tenure track. Essentially the only ones left do out of idealism. (I also believe this long uncertainty is a big factor in the large gender imbalance, women are much less tolerant of this then men)
The other irony is that Teller himself was part (came out) of one of the biggest counter examples. The Los Alamos project pushed unprecedented funding into science, and it let to many discoveries that would have taken decades longer otherwise.
Since you specifically mention Germany: professors are incredibly well paid in Germany and also enjoy high social status. Of course, a lot of them don't do any science, they are the CEO equivalent of an institute.
Salaries for PhDs and PostDocs are a different story of course. They are alright in theory, but often you can only get a part time job, which in reality means a > full time job with a part time salary.
I was specifically talking about the days of the founders of quantum mechanics or even the times of Teller et al.. Today professors are in fact not that incredibly paid anymore in Germany. Sure they still have reasonably high salaries but compared to junior developers they earn approximately the same. Regarding PhD students yes depending on field, you might get a half position, however I'm not aware that postdocs get low paid positions. Compared to the US differences between professor and PhD salaries are much lower AFAIK.
> because money doesn’t buy science, money buys technology; we actually don’t have enough scientists on whom to spend the money.
I can only speak from personal experience and observations in my environment, but to me this statement sound like a dark, cynical joke.
There is plenty of scientists who would love to work on interesting things, are hired as scientists, but once they are hired, no one actually lets them work on scientific problems.
this is restricted in physics, e.g., life sciences need as more as they can get money, they even do not have enough money to pay assistant, there are so many species waiting for identifying, even use old protocol, scientists still need to pay salary, specimen storage fees...
Off topic but I think movies and TV shows are suffering the same problem. We have exhausted every way of telling good stories. Nowadays if you want to give the audience "something new" or "something they've never seen before", the only way is to increase craziness and intenseness, which is the opposite of good stories.
I think that's less a question of having ways to tell good stories and more one of a creative space being exhausted. There's a desire in a long-running series to keep raising the stakes as the series progresses, but doing so reaches a point where there's nowhere left to raise the stakes. I think that's been a bit of the problem with Doctor Who in the last few years. As they've kept raising the stakes (right up to the existence of the universe), there's nowhere left to go.
This should worry everyone. Economic progress is intimately tied to scientific discoveries. And if that’s slowing down, we may still see some business model innovation (think: Uber), but less progress that increases humanity’s collective wealth.
Making matters worse, population count in highly educated countries is shrinking, inevitably leading up to fewer brains trying to solve hard problems than before. For now, I don’t see the remaining countries with a growing population making up for this loss in collective cognitive capacity devoted to science.
Perhaps we shouldn’t fear universal AI as much as we should cheer it on as the future saviour of science.
There is less chance of escaping past mistakes, or more chance that a mistake will ruin you. There are more things you have to stay on top of benefits and finance wise. There are more options for many things, which leads to more analysis of what's best. The systems we use every day, from cars to computers, are more complex.
At the same age? Your comment just sounds like someone that is getting wiser with age. I agree to all you said, but I think it's because I am wiser with age. Is life really getting harder at the same age and at the same opportunity? Man, this is hard question to ask because you are looking at the rear view mirror.
It seems like an objective increase vs prior generations. Just look at stuff like child anxiety, how much different school is for kids today (more complex), and how many more financial instruments exist today (401k, 403b, IRA, Roth vs std, HSA, FSA, tax complexity, etc.
Agree with this. There is a lot more information to process, and it's being generated at greater and greater rates. Ideas that used to be an entire paper in the 1970s have now become an extra paragraph in an undergrad textbook from the 2020s. Further and further specialization in every field--e.g. in SWE: SysAdmin branched out into DevOps and SRE, Data Science branched out into Data Engineering, Analytics Engineering, ML Engineering, etc. It seems we have built more efficient tools, but the barrier to entry to use them is taller; and once you can use them they seem to require longer and longer hours of tinkering and supervision. Yet if you don't put in the hours your tools/systems start to decay simply by virtue of progress[1]. Tool-integreation costs are also higher: new tools don't exist in a vacuum but rather need to be adapted and vetted by the host system.
Somewhat related, but I believe this is the reason why academia will continue to lag behind industry. Academia has a bias for fundamental knowledge, but as information grows, fundamental knowledge represents a smaller proportion of all knowledge[2]. Anecdotally I have observed a 2-5 year gap between innovation in industry vs academia (CS). Academia often far behind looking to monetize the success of industry via newly minted formulaic programs. This was not always the case, e.g. databases came mostly from academia into industry in the 1970s, but nowadays innovation in the field seems industry-driven. To be fair, perhaps academia has already explored much of the relevant concepts, and it's easier to innovate by way of implementing these same ideas in different programming languages and business models.
[1] What are we left with then? More efficient but also less robust systems? Being more efficient, i.e. generating more output, seems to be the chosen trade-off today, when you can afford extra human intervention, but will it ever be the case when we reach an inflexion point and economics favor robustness/stability over output?
[2] Not to mention the issues with defining something as fundamental. Should fundamental knowledge be frozen in time, or instead act a sliding window? I.e. when, if at all, does something go from fundamental to obsolete? and when does something go from innovation to fundamental? Is it merely popularity driven?
I've used as a general theory for why change in the world seems to be slowing down, since although it's stayed constant, it has slowed down on a % basis.
This should be apparent to anyone who observes fundamental physics.
The 20th century saw massive increases in knowledge with relativity and then quantum mechanics (ie QM/QFT/QCD). There have been accomplishments in the 21st century but a lot of then (eg the Higgs boson) were confirming things theorized 50+ years ago. Despites decades of trying we're still to find a crack in the Standard Model. Yes we've eliminated a lot of options and that's important in the process of elimination but we don't really seem to be any closer to resolving the fundamental contradictions of QM and GR.
There's a fun list of unsolved problems in physics [1]. My favourite is the so-called vacuum catastrophe [2] where the predicted (from QFT) and actual vacuum energy diverge by roughly 120 orders of magnitude. At the other end of the spectrum is the magnetic moment prediction vs actual [3] that differ by less than 10 parts per billion.
I'm actually losing faith we'll see meaningful progres sin a lot of these areas in my lifetime and I never used to think that.
I'm a scientist, though not a leading researcher by any means. But when I'm tempted to think that science is hard, I think about the science in the past and what primitive tools they had: No computers before 1950. No electronics before 1900 (all dates give or take). No calculus, etc. Al Khwarizmi had no equations! And when algebra and calculus came out, you practically had to be a philosopher just to grasp them. Now they're taught to schoolchildren.
There are some things to think about with respect to the data analysis presented here:
1) On number of papers published, citation value, etc. There may be an inflation in the numbers of PhD's granted for little more than serving as a PI's lab tech for a few years. That requires a few publications. Often the publication count is inflated by splitting a paper that would be more coherent as a single publication into several publications to fluff a CV. Such papers may not get many citations. This is about politics and careers, not about the quality of science being done.
2) Universities are in the market trying to patent discoveries and license them to big corporations. This is very bad for basic scientific discovery in many fields, as funding is directed away from those fields. Basic science underlies later applied science applications. Shutting off basic research is killing the goose that laid the golden eggs, but the administrators want those sweet patent license payments more than anything else.
3) People with the potential to do difficult science and math, but who happen to come from a background where they can't afford college without large loans, are much less likely to go into academic research than in prior eras in the USA when college was less expensive. Many would like to do research but the economics of current education systems are incompatible with that goal.
4) Some areas of research are simply not being funded anymore due to pressure from entrenched interests in areas like energy, pharmaceuticals, etc. Renewable energy research is a threat to fossil fuel interests. One-shot cures for diseases like HIV/AIDS are a threat to the profits of pharmaceutical companies selling lifetime maintenance programs. Research into environmental pollution is a threat to a wide variety of interests, as is research into declining biodiversity.
The corporatization and financialization of literally everything is really out of control, and these are the side effects.
Although we could try to use science to prove that science is harder, it seems pretty clear just through intuition that it must on the average be harder. Inventing calculus was amazing, but it's pretty simple compared to what people called advanced math today. Imagine trying to make an interesting discovery in math today compared to adding something to number theory 300 years ago.
There might be a lot of space for applied math/physics, but perhaps this is closer to engineering than research though the distinction is often blurry.
The thesis of the article is plausible, but I think it’s missing the broader perspective.
It makes sense that science picks “low hanging fruit” early on. Then, on average, later discoveries require more effort.
But the rate of progress depends on both how much effort a discovery takes AND how much effort is available. The progress of society has made it possible to aim exponentially more resources at solving problems. Computers let us automate things. Medicine & farming mean more humans can do higher value work. Better politics means fewer people dying in wars.
So I don’t care if each discovery takes more effort than the last. As long as we get exponentially more resources to go along with it, we can keep creating exponentially more knowledge for a looong time.
We are in need for new metrics beyond citation count or quantity of published papers. A century ago, there was more content in each paper than it is today. This is mainly related to the incentives that exist today to publish, e.g. for getting a raise in salary or getting a getting a new academic position. I like to cite older papers because I can reference them multiple times in a single manuscript, reducing the number of references (e.g. below 60). If I would cite new papers, results are spread across 10 or more publication what was once published in a single paper.
I haven't found a fair way to assess quality of papers - I think it depends highly on individual author motivations. If we are able to identify the right patterns among authors' motivations, it may be possible to come up with a better rating system for papers published. Current rating systems focus too much on individual papers/journals, and too less on authors.
I have a feeling whatever metric you define will just create different problems. The high emphasis on metrics itself is the problem, because quantifying someone's research quality is not at all trivial.
To your other point: I get what you mean, and I don't even disagree, but I could also make the exact opposite point. Papers today often are these big, overly structured manuscripts that actually cointain a multitude of ideas so that it can become hard to see what exactly the point of the paper is. In the past, papers often were much more atomic. They presented one idea with some relevant context. There was no need to prove you had surveyed the entire field just to present an idea. No useless context, more relevance, more focus.
When a field is young, it is simpler to make important discoveries, so in that sense, science is (obviously) getting harder.
But most of the graphs can be explained by the sheer number of papers being published.
Academics have the perverse incentive to publish as many papers as possible, instead of discovering as many interesting things as possible, because items published is way easier to measure than real progress made.
Therefore there is a significant paper inflation problem. People might today write 10 papers with LPUs (least publishable unit) with stuff that would have been a single technical report 30 years back.
This is much worse than it sounds, because it affects productivity significantly in subtle ways: it takes effort to publish those 10 papers, it takes effort to read up and review the 10x papers of others, and it takes time to think about LPUs instead of concentrating on the bigger picture.
Unfortunately, due to the way the research system is set up, it is not likely things will improve.
149 comments
[ 2.9 ms ] story [ 30.8 ms ] threadIn my field, the latest papers are a bit "click-baity". They have interesting titles with many trending words but when you read them it is hard to find either a coherent vision or a single result that one can build on top of.
Of course, I am generalizing. There is great research being done today and low-quality papers in the past. But it is completely true that the need to publish has distorted the goal of research communication.
Nothing worth doing is easy. We have cleared the low hanging fruit and can now do the actual work. This is so exciting :)
(also I wonder sometimes if as a society we are teaching people less to be critical thinkers, who are not afraid to disagree with the mainstream)
Breathing is pretty easy.
But more seriously, I think what you said only applies in iterations of revolutions. A new breakthrough (e.g. new physics, new invention) could create a lot more low hanging fruit.
I think a major problem is ever since general relativity, all of our (existing and potential) new revolutions are very abstract and hard to relate back to human perception.
Take dark matter/energy for example - if we get a better model to understand that, it will revolutionize how we think about the universe, but it will (hopefully) have less impact on day-to-day society than the nuclear model of atoms did.
>Take dark matter/energy for example - if we get a better model to understand that, it will revolutionize how we think about the universe, but it will (hopefully) have less impact on day-to-day society than the nuclear model of atoms did.
Is unrelated to:
>I would argue that general relativity barely affects humans day to day lives, whereas the nuclear model of the atom completely changed the global political system.
Maybe to help you, I was using General relativity as a time marker (i.e. it was the first sea-change scientific discovery that had limited impact on day-to-day human life)
You then argued that my premise was incorrect, because before general relativity was formalized, people could have believed that all future discoveries were too abstract to affect day-to-day human life.
I then argued that you were correct in that general relativity in fact was too abstract to affect humans' day-to-day life.
Then you made a personal attack against me.
Now we are here.
Also, many major historical scientific breakthroughs have been just a smart (usually rich) dude/dudette with a lot of free time and not a member of the conventional 'academic' system.
So somehow figure out a way for public to get in on the research instead of government restrictions on who can purchase supplies which limits research to the .0001% who have PhD's.
This opens the door to a lot more mind power going into research.
See the book "the structure of scientific revolutions".
Most scientific breakthroughs through out history were just private citizens outside of the conventional academic establishment.
We need to put an end to these government restrictions on buying supplies and equipment which keeps the research down to just a tiny fraction of the population.
Allowing private citizens to do research would put a fire under those old codgers asses.
I have a few published papers in peer-reviewed journals with 30+ citations and I'm a college dropout, but I run a company that can fund/produce this kind of research as part of our mission statement. It's not really restrictions, but rather money/funding, and willingness of private companies to publish publicly (which is a big blocker for most).
If I, with my current education, was sent back 200 years... I could probably do OK with inventing stuff (engineering education, so I have some good answers backed by a nice model, much of which I'd have to work backwards to derive because who remembers 200 level engineering classes?). It took Marconi a while to realize that he had to ground the antennas, so there's a pretty good discovery for free.
On the other hand, if I was actually born then... I think I'm a pretty clever guy, but not a genius. Someone like me 200 years ago would probably... I dunno, be trusted to fix mill equipment. Or maybe work on watches, depending on where I was born. Maybe I'd be a town pharmacist or something.
When the R&D workforce expands at the rate it has over the last few decades you will invariably see less talented and less talented people entering the field. Also, various forms of overhead will grow with the size of the workforce.
As an example about 20 people attended the 1911 Solvay Conference on Physics. How many attend a conference today?
With the growing size of the R&D workforce also comes the most destructive force in any human endeavor: politics.
Volumes goes up, quality goes down.
Also, an average Earthling is now much richer than before. A random Korean or Turk of 1911 had much smaller chance to study physics, much less money to even buy a ticket to travel to the Solvay Conference. Both is now easier and cheaper.
On the downside, there is a lot more prestigious and well paying jobs today. Instead of tackling physics, best minds of today may be sitting in Facebook, trying to discover yet more ingenious ways to push ads on the rest of us.
I imagine you were being tongue-in-cheek, but just to unreasonably latch onto that statement a bit: politics is certainly infuriating, and it's easy to treat it as some unalloyed Bad Thing that we wish would just go away, but politics is essentially communication, and so it can be a good thing, too. The bigger the group, the more communication you need to keep everything (somewhat) together.
For instance, on a larger "human endeavor" scale, I am grateful that WW3 has not happened yet — thank you politics (:
As with many things, it's a tool that can be used poorly or used well. But the tool itself is not inherently bad or destructive, I don't think. Or maybe it is both inherently good *and* bad, and you can't entirely separate the two.
"Politics" in this context means a form of communication, consensus and decision making in a group.
Perhaps it is impossible to nip talk of the latter type of politics in the bud, though.
My intent in bringing up the issue originally was that when you have larger groups of people interacting, you get larger-scale patterns of communication. People form groups and sub-groups; people give support to groups based on trust of that group or one of its representatives, rather than getting to know everyone individually; higher-level conglomerations of ideals form because tracking every issue in detail is too much, etc.
There's a lot of gunk that comes along with that. But it is not all bad, or destructive. It can even be _con_structive.
I think that generally applies to institutional politics as well as national/world politics.
People don't (usually) get pissed off about idle chitchat. They do when their project is spiked.
Economics is about resource allocation using money and markets. Politics is about resource allocation using... influence? Connections? Coercion?
If we had an open, transparent market where sponsors could bid on research, it might work better. (Hey, HN: Anybody want to set up such a market, funding it by taking a small cut of the bid?)
Of course we can only speculate, but to me it seems self evident that there would be effectively 0 overlap. The issue is that politics invariably turns into a viral team sport where people pick some side, adopt that side's views wholesale, convince themselves they're the most important thing ever, and then set out to convert everybody else to their team, beat the other teams in any way possible, and of course always fanatically cheer on their own team regardless of whether or not its deserved.
And then when things get hot enough, you get violence. Politics is precisely what started WW1. An otherwise irrelevant Archduke was assassinated and then that politically snowballed into everybody killing everybody everywhere, because politics. Then we chose to engage in extreme political myopia imposing absolutely harsh punishments on the losers of the war which, shockingly enough, didn't really lead to them rejoining the "world order" but instead declaring war on it again, and nearly winning.
And the only reason we haven't yet had WW3 is nuclear weapons. War doesn't really work when you can guarantee that your country (and you, for that matter) will most likely not exist at the end, "win" or lose. But now political frenzies are overriding even that most basic aspect of self preservation and inching us closer to WW3 than we've ever been. So no, I do not think he was being tongue in cheek.
I realize "politicking" is often used in a derogatory sense, and perhaps that's how you mean it here, but there is a whole lot of plain-old-politics that takes place on a regular basis which has good overlap with those issues you hinted at. Things like: planning and building infrastructure, making sure people do not go hungry or thirsty and that they have places to live, having common laws, etc. A list kind of like "what have the Romans ever done for us?"[1].
Certainly, politics/politicking can also have an amplifying effect on the bad stuff too, as you mention.
I suppose my main issue with what you say is the bit starting: "politics invariably turns into a viral team sport ..." I don't see it as invariable. The pendulum has been swinging that direction recently, and I don't like it either, but I don't see it as a one-way road to doom.
Though I respect if you do feel that way, of course!
----
Aside:
Your comment led me to read a bit about history, and I found on Wikipedia[2]:
"According to a 2021 study, Franz Ferdinand's absence was key to the breakdown of diplomacy and escalation into war, as Ferdinand had been the most powerful and effective proponent for peace in Vienna."
Like you, I had the "an otherwise irrelevant Archduke" idea in my head, but I guess now I'm not so sure. So, thanks for the indirect new info (:
[1]https://www.youtube.com/watch?v=Qc7HmhrgTuQ
[2]https://en.wikipedia.org/wiki/Assassination_of_Archduke_Fran...
"I’m claiming that science is getting harder, in the sense that it is increasingly challenging to make discoveries that have comparable impact to the ones in the past".
So, unless you're arguing that there aren't an equivalent 20 people alive today as prestigious as the 20 people that went to the 1911 Solvay conference. Then the fact that there are greater percentage of less talented people now, shouldn't make it harder for the greater number of talented people to make "discoveries <of> comparable impact". (With some caveats of course.)
The Solvay conference was invite only, but I think your point still stands.
The focus is quite different in private companies and research institutions.
It is not obvious that fundamental research does not “increase our collective wealth of lead to some useful application”. Just not in the short term, usually. Despite a lot of economics theories, companies being directed by their self-interest or that of their shareholders does not lead to good outcomes over the long term. Someone needs to do it, who’s not going to be limited by ROI over the next 5 years. In practical terms, this can only be public funding.
Taxpayer-funded institutions justify it by pointing out that if there are short-term applications, companies tend to already do the R&D themselves, therefore public money is not needed to have good outcomes. The alternatives are:
- research in private institutions with public money (the big pharma model), which has obvious ethical issues when said private institutions make billions off of publicly-funded research.
- public institutions funding short-term research through things like patent licensing, which is quite impopular (there is a strong argument that public institutions’ output should be public; governments are not corporations).
- governments not funding research at all, which quickly leads to countries falling hopelessly behind as you cannot count on corporations to give a fuck about public good.
Overall, it seems like a good idea to let everyone do what they do best: corporations focus on bringing stuff to market and applications developments, whilst institutions with a longer outlook focus on things that could be used in applications 20 to 100 years from now.
Deep learning is beyond what VLSI was for the transistor, for example.
But even STEM research itself wastes too much time with costly self-serving objectives, rather than shooting for breakthroughs that lead to actual applications. Take CERNs large hadron collider, which has produced preciously few new insights, despite costing a fortune in taxpayer money. The Higgs boson is all well and good, but it’s hardly a new finding.
Or take mathematics’ famous millennium problems, where only a subset (eg. P vs NP) would lead to practically useful new insights. By contrast, solving some obscure numbers theory problem would benefit humanity how exactly?
For a few decades, the prevailing school of thought in public administration has focused on justifications. You must always justify how the public will benefit from the proposed use of money. Then you must report that you used the money for justified reasons. The administrators will also audit you to ensure that you didn't use the money for frivolous purposes.
As a consequence, you can't get funding for studying something you find interesting. Instead, you must always justify how your research will benefit the public. If you manage to get funding, you must spend a lot of time reporting how you used that money and what it did enable you to do. Those reporting requirements are also one of the major reasons for the administrative bloat in the academia.
The general public does not want to fund BS research, and science stagnates largely as an unintended consequence of that.
Having a position that doesn't require me to peg everything to a practical deliverable has done wonders for my ability to actually chase down ideas.
When was this golden age in the past when the world was magically easier and “better”? I see complaints about social media brainwashing people but what is religion and nation state populism?
I have a hard time taking contemporary opinions on “life is getting harder” seriously or sincerely. It’s just another clickbait trope.
No one wants to tax rich people who are of the age to have benefited from an era of high taxes. We let them pull the ladder up behind them.
There is no real reason to follow their orders or coddle their sensibilities. What are they going to do? Instigate a civil war from their Lay Z Boy? 60-70% of the population want to reverse their political policies across the spectrum of contexts. Just do it.
Maybe just spit balling. Maybe it isn't that science has gotten harder, but science needs more engineering support than what it currently has to develop the technologies to allow more a more rapid discovery.
Also any physics paper you read today is about one of those major discoveries made over past 120 years, except maybe extromagantism or optics
I just want to state this as context, not as an invitation to tangent into an argument: I think general purpose AI is going to fade from consumer software again, as it has so many times before. But I suspect that some of the tools may find a home in areas where all of the problems are multivariate, and things like advanced techniques in linear algebra can help find signals in the noise when you can't control an environment.
I recall years ago someone discovering that chemo works better on an empty stomach, and not just for the obvious reason of not having anything to throw up. Normal cells in "starvation mode" absorb toxins slower, while many tumors ignore this signal. If we can nail down things like "If you have these genes and your serum vitamin D is > 120 and you fast for >8 hours and ingest 20-40 mg of caffeine an hour before infusion, tumor shrinking is increased by 40%" by mining through mountains of telemetry and then working backward from there to find the causal link.
In general, I think the paper needs a stronger argument about what a null hypothesis should be and why that is violated.
Psychology is sorta the opposite: there is no limit to the number of experiments you can run on people or possible associations between causes and effects. It's not like psychology , literature, philosophy, or history has gotten harder over the past century, unlike math, physics, or economics. Sure, there are more advanced statistical methods, but running experiments hasn't gotten harder. This is also why the vast majority of physicists and mathematicians are teachers rather than researchers, and why econ papers have gotten much longer and are full of dense stats methods. There are always going to be new discoveries in biology and medicine, and same for applied math and applied physics, such as engineering or astronomy, but theoretical math and probably also theoretical physics are as saturated as can be.
This hasn't been my experience in fluid dynamics. There are so many places where one could apply a standard approach that people haven't. It takes some effort and experience to recognize that, but it's still true in my experience.
I think what you described is probably true for sexier areas like high-energy physics, but not true for more "boring" areas like turbulent flows. Yes, turbulence is "unsolved", but that doesn't mean that useful information couldn't come from applying standard approaches to particular turbulent flows. The variety of industrially-relevant flows is quite large.
Although less famous I would argue that prediction was just as wrong as "640kB should be enough for everyone", lots of extremely fundamental discoveries had been made since then, and we don't seem to significantly closer to a theory of everything.
At any rate, that is how I understood him. Not that we're nearly there, but that our current path of discovery seems finite.
However, as you note, in Applied Maths there are more unsolved problems than you could address with a million researchers.
A physicist or pure Mather social looks at Navier-Stokes and says, “Oh, we know how that works, nothing to do there I guess”, whereas the applied mathematician looks at it and goes, “holy cow, this could keep my entire department busy for the rest of our natural lives”.
I don't know if that's the best example, it's one of the Millennium Prize problems to prove that smooth solutions always exist to the Navier-Stokes equations. Pure mathematicians do, by and large, still consider proving the existence of things to be "something".
Maybe I'm misunderstanding what you're saying here.
>> no matter what problem you can think of, either it has already been solved to the highest level of abstraction, or it's an unsolved and famous problem, or not worthwhile/trivial. <<
So, Millennium Prize is an unsolved and famous problem like Reimann Hypothesis, etc..
If it works well, you get to push clinical science forward.
Then about half the time, "This seems like a boring classification problem" turns out to be way harder than expected.
My guess is this slowdown is happening for a few reasons:
1. Increased number of researchers = increased team size = less innovation. The article shows increase in team size but doesn't ponder the implications. Teams shy away from bold ideas, in my experience. If you want innovation it has to come from individuals empowered to work alone and recruit slowly. The moment you're put in a team situation you are suddenly expected to pitch and convince others to take a risk on an idea that perhaps you aren't even sure about yourself yet, which is a high bar to meet. And the team won't want to try it because if it works the glory will associate with the individual who came up with the idea and drove it forwards, leaving the others in the shade. So teamwork puts pressure on people to propose 'safe' ideas that were found outside the group, which nobody will object to and which everyone can share equally.
NB: non tech firms struggle to create new tech partly for this reason. They have a culture of creating so-called innovation teams. This practice is rampant in finance for example. I never saw an innovation team do anything truly surprising. You could always guess up front what topics they'd be "researching" before learning anything about them because the range of topics was so narrow.
2. State subsidies. We know these kill worker efficiency. If that weren't true the USSR would never have fallen behind the USA in terms of wealth. What the article refers to as science is really academia, and academia is dominated by ever increasing amounts of government money. Whilst the article phrases this as science getting "harder" it can also be seen as researchers simply becoming less efficient than they were in the past, which is exactly what we'd expect to happen given that academia is a parallel planned economy. Efficient here means in terms of discovery production not paper production, of course.
3. Falling paper quality. Another way to view (2). I feel like half my HN comments are about this problem these days but the quality of papers in some research fields is staggeringly low, sometimes junk quality. As in, you could throw out 90%+ of the papers and the field would get better not worse. In a few fields like "social bot" research or epidemiology I'd struggle to name any recent papers that weren't intellectually fraudulent in some way. If you join a field as a researcher because you feel like it's an important topic, and then discover that the papers published in the last 10-20 years are much more likely to be non-replicable or have nonsense methodologies than those published 50 years ago, then you'll probably end up reading older papers because you feel you get more out of them. Then you'll end up citing them more often as a result.
I definitely feel I saw this when reading the epidemiology literature. Papers from the 1950-1990 period were quite different to modern papers. Way less fancy maths, much easier to read, more obvious and logical questions being asked and no WTF moments. You definitely got a feeling that the authors were intellectually curious and wanted to understand epidemics. From 2000 onwards the papers became nearly always useless.
A big part of this is the post-2000s era explosion in the use of advanced statistical methods and, especially, the acceptance of unvalidated models as "science". The creation of free tools like R and STAN made it much easier and so many papers now are just people playing around with R and random arbitrary datasets. They plot some regressions and publish a paper. Unvalidated modelling seems to have destroye...
This isn't actually true.
"For the first time in the post–World War II era, the federal government no longer funds a majority of the basic research carried out in the United States. Data from ongoing surveys by the National Science Foundation (NSF) show that federal agencies provided only 44% of the $86 billion spent on basic research in 2015. The federal share, which topped 70% throughout the 1960s and '70s, stood at 61% as recently as 2004 before falling below 50% in 2013."
https://www.science.org/content/article/data-check-us-govern...
https://www.science.org/content/article/massive-2021-us-spen...
"Small boost this year completes 4 years of substantial growth despite Trump ... NIH's budget now stands at $42.9 billion, a 33% rise over its 2016 level of $32.3 billion. Similarly, spending by DOE science tops $7 billion, compared with $5.4 billion in 2017, a boost of 30%. NASA science programs rose by 8% and 11% in 2018 and 2019, respectively, before slowing in 2020 and 2021. NSF's budget, now nearly $8.5 billion, has grown the least among the four biggest federal science agencies. But even so, a 14% rise since 2017 compares favorably with an overall increase of only 4% during the second term of former President Barack Obama."
So these are massive spending increases, regardless of what industry is doing.
There's something else we need to watch out for here. Corporate research isn't famous for its high production of papers, but the evidence that "science" in general is getting harder is based on paper-related metrics. It's possible that science hasn't really been getting harder, but that corporate R&D has been sucking the best science out of the academic realm into a place where fewer papers are published overall, so there's less to cite. The fact that the most cited paper of all time is an obscure biology paper and that pharma spending is such a large component of "basic science", is suggestive of this.
The USSR never did fall behind the USA in terms of wealth; it started very far behind when it came into existence in the ashes of czarist Russia, and caught up quite a bit through one of the most rapid industrializations the world has seen. Even after the initial phase of industrialization, from the 1950s throughout early 1980s, the Soviet economy grew faster than the US economy.
At any rate, a full discussion of this might be off-topic, but let's say that it's a pretty uncontroversial idea in economics that a state run economy (like science) tends to be less efficient and produce lower quality outputs
For example, the current infant formula crisis was entirely the result of government management of production, distribution, and pricing of it. But the suggested fix is to have the government launch a criminal investigation of Abbot.
I did not experience this in my 8 years in academic math (PhD + postdoc). Maybe it differs a lot from field to field, but mine was positively teeming with unsolved problems. More importantly, it's teeming with paths that will clearly lead to more unsolved problems (of all difficulty and importance levels).
In fact, the thing I miss the most from academia is probably helping new students blaze some of those new paths. Both for their sake, and as a way for me to discover new problems for myself.
More scientist will mean that overall we have more time to spend in the dusty cobweb filled corners that have yet to be studied in fine detail. And while many of those corners contain nothing interesting. If even one of them contains something truly undiscovered it's worth it right?
I don’t think this system will last for too long. The environment is increasingly filled with status hungry people. Balaji has a good prediction on that.
>I don’t think this system will last for too long. Balaji has a good prediction on that.
link?
He was asked about his views on academia vs industry, having been in both himself. He said capable people in academia will gradually realize that it’s better to start companies or do the same work in industry, rather than writing these grants.
Unless the CEO actually was one of the inventors, that is grounds to invalidate the patents and there may be other consequences.
Really compared to the shenanigans with Newton, Leibnitz or Hooke and Newton, much of what we see today is quite harmless. Or take for a more recent example Teller and Oppenheimer. I think in science (but not only in science) we tend to idealise the old days and scientists, because we remember the discoveries not the politics and their personal flaws. I would even argue that this greatly demonstrates the success of the scientific process. In the end the best science won, despite all the political infighting.
Even looking at the peer review process: it was much more informal in the distant past, and maybe more akin to what happens today with invited papers in my experience, or with small open source software projects.
There were certaintly very petty but intense personal squabbles in the past but that's a bit different from the pervasive structural issues in science today.
At some level too, I don't care what it was like 100 years ago. I can still see the problems today.
Watson and Crick's famous 1953 paper where they presented the double helix of DNA is one such example.
I wish things like the research in this blog post got more attention, because it's important in understanding how things have and have not changed.
To some extent it depends on the weakness of humans for status and competition, because they are powerful motivators to drive advances.
I personally left science because I could see it had 98% overlap with what I'd do as an engineer (or entrepreneur, in the case of a PI), with a tiny fraction of the monetary benefit. Being a winner in that system is a great life, but it rests upon a huge pyramid of people who are working their butts off at low pay and lower prestige, for a brass ring that few will ever grasp.
I said exactly the same thing when I first left academia for tech, and people would look at me suspiciously and ask why I'd ever do such a thing. I looked at every PI I knew, and saw a task essentially identical to running a startup, but with no financial upside -- and the downside risk of being jobless and adrift if you don't get tenure.
I am a PI.
What I spent my day doing was dealing with managerial tasks, fiddly stuff for funding applications, etc.
I got to actually code for a whole day last week, and it was marvelous. It was also only possible because it was an all-hands-on-deck, everything else can wait situation.
Politics, of course, is part of the humanity, but it manifests differently in different environments (academia vs industry, or then vs now). There is undoubtedly politics in industry, however, there are also products and reality checks. The companies won’t survive if they operate entirely based on politics. In academia, on the other hand, the environmental feedback can be weaker depending on the subject; there, politics could come to the fore. Amusingly, Kissinger said, politics in academia is specially vicious precisely because the stakes are so low.
As for then versus now, there used to be a lot of low-hanging fruit that could be taken until 1980s, and there weren’t many researchers. But now there are fewer accessible opportunities, with far more researchers. If you consider that the size of the cake has not increased proportionally (due to factors mentioned in the article), while the number of players has increased substantially, you may conclude that the environment has become much more competitive. Increased competition indicates increased politics, emphasis on selling, networking, and other non-scientific factors relevant to industry, except there you actually get paid. The conditions are nowadays totally different. There all sorts of performance metrics (H-index, citations, number of papers etc), that have become target, in place of good science, which are gamed.
Injecting a ton of money creates perverse incentives and attracts people who are not interested pursuing something out of curiosity and interest, but for the money.
I think it probably creates a hostile environment for people who are there out of pure interest, because if there is a way to earn more money/status/power/whatever by doing something other than good work people will do it. So you effectively get punished for doing what you're supposed to instead of playing the game.
I think the "professional money extractor" explanation here is an apt description: https://news.ycombinator.com/item?id=18003253.
If you only give money and positions to people with "passion" when there is a clear demand for more, that's called exploitation. Maybe consider becoming an aerospace investor instead, that field is full of talent chasing after lost dreams for pennies. If you are a software engineer on this forum, you know damn well what your net worth is in the current North American market compared to the other legacy engineering fields. It is patronizing to expect the other scientific fields to be worth "less" than CS.
That's just an excuse to exploit people. The same nonsense as with teachers. "We want teachers that are passionate, not there just for the money". Bah.
In the end we're trying to get stuff done, and it doesn't really matter whether the pipes are kept clean by somebody really passionate for sanitation, or somebody who just thinks the pay is good enough to tolerate the smell.
The other irony is that Teller himself was part (came out) of one of the biggest counter examples. The Los Alamos project pushed unprecedented funding into science, and it let to many discoveries that would have taken decades longer otherwise.
Salaries for PhDs and PostDocs are a different story of course. They are alright in theory, but often you can only get a part time job, which in reality means a > full time job with a part time salary.
Around 80k €/year as W2 professor base salary
Post-tax 55k €
Guess that might be considered incredibly well paid in Germany
I can only speak from personal experience and observations in my environment, but to me this statement sound like a dark, cynical joke.
There is plenty of scientists who would love to work on interesting things, are hired as scientists, but once they are hired, no one actually lets them work on scientific problems.
Making matters worse, population count in highly educated countries is shrinking, inevitably leading up to fewer brains trying to solve hard problems than before. For now, I don’t see the remaining countries with a growing population making up for this loss in collective cognitive capacity devoted to science.
Perhaps we shouldn’t fear universal AI as much as we should cheer it on as the future saviour of science.
There is less chance of escaping past mistakes, or more chance that a mistake will ruin you. There are more things you have to stay on top of benefits and finance wise. There are more options for many things, which leads to more analysis of what's best. The systems we use every day, from cars to computers, are more complex.
Somewhat related, but I believe this is the reason why academia will continue to lag behind industry. Academia has a bias for fundamental knowledge, but as information grows, fundamental knowledge represents a smaller proportion of all knowledge[2]. Anecdotally I have observed a 2-5 year gap between innovation in industry vs academia (CS). Academia often far behind looking to monetize the success of industry via newly minted formulaic programs. This was not always the case, e.g. databases came mostly from academia into industry in the 1970s, but nowadays innovation in the field seems industry-driven. To be fair, perhaps academia has already explored much of the relevant concepts, and it's easier to innovate by way of implementing these same ideas in different programming languages and business models.
[1] What are we left with then? More efficient but also less robust systems? Being more efficient, i.e. generating more output, seems to be the chosen trade-off today, when you can afford extra human intervention, but will it ever be the case when we reach an inflexion point and economics favor robustness/stability over output?
[2] Not to mention the issues with defining something as fundamental. Should fundamental knowledge be frozen in time, or instead act a sliding window? I.e. when, if at all, does something go from fundamental to obsolete? and when does something go from innovation to fundamental? Is it merely popularity driven?
https://twitter.com/ThomasPHI2/status/1516079802184343560?s=...
I've used as a general theory for why change in the world seems to be slowing down, since although it's stayed constant, it has slowed down on a % basis.
The 20th century saw massive increases in knowledge with relativity and then quantum mechanics (ie QM/QFT/QCD). There have been accomplishments in the 21st century but a lot of then (eg the Higgs boson) were confirming things theorized 50+ years ago. Despites decades of trying we're still to find a crack in the Standard Model. Yes we've eliminated a lot of options and that's important in the process of elimination but we don't really seem to be any closer to resolving the fundamental contradictions of QM and GR.
There's a fun list of unsolved problems in physics [1]. My favourite is the so-called vacuum catastrophe [2] where the predicted (from QFT) and actual vacuum energy diverge by roughly 120 orders of magnitude. At the other end of the spectrum is the magnetic moment prediction vs actual [3] that differ by less than 10 parts per billion.
I'm actually losing faith we'll see meaningful progres sin a lot of these areas in my lifetime and I never used to think that.
[1]: https://en.wikipedia.org/wiki/List_of_unsolved_problems_in_p...
[2]: https://en.wikipedia.org/wiki/Cosmological_constant_problem
[3]: https://en.wikipedia.org/wiki/Precision_tests_of_QED
1) On number of papers published, citation value, etc. There may be an inflation in the numbers of PhD's granted for little more than serving as a PI's lab tech for a few years. That requires a few publications. Often the publication count is inflated by splitting a paper that would be more coherent as a single publication into several publications to fluff a CV. Such papers may not get many citations. This is about politics and careers, not about the quality of science being done.
2) Universities are in the market trying to patent discoveries and license them to big corporations. This is very bad for basic scientific discovery in many fields, as funding is directed away from those fields. Basic science underlies later applied science applications. Shutting off basic research is killing the goose that laid the golden eggs, but the administrators want those sweet patent license payments more than anything else.
3) People with the potential to do difficult science and math, but who happen to come from a background where they can't afford college without large loans, are much less likely to go into academic research than in prior eras in the USA when college was less expensive. Many would like to do research but the economics of current education systems are incompatible with that goal.
4) Some areas of research are simply not being funded anymore due to pressure from entrenched interests in areas like energy, pharmaceuticals, etc. Renewable energy research is a threat to fossil fuel interests. One-shot cures for diseases like HIV/AIDS are a threat to the profits of pharmaceutical companies selling lifetime maintenance programs. Research into environmental pollution is a threat to a wide variety of interests, as is research into declining biodiversity.
The corporatization and financialization of literally everything is really out of control, and these are the side effects.
There might be a lot of space for applied math/physics, but perhaps this is closer to engineering than research though the distinction is often blurry.
It makes sense that science picks “low hanging fruit” early on. Then, on average, later discoveries require more effort.
But the rate of progress depends on both how much effort a discovery takes AND how much effort is available. The progress of society has made it possible to aim exponentially more resources at solving problems. Computers let us automate things. Medicine & farming mean more humans can do higher value work. Better politics means fewer people dying in wars.
So I don’t care if each discovery takes more effort than the last. As long as we get exponentially more resources to go along with it, we can keep creating exponentially more knowledge for a looong time.
I haven't found a fair way to assess quality of papers - I think it depends highly on individual author motivations. If we are able to identify the right patterns among authors' motivations, it may be possible to come up with a better rating system for papers published. Current rating systems focus too much on individual papers/journals, and too less on authors.
To your other point: I get what you mean, and I don't even disagree, but I could also make the exact opposite point. Papers today often are these big, overly structured manuscripts that actually cointain a multitude of ideas so that it can become hard to see what exactly the point of the paper is. In the past, papers often were much more atomic. They presented one idea with some relevant context. There was no need to prove you had surveyed the entire field just to present an idea. No useless context, more relevance, more focus.
But most of the graphs can be explained by the sheer number of papers being published.
Academics have the perverse incentive to publish as many papers as possible, instead of discovering as many interesting things as possible, because items published is way easier to measure than real progress made.
Therefore there is a significant paper inflation problem. People might today write 10 papers with LPUs (least publishable unit) with stuff that would have been a single technical report 30 years back.
This is much worse than it sounds, because it affects productivity significantly in subtle ways: it takes effort to publish those 10 papers, it takes effort to read up and review the 10x papers of others, and it takes time to think about LPUs instead of concentrating on the bigger picture.
Unfortunately, due to the way the research system is set up, it is not likely things will improve.