This pattern can be seen in golden age vs current CS papers: the former[0] contain multiple ideas per paper, derived from a short[1] list of references; the latter often spend pages upon a single idea, derived from a huge list of references.
[0] at least: the classics which still have attention drawn to them today
[1] to be fair, they didn't have much literature from which to draw — maybe a fairer comparison would be "references as percentage of literature extant"?
[0] is extremely important here, maybe too important to be left in a footnote. When you look back at history it's all too easy to only recall a select few gems and discard the rest. It's possible that the vast majority of research published today will go by the wayside as dead ends or insignificant, while the stuff that ends up having world changing impact will also contain "multiple ideas" and a "short list of references".
I don't have any proof to disagree with you, not even an anecdote, but be cautious of making claims such as these without data. Now will almost always seem uninteresting and pedestrian while the past will seem mysterious and interesting.
I tend to agree with this - one time I tried reading through all the Internet RFCs starting at the first one and I would say that probably 80% of the ideas in first 1000 were abandoned quickly, and after that that the % of abandoned went up to 90-95 percent, with a high quantity of vendors trying to push their own solution a standard.
This is a weak claim unless you provide outside evidence of survivorship bias. Just because a bias could exist isn’t evidence that it does exist in and of itself.
This feels like an example of our trigger happy use of biases to dismiss opinions we disagree with. What perspective, experience, or evidence makes you think survivorship bias is at play here?
In the comment you’re replying to, for example, the rate of un-abandoned RFC started ~20%, and decreased to ~5-10% over the period of GP’s review. That seems lije a very significant change to me, which is hidden by the surprise that the original number wasn’t higher.
Fair argument. I didn't put too much rigorous thought into my post, and really didn't mean offence, but to roughly explain my thinking:
- First, I wasn't trying to dismiss any opinions by bringing up survivorship bias, I brought it up because I think it's valuable to have a term to describe the general and probably bi-directional relationship between how notable something is and how well it's known many years after creation
- I'd argue that the original comment's footnote [0] talking about "classic" papers is explicitly bringing survivorship, if not the survivorship bias, into the conversation.
Honestly, at this point I've kinda lost track of what I wanted to clarify here, so - I apologize for the confusion and, if interpreted as such, the offensive posting.
Isn’t what you describe simply the consequence of picking low hanging fruits?
it’s only seminal papers that involve multiple ideas together, and even those have a lot of mathematics involved to make them stand up to scrutiny tests.
I’d even go as far as to argue that if you took any of the highly cited papers of the last 5 years in ML, 20 years or so back, they’d be even more groundbreaking than that time’s seminal work.
I think you see the same in physics. The extreme example is probably Einstein’s “On the electrodynamics of moving bodies” [1], where he laid out the special theory of relativity. It has some incredibly original ideas, and not a single reference. :)
I noticed the same thing when doing my Master's (graduated 2012, work in embedded debuggers), for which I read a heck of a lot of early papers, both the better and the worse (Sturgeon's Law is very real). My SWAG is that a lot of early CS papers were being written by people doing very serious programming; they often did a few years in industry (or they were doing very focused Physics coding) before coming back for the PhD. Too, they were not worried about working within the dominant operating system framework - such things didn't quite exist. So they could give things a fresh go by the simple virtue of starting out on a new computer.
I think the straight BS->PhD pipeline has introduced an "academic" bias, in the bad sense of the word. I also think that the dominance of Linux, Windows, etc have put blinkers on our research.
That is extremely interesting. I always thought that the debugging tools we have are cool... but we could do _so much more_ to observe the systems we develop. And the scripting languages in most of the embedded debuggers I had worked with sucked.
In general, our debugging tools today are just terrifyingly crude. Some improvements have been made since my thesis in the industry state but overall... yuck.
To get us to the state of the art debugging research in the 90s:
- FOL scripting trap on every variable, function, etc. essentially you need this "monitor" or supervisor behind the scenes observing the total state of the system _as it exists in source code, not in hex_.
- reversible programs that can go forward and backward in program state & history. Not sure if the 90s research had a full "tree" of possible executions you could walk or if it was linear.
- if you attach something like a Lamport state matrix with the a transport layer you can move to a parallel/concurrent debugger
OTel and friends are essentially trying to build points 2 and 3 there, but its very mushy, since you're not actually capturing a full system state as part of the system, but whatever the coders chose to put in the system capture.
I remember in 2012, 2013 or so, Green River came out with a single core reversible debugger. gdb had one too in '09, or so.
Isn’t this at odds with highly optimized AOT code? I suspect that’s the ultimate blocker but certainly you have this today for any kind of JIT/Interpreted language.
EBF does kind of let you get some of it back, but ultimately it turns out that writing code to debug other code is really complex and you experience rapidly diminishing returns for most bugs (except the super gnarly ones) because a little bit of state and system’s understanding proves sufficient most of the time.
I’m sure you’re aware but GDB has reversible debugging via rr. It’s still hard to get all the way back to source if you’re debugging an optimized build of your code. And polyglot environments can be extra challenging unless special care is taken. I agree it’s a mess, but I think that’s because it’s actually extremely hard to meet competing goals. Do you see the reason why it’s so bad differently with your greater experience in the space?
Yes, I mentioned gdb's reversible debugging capability late in my comment. :)
My experience was that C++/Java debuggers beat interpreted language debuggers shockingly well. So much so I still don't touch interpreted language debuggers today. I should check in sometime on Python soon.
Debug code has to, essentially, include debug symbols and tracking, or, alternatively, there's that supervisor monitoring all source-code relevant locations. Things will be slower ... on a conventional x86 chip.
I would suggest that its plausible today that a separate _core_ could be developed for internal system monitoring (think on-SoC JTAG and you get the idea) and that core could be dedicated to more OS functionality otherwise. Notice that this involves a different ISA.
The core tooling advancement problem I believe reduces to this:
> because a little bit of state and system’s understanding proves sufficient most of the time
After a decade being an industry professional, I would say most is 99.5% or greater. 5 out of 1000 bugs seems about right.
So the genuinely advanced tools only start to come into play in very difficult situations - with the implication that only groups with difficult situations will _pay_ for the advanced tooling - with the implication that there's virtually no commercial market, and open source developers have to do this on their own time. Which in turn seems to be generally decreasing IMO - I believe the problem is approximately (say, 90% of the way) isomorphic to "cost of housing market" discussions essentially (how do I spend 1-2 years being someone who can understand graduate-level CS/SWE papers and write high quality code, and then write this state-of-the-art tool that no one will pay for, and still have housing, food, make debt payments etc).
Polyglot environments have to have an element of a debuggable protocol baked in at a pretty deep level to really make it work at, say, the level of Visual Studio in 1999 or something. There's all sorts of issues there around non-aligned interests that shouldn't need to exist. For example, a simple polyglot problem is "how do I send structured data from one process to another in a standardized way". The standard Unix way is "line structured", which works until it doesn't, and then you're staring at writing your own marshal/unmarshal system, which is, definitionally, only your standard. If we don't have _that_ solved, then the social pressure to have a interop debugger between, e.g., Python and Ruby is ... well, take your own guess there.
I think there's absolutely space for a hardware-software codesign to occur which builds out a hardware system, a compiler qua LLVM, a standardized interpreter layer, interop protocols, etc, and to produce a developer system that is a quantum leap beyond what we have today. But that would be a decade long project and have a very long term payoff.
the tl;dr then, imo, is that profound advancement in the field only directly benefits a few, and then only sometimes; so the incentive problem is the basic issue.
This is a very cool conversation, for some reason, I always found debugging tools to be fascinating bits of tech.
I do have to admit though, that my first read of your post was wrong tough, I read it as "debuggers used in embedded" like JTAG stuff, OpenOCD, etc. but you mean it as, if I read this right "debuggers embedded in the runtime/program" stuff like Python, or GDB server running on system right?
an easy way to read golden-age papers (say, 01952–01972) that don't have attention drawn to them today is to look at the other papers in the same issue of the journal that had some well-known paper in it
when i've done this i've found that most of the papers present trivial and wrongheaded approaches, just like most papers today, but they're still a lot more pleasant to read because they're much better written; however, it's much slower going because the terminology is pretty alien
they also have much shorter lists of references but of course they didn't have bibtex
This pattern can be seen in golden age vs current kitchen utensils: the former[0] perform admirably today; the latter often are so poorly constructed that they break or have to be thrown away within a couple of years.[1]
[0] at least: the utensils which still have attention drawn to them today
This may be a confounding effect where there's pressure to publish faster. But I'd also say that this same pressure can push towards consolidation of ideas as ideas on the boundaries are often more difficult and may take longer to compete.
The reference lists were short because you were limited to journal issues held in your university library that you happened to pull off the shelf. There was no google scholar to look for each and every paper ever published containing your keyword.
It reminds me of a chapter in Alex Pentland's book Social Physics where he looked at the performance of IIRC eToro traders and the degree to which ideas spread among traders and the collective returns on investment.
When there was extreme levels of connectivity and copying between traders collective returns went down because diversity of trading strategies went down, when there was too little connectivity the same thing happened as optimal strategies could not spread effectively.
The maximum returns were achieved somewhere in the middle when there was both room for individual new strategies to emerge but enough connectivity for good strategies to spread avoiding both a sort of herd dynamic and isolation.
mmmh.. Paving the way for ML-generated/immitated stuff..
IMO same thing in music, recent years. It's easier and easier to produce yet-another-immmitation.
Honestly, I think this is just an academic phrasing of the Eternal September concept. Which... is observed all over. The paper is a bit more general than that though and applies the concept to other social domains.
When something is explained in the most convoluted way available, sometimes is hiding the fact that when you grasp the bark is not really alive. I can't find a lot of real substance in the article.
Interesting paper. Nice to see things laid out ba, and examined in data. Seems like this would have a lot of important implications.
It seems like their basic framework (ironically) could be extended to include other things, like incentives (eg for apparent production of "ideas").
One thing about these models is that they don't necessarily get into why things start to be imitated. It seems like the ability to imitate would be related to the spread of the ideas? That is, even if there's a valid idea if the imitators can't or won't imitate it, it won't be imitated. I guess it speaks to whether ideas have value outside of the social milleu they are presented in at a particular point in time. This might seem self evident, but then you have to think about the diffusion dynamics being a property of ideas vs imitators. The authors discuss this in terms of experts not being able to produce novel ideas fast enough, but how would your distinguish that from imitators not understanding how to imitate novel ideas?
humans are natural "copy-cats". this is how kids learn.
so why are ideas imitated? it's human nature.
and not only humans, other apes are known to do this "monkey see, monkey do"
there was an interesting experiment in that human children, as opposed to chimpanzees, would imitate the nonsensical aspects of behavior, making it as though chimpanzees are more 'rational' or more 'logical' than human kids.
it's a disappointing choice of words to say the expertise is getting diluted. it's not diluting, it's spreading, getting disseminated.
to say that it's diluting softly implies that it's getting lost, that we're running out of a specific 'innovation' when in fact, it's getting adopted, integrated into our regular lives. Of course it's initially imitated, this is part of how it's getting spread (copied). People will imitate it so to learn what the 'innovation' (as a noun) actually is.
But each person in this expertise is less expert. Today you can totally “cargo cult” being a software architect, which means recommending the solutions that everyone recommends, whereas earlier in computing you had to understand the entire lower layers and be an ace at everything before you’d recommend the same solutions.
The difference being that the architect today has problems reasoning about problems, and can’t advise much. They’d tell you what the cargo cult says.
This is a problem of mimicking/learning the cargo cult subject and missing the prerequisite lessons leading up to it. If expertise is defined as knowing the whole stack leading to the top-end subject and it's relationships, then yes it's a problem of incorrect growth and therefore the pool is widened and yet has less actual experts in it.
Jonathan Blow's 'collapse of civilization' talk goes into this quite a bit.
Okay, I'm not sure I grasp everything from this paper, but - maybe it's wishful thinking - it seems this is an argument against the idea that strong IP protection is NOT needed to foment innovation. That remix culture can give rise to innovation despite seemingly not incentivising it (through legal protection).
As the sample size approaches the population size, the average value of any trait in the sample approaches the average value of that trait in the population.
Much like Little's Law, this simple and retrospectively obvious observation has wide-ranging consequences. Any organization that aspires to eliteness is thus necessarily selective.
32 comments
[ 5.4 ms ] story [ 81.6 ms ] thread[0] at least: the classics which still have attention drawn to them today
[1] to be fair, they didn't have much literature from which to draw — maybe a fairer comparison would be "references as percentage of literature extant"?
I don't have any proof to disagree with you, not even an anecdote, but be cautious of making claims such as these without data. Now will almost always seem uninteresting and pedestrian while the past will seem mysterious and interesting.
Very pareto principle/sturgeons law there.
This feels like an example of our trigger happy use of biases to dismiss opinions we disagree with. What perspective, experience, or evidence makes you think survivorship bias is at play here?
In the comment you’re replying to, for example, the rate of un-abandoned RFC started ~20%, and decreased to ~5-10% over the period of GP’s review. That seems lije a very significant change to me, which is hidden by the surprise that the original number wasn’t higher.
- First, I wasn't trying to dismiss any opinions by bringing up survivorship bias, I brought it up because I think it's valuable to have a term to describe the general and probably bi-directional relationship between how notable something is and how well it's known many years after creation - I'd argue that the original comment's footnote [0] talking about "classic" papers is explicitly bringing survivorship, if not the survivorship bias, into the conversation.
Honestly, at this point I've kinda lost track of what I wanted to clarify here, so - I apologize for the confusion and, if interpreted as such, the offensive posting.
Have a good day.
it’s only seminal papers that involve multiple ideas together, and even those have a lot of mathematics involved to make them stand up to scrutiny tests.
I’d even go as far as to argue that if you took any of the highly cited papers of the last 5 years in ML, 20 years or so back, they’d be even more groundbreaking than that time’s seminal work.
1. https://www.fourmilab.ch/etexts/einstein/specrel/specrel.pdf
I think the straight BS->PhD pipeline has introduced an "academic" bias, in the bad sense of the word. I also think that the dominance of Linux, Windows, etc have put blinkers on our research.
In any case, you can trace the evolution I describe in the journal https://onlinelibrary.wiley.com/journal/1097024x which has been around for decades.
That is extremely interesting. I always thought that the debugging tools we have are cool... but we could do _so much more_ to observe the systems we develop. And the scripting languages in most of the embedded debuggers I had worked with sucked.
To get us to the state of the art debugging research in the 90s:
- FOL scripting trap on every variable, function, etc. essentially you need this "monitor" or supervisor behind the scenes observing the total state of the system _as it exists in source code, not in hex_. - reversible programs that can go forward and backward in program state & history. Not sure if the 90s research had a full "tree" of possible executions you could walk or if it was linear. - if you attach something like a Lamport state matrix with the a transport layer you can move to a parallel/concurrent debugger
OTel and friends are essentially trying to build points 2 and 3 there, but its very mushy, since you're not actually capturing a full system state as part of the system, but whatever the coders chose to put in the system capture.
I remember in 2012, 2013 or so, Green River came out with a single core reversible debugger. gdb had one too in '09, or so.
Some cool information in an old SO thread: https://stackoverflow.com/q/1470434/26227
Isn’t this at odds with highly optimized AOT code? I suspect that’s the ultimate blocker but certainly you have this today for any kind of JIT/Interpreted language.
EBF does kind of let you get some of it back, but ultimately it turns out that writing code to debug other code is really complex and you experience rapidly diminishing returns for most bugs (except the super gnarly ones) because a little bit of state and system’s understanding proves sufficient most of the time.
I’m sure you’re aware but GDB has reversible debugging via rr. It’s still hard to get all the way back to source if you’re debugging an optimized build of your code. And polyglot environments can be extra challenging unless special care is taken. I agree it’s a mess, but I think that’s because it’s actually extremely hard to meet competing goals. Do you see the reason why it’s so bad differently with your greater experience in the space?
It's locked behind the really expensive (like 10k or so? For I guess a seat a year?) Visual studio license and IIUC windows-only.
My experience was that C++/Java debuggers beat interpreted language debuggers shockingly well. So much so I still don't touch interpreted language debuggers today. I should check in sometime on Python soon.
Debug code has to, essentially, include debug symbols and tracking, or, alternatively, there's that supervisor monitoring all source-code relevant locations. Things will be slower ... on a conventional x86 chip.
I would suggest that its plausible today that a separate _core_ could be developed for internal system monitoring (think on-SoC JTAG and you get the idea) and that core could be dedicated to more OS functionality otherwise. Notice that this involves a different ISA.
The core tooling advancement problem I believe reduces to this:
> because a little bit of state and system’s understanding proves sufficient most of the time
After a decade being an industry professional, I would say most is 99.5% or greater. 5 out of 1000 bugs seems about right.
So the genuinely advanced tools only start to come into play in very difficult situations - with the implication that only groups with difficult situations will _pay_ for the advanced tooling - with the implication that there's virtually no commercial market, and open source developers have to do this on their own time. Which in turn seems to be generally decreasing IMO - I believe the problem is approximately (say, 90% of the way) isomorphic to "cost of housing market" discussions essentially (how do I spend 1-2 years being someone who can understand graduate-level CS/SWE papers and write high quality code, and then write this state-of-the-art tool that no one will pay for, and still have housing, food, make debt payments etc).
Polyglot environments have to have an element of a debuggable protocol baked in at a pretty deep level to really make it work at, say, the level of Visual Studio in 1999 or something. There's all sorts of issues there around non-aligned interests that shouldn't need to exist. For example, a simple polyglot problem is "how do I send structured data from one process to another in a standardized way". The standard Unix way is "line structured", which works until it doesn't, and then you're staring at writing your own marshal/unmarshal system, which is, definitionally, only your standard. If we don't have _that_ solved, then the social pressure to have a interop debugger between, e.g., Python and Ruby is ... well, take your own guess there.
I think there's absolutely space for a hardware-software codesign to occur which builds out a hardware system, a compiler qua LLVM, a standardized interpreter layer, interop protocols, etc, and to produce a developer system that is a quantum leap beyond what we have today. But that would be a decade long project and have a very long term payoff.
the tl;dr then, imo, is that profound advancement in the field only directly benefits a few, and then only sometimes; so the incentive problem is the basic issue.
I do have to admit though, that my first read of your post was wrong tough, I read it as "debuggers used in embedded" like JTAG stuff, OpenOCD, etc. but you mean it as, if I read this right "debuggers embedded in the runtime/program" stuff like Python, or GDB server running on system right?
Think something like an Arduino.
when i've done this i've found that most of the papers present trivial and wrongheaded approaches, just like most papers today, but they're still a lot more pleasant to read because they're much better written; however, it's much slower going because the terminology is pretty alien
they also have much shorter lists of references but of course they didn't have bibtex
[0] at least: the utensils which still have attention drawn to them today
[1] to be fair: it’s probably survivorship bias.
When there was extreme levels of connectivity and copying between traders collective returns went down because diversity of trading strategies went down, when there was too little connectivity the same thing happened as optimal strategies could not spread effectively.
The maximum returns were achieved somewhere in the middle when there was both room for individual new strategies to emerge but enough connectivity for good strategies to spread avoiding both a sort of herd dynamic and isolation.
It seems like their basic framework (ironically) could be extended to include other things, like incentives (eg for apparent production of "ideas").
One thing about these models is that they don't necessarily get into why things start to be imitated. It seems like the ability to imitate would be related to the spread of the ideas? That is, even if there's a valid idea if the imitators can't or won't imitate it, it won't be imitated. I guess it speaks to whether ideas have value outside of the social milleu they are presented in at a particular point in time. This might seem self evident, but then you have to think about the diffusion dynamics being a property of ideas vs imitators. The authors discuss this in terms of experts not being able to produce novel ideas fast enough, but how would your distinguish that from imitators not understanding how to imitate novel ideas?
so why are ideas imitated? it's human nature.
and not only humans, other apes are known to do this "monkey see, monkey do"
there was an interesting experiment in that human children, as opposed to chimpanzees, would imitate the nonsensical aspects of behavior, making it as though chimpanzees are more 'rational' or more 'logical' than human kids.
to say that it's diluting softly implies that it's getting lost, that we're running out of a specific 'innovation' when in fact, it's getting adopted, integrated into our regular lives. Of course it's initially imitated, this is part of how it's getting spread (copied). People will imitate it so to learn what the 'innovation' (as a noun) actually is.
The difference being that the architect today has problems reasoning about problems, and can’t advise much. They’d tell you what the cargo cult says.
Jonathan Blow's 'collapse of civilization' talk goes into this quite a bit.
Much like Little's Law, this simple and retrospectively obvious observation has wide-ranging consequences. Any organization that aspires to eliteness is thus necessarily selective.