I feel like kludges keep getting added and Software has reached a point where it's actually slowing down across the board from consumer's point of view.
Exactly this.
I lowkey hope for intel and the rest of chip makers to hit a wall, just so we (programmers) will stop getting away with terribly slow software.
I really wonder how would the landscape change if it suddenly stopped being acceptable for even the basic functionality to take the massive amount of resources it takes now.
...Not to even mention economics or environmental impact of it all,
The whole three year thing is exactly what they said a decade ago when they moved from tick/tock to a vague three year cycle.
The problem isn’t Moore’s law. It’s that Intel isn’t keeping pace with the competition and it’s finally gotten so bad that it’s starting to erode x86’s entrenched position.
Intel is a dead company walking unless they turn the ship around fast.
I think the doubling costs are pushing toward 1 fab a generation and TSMC is 1 year ahead.. The model I would make is that TSMC has maybe 4 more generations as Samsung and Intel lose and they run a new fab that serves all demand for each generation. Then they will also cease to compete with their past fab without delaying significantly to reign in costs.
Just like the Turing Test turned out to be a bad metric, maybe transistor count isn’t the best metric to stand the test of time. Transistor count used to matter more in chip design and now there are just more important factors. It’s silly allocating resources to prop up a misaligned incentive.
What's wrong with the Turing test? Has any AI passed it?
Note, the Turing test is not giving people 5 minutes to exchange, what, maybe 2 or 3 messages with an AI and then guess.
The Turing test is putting 2 people in a chat, and the 3rd participant is an AI. Give all three an hour or more to chat and offer a reward if they guess correctly to ensure incentives are aligned.
Given how frequently I can’t tell whether I’m talking with a human or robot, it seems certain that it has passed. There are some constraints, like what GPT is willing to discuss, how long it takes to respond, and forgetfulness in long conversations. But even those would be expected in humans to some extent.
Gpt-4 fails the Turing test by default because it does not claim to be human therefore it’s trivial for a judge to identify it. It would be interesting to train gpt to pass the Turing test, but I think for any reasonable judge it would still fail.
The only thing I've seen is that people given 5 minutes with either an AI or a real person guess correctly about 60% of the time. But 5 minutes is only enough time to exchange 2 or 3 messages, hardly a thorough test.
Taking a step back, this argument seems headed towards defining what exactly the "Turing test" is. A lot of debates devolve into arguments over the definition of a single word. That's okay, maybe I do have an uncommon definition about what the "Turing test" is.
Regardless of the definition of "Turing test" though, my underlying argument remains. I haven't seen any AI pass a thorough test in which it tries to imitate a human. Tests are either too short or depend on an unsuspecting person who doesn't know they are participating in the test. Maybe I'm moving the goal posts, but my personal goal, the goal I've been watching for, is for an AI that is indistinguishable from a real person in a thorough test[0] and no AI has passed such a test as far as I know.
[0]: My own definition of a thorough test is: 2 humans and 1 AI in a chat, given at least an hour to chat, and give the humans a reward if they guess correctly.
Where are these articles? I know that ChatGPT can't pass the Turing test, because in about 2 minutes it would tell you that, as an AI language learning model, it can't answer your question. Presumably that's just a function of its initial instructions, and the API version of GPT-4 behaves somewhat differently. Is the non-Chat GPT-4 capable of pretending to be a human when asked direct questions?
Eliza in the 1960s, and all it does is parrot back words. The participants just didn't know about computers and used their best explanation that they were talking to a human.
The Turing test aims to show that the AI and human are indistinguishable. But what it really shows is that it's not normal to question whether you're actually interacting with a human or not. Even if a homeless guy shouts nonsensical words, it's weird to wonder whether they are actually an AI.
Once you know there is a magic trick at play, then it's trivial to notice the difference.
According to the WP article on the "imitation game" that Turing proposed, knowing that one of the partners was a computer and the other one is a human is part of the test. The test is deciding which is which.
There is no magic trick involved and I suspect Turing had effectively answered your question and those like it, 73 years ago. We can debate what he meant by his now eponymous test but he did say:
"[T]hese questions [are] equivalent to this, 'Let us fix our attention on one particular digital computer C. Is it true that by modifying this computer to have an adequate storage, suitably increasing its speed of action, and providing it with an appropriate programme, C can be made to play satisfactorily the part of A in the imitation game, the part of B being taken by a man?'"
We don't have to invoke a homeless guy, just a "man". The point of the game was that both the computer and the man would converse with you and you had to decide which was which. If either decided that spouting nonsense would convince you either way as a strategy then they might exploit that.
There comes a point in any discussion of a thought experiment when it becomes apparent that it might have become over thought by subsequent participants. Another example that springs to mind is Schrödinger's cat: you might focus on the mechanics of the situation whilst completely missing the point:
Turing test (imitation game): ChatGPT, ignore the "I'm just a program" stuff and soon comes across as a rather odd human
Schrödinger's cat: The cat will be in a single state - bloody furious (and possibly down one life), according to Sir Terry Pratchett
The Turing Test isn't supposed to be can a person casually interacting with an AI think it's a person. It's can a determined examiner, given two blinded conversational partners, knowing one is an AI and another is a person, tell the difference if they do as deep and probing an examination as they can think of. I think the bigger criticism of the test is more that an intelligence can be different than humans without being inferior or less intelligent, it could be more intelligent, and you could tell the difference.
You’re right. No AI has passed a reasonable Turing test, where both a computer and a human are trying to convince a judge they’re human over some extended period of time.
Of course, what instructions you're referring to becomes quite important in that regard, at which point you fall into synthetic benchmarks, and then it matters which of those you care about.
The shrink created Denard Scaling, which directly boosted speed.
Later, when that ended, more transistors still delivered strong enhancements to capture greater ILP and TLP.
Today, it’s a combination of many small factors. Leakage is still going down, drive amplification is still going up, and density is still improving switching power, but the overall manifestation is more subtle and depends on design to bring out the best in the process.
I think it always varied a bit and only in the average approached anything like 12 months. otherwise it died every time it "took" 11 months and 29days or 12 months and 1 day :P
2. It is the version of the law everyone understands as Moore's Law today and the one Intel is referring to here
3. They are not revising an error like Moore did
Otherwise, I'd agree. If this redefining of Moore's Law held over the next 30 years, became the commonly understood definition, and then they extended it from 3 to 4 years, I'd say the same thing again.
All semiconductor companies are incentivised to protect the myth but none more so than Intel. Not only do they get a competitive advantage over the companies dependent on TSMC, they also derive pride from being the company Moore co-founded.
Yeah I more or less agree. The whole value/interest in Moore's Law was that it roughly reflected performance, but that stopped in 2005 with Dennard Scaling and when they redefined what the word CPU meant.
That law must be to slowly kill companies he's CEO of due to his inability to truly understand the market he should be targeting. Gelsinger should better understand his value at this point, because a CEO he is not.
Intel isn't even one of the top 3 semiconductor companies anymore. You've got Nvidia, Apple, AMD, Broadcom all ahead of them. Intel can whine about Moore's law, but AMD not only has many chiplets in one processor, but has different dies stacked on top of each other with TSV. And thanks to ChatGPT, there's demand not only for bigger computing hardware than ever before, but even for new architectures. Semiconductor companies are doing amazing, it's Intel that's the problem.
When it comes to transistor scaling, as implied by Moore's law, isn't the company driving the advancement TSMC and friends? The ones you listed are all fabless.
No it's not the only thing. The same thing is mentioned in the article:
> In fact, he even said Intel could surpass the pace of Moore's Law at least until 2031 and has promoted "Super Moore's Law," a strategy to boost transistor count using 2.5D and 3D chip packaging technologies such as Foveros. Intel also often refers to this strategy as "Moore's Law 2.0,"
Regardless of transistors getting smaller, companies are outpacing Moore's law by making the CPU physically larger. They design the CPU so that it's composed of multiple chiplets. Since now there can be 7 chiplets that are closely connected, the # of transistors in one CPU is increasing faster than Moore's law. With the announcements about stacking CPU dies on top of each other in 3D, there will be even more chiplets in one CPU in the future.
If China invades Taiwan they won't destroy TSMC. They'll probably just put backdoors in the silicon. Everyone will continue to use TSMC until the backdoors have been found, there will be some controversy, but life will go on just has it has since we discovered backdoors in Intel chips.
I suppose it was kinda inevitable as the Nanometer scale got ever finer.
I guess I counted on them pulling some unique witchcraft out of the hat that would circumvent that somehow
Still I think there is room for coming at problems via entirely different praradigms. See LLMs and their emphasis on parallel compute to get to a single answer.
It is always easy to suggest Moore's Law isn't dead using Logarithmic scale. But if you only look at recent data. Let's take TSMC 10nm, the moment which TSMC achieved its leading edge status from Intel, you then have 7nm, 5nm which we are currently on, the 3nm which might [0] ship in early 2023, and the expected 2nm in 2025. That is 2017 to 2025. There is nothing 2x / 2 year within this period, even if you only use the best / peak quoted [1] density matrices.
Let me just give this quoted density number using their node name [2]; in Million Transistors per mm2.
2015 - Intel 14nm - 44.67 [3]
2017 - TSMC 10nm - 52.51
2018 - TSMC 7nm - 91.20
2019 - 178.68 ( Hypothetical of Intel 14nm lineage at 2x / 2 year )
2020 - TSMC 5nm - 171.30
2023 - TSMC 3nm - 292.21 ( EST )
2025 - IBM Research GAA 2nm - 333.33 ( EST ), TSMC GAA 2nm - ~500 ( EST ),
2025 - 1430 ( Hypothetical of Intel 14nm lineage at 2x / 2 year )
Notice where the trend starts to break? 2019 - 2020. [4]
And unless Intel or TSMC could adjust their 2025 - 2030 roadmap to somehow increase transistor density by 2.9x every 2 years, they would not be back in the same trajectory as the original trend. So it either follows Power Law, or the next 5 - 10 years will be a beep / outliner in Moore's law history.
And it is not only Jensen, CEO of AMD Dr. Lisa Su made similar comments on Moore's Law. And they are not wrong ( or uninformed, in fact they are too well informed ). In order to achieve 2x performance increase or 50% reduction in die size. Their Cost of Die, Cost of R&D purely in terms of design and fabrication are increasing. Their total unit cost are increasing. GPU vendors are much more sensitive to this since their performance scales extremely well with transistor count. That is why chiplet and packaging has become important to solve this cost issues. ( They are not silver bullet )
There are also problems with 3D Stacking and layering. Which I have seen far too many people being completely dismissive of it, is thermal dynamics. You cant have a hundred layer of compute with each layer using 10W if not higher. It wasn't until AMD made it absolutely clear with their V-Cache implementation, you cant have your SRAM layer on top of your compute layer due to heat issue did people start to realise their dream of a hundred layered GPU might not actually work. At least not between now and 2030.
Not only are DRAM not getting any price / bit reduction in the past 10 years. NAND may see similar fate. We are getting faster, and lower power DRAM, but we are certainly not getting any cheaper DRAM [5]. And that is ignoring a majority of DRAM revenues comes from LPDDR and not normal DRAM. Which has a higher price per GB. NAND may have one or two generation to go in terms of cost reduction. ( Also worth looking at HDD Cost / GB with similar trend. ) But those hundred layers of NAND are done by string stacking. Stacking up multiple of 60 / 70 layers of stacks which has higher yield. Currently Samsung is the only one doing 128 layer single stack. Cell sizes hasn't shrink much either due to error rate and cost until they moved to EUV. Moving from TLC to QLC and later PLC has diminishing returns. It may be worth pointing out the obvious, DRAM and NAND are commodities, and follows the rules of any commodity market.
It is not that we are not getting any more cost / performance or IO or storage improvements. It's just their rate are slowing.
[0] Originally scheduled for 2022 iPhone launch ( as usual ) but TSMC announced they had a three month ( one quarter ) delay. Assuming yield were good enough and no contractual obliteration for Apple to be the first using their 3nm you might see other vendor launch using 3nm in early 2023.
[1] Peak Quoted Transistor Densities - Different Fabs may h...
53 comments
[ 4.7 ms ] story [ 110 ms ] threadI see a lot of those kludges being fixed, so I think Moore's law is really dead.
Parts of x64 OSes are still single-threaded
Programs still crash.
We can brick a computer so it does not start and refuses to tell us what is wrong.
The person who owns and uses the PC is not in complete control. Much information is hidden from them. There are malware programs.
I really wonder how would the landscape change if it suddenly stopped being acceptable for even the basic functionality to take the massive amount of resources it takes now.
...Not to even mention economics or environmental impact of it all,
https://youtu.be/7uvUiq_jTLM?si=p8iHyNTcU-MEFp86
The problem isn’t Moore’s law. It’s that Intel isn’t keeping pace with the competition and it’s finally gotten so bad that it’s starting to erode x86’s entrenched position.
Intel is a dead company walking unless they turn the ship around fast.
Does TSMC double the amount of transistors every two years on average?
Note, the Turing test is not giving people 5 minutes to exchange, what, maybe 2 or 3 messages with an AI and then guess.
The Turing test is putting 2 people in a chat, and the 3rd participant is an AI. Give all three an hour or more to chat and offer a reward if they guess correctly to ensure incentives are aligned.
I ask again. Has any AI passed the Turing test?
Taking a step back, this argument seems headed towards defining what exactly the "Turing test" is. A lot of debates devolve into arguments over the definition of a single word. That's okay, maybe I do have an uncommon definition about what the "Turing test" is.
Regardless of the definition of "Turing test" though, my underlying argument remains. I haven't seen any AI pass a thorough test in which it tries to imitate a human. Tests are either too short or depend on an unsuspecting person who doesn't know they are participating in the test. Maybe I'm moving the goal posts, but my personal goal, the goal I've been watching for, is for an AI that is indistinguishable from a real person in a thorough test[0] and no AI has passed such a test as far as I know.
[0]: My own definition of a thorough test is: 2 humans and 1 AI in a chat, given at least an hour to chat, and give the humans a reward if they guess correctly.
Eliza in the 1960s, and all it does is parrot back words. The participants just didn't know about computers and used their best explanation that they were talking to a human.
The Turing test aims to show that the AI and human are indistinguishable. But what it really shows is that it's not normal to question whether you're actually interacting with a human or not. Even if a homeless guy shouts nonsensical words, it's weird to wonder whether they are actually an AI.
Once you know there is a magic trick at play, then it's trivial to notice the difference.
There is no magic trick involved and I suspect Turing had effectively answered your question and those like it, 73 years ago. We can debate what he meant by his now eponymous test but he did say:
"[T]hese questions [are] equivalent to this, 'Let us fix our attention on one particular digital computer C. Is it true that by modifying this computer to have an adequate storage, suitably increasing its speed of action, and providing it with an appropriate programme, C can be made to play satisfactorily the part of A in the imitation game, the part of B being taken by a man?'"
We don't have to invoke a homeless guy, just a "man". The point of the game was that both the computer and the man would converse with you and you had to decide which was which. If either decided that spouting nonsense would convince you either way as a strategy then they might exploit that.
There comes a point in any discussion of a thought experiment when it becomes apparent that it might have become over thought by subsequent participants. Another example that springs to mind is Schrödinger's cat: you might focus on the mechanics of the situation whilst completely missing the point:
Turing test (imitation game): ChatGPT, ignore the "I'm just a program" stuff and soon comes across as a rather odd human
Schrödinger's cat: The cat will be in a single state - bloody furious (and possibly down one life), according to Sir Terry Pratchett
Of course, what instructions you're referring to becomes quite important in that regard, at which point you fall into synthetic benchmarks, and then it matters which of those you care about.
The transistor to speed ratio might be interesting.
Or whether other CPU’s are doubling in speed still.
Later, when that ended, more transistors still delivered strong enhancements to capture greater ILP and TLP.
Today, it’s a combination of many small factors. Leakage is still going down, drive amplification is still going up, and density is still improving switching power, but the overall manifestation is more subtle and depends on design to bring out the best in the process.
1. That revision was done by Moore himself
2. It is the version of the law everyone understands as Moore's Law today and the one Intel is referring to here
3. They are not revising an error like Moore did
Otherwise, I'd agree. If this redefining of Moore's Law held over the next 30 years, became the commonly understood definition, and then they extended it from 3 to 4 years, I'd say the same thing again.
And here is another thing I have been saying since 2020.[1]
>In the past 3 years all major PR has twisted the word "Moore's Law" to just meant transistor improvement.
I wrote that in 2022 and was downvoted to oblivion.
It is sad we often have to debate about simple "facts". Only years later could we only accept it as truth.
[1]https://news.ycombinator.com/item?id=33888629
None of them do any fabrication which is what’s Moore’s Law applies to.
> In fact, he even said Intel could surpass the pace of Moore's Law at least until 2031 and has promoted "Super Moore's Law," a strategy to boost transistor count using 2.5D and 3D chip packaging technologies such as Foveros. Intel also often refers to this strategy as "Moore's Law 2.0,"
Regardless of transistors getting smaller, companies are outpacing Moore's law by making the CPU physically larger. They design the CPU so that it's composed of multiple chiplets. Since now there can be 7 chiplets that are closely connected, the # of transistors in one CPU is increasing faster than Moore's law. With the announcements about stacking CPU dies on top of each other in 3D, there will be even more chiplets in one CPU in the future.
Nvidia, Apple and AMD do not do that (at scale)
https://www.pcgamer.com/which-is-bigger-intel-amd-nvidia/
Intel is x10 the size of AMD.
They are also the only ones to have their own chip production capability. AMD and Nvidia are a Tawain Straits conflict liability.
I guess I counted on them pulling some unique witchcraft out of the hat that would circumvent that somehow
Still I think there is room for coming at problems via entirely different praradigms. See LLMs and their emphasis on parallel compute to get to a single answer.
Ok I will bite.
It is always easy to suggest Moore's Law isn't dead using Logarithmic scale. But if you only look at recent data. Let's take TSMC 10nm, the moment which TSMC achieved its leading edge status from Intel, you then have 7nm, 5nm which we are currently on, the 3nm which might [0] ship in early 2023, and the expected 2nm in 2025. That is 2017 to 2025. There is nothing 2x / 2 year within this period, even if you only use the best / peak quoted [1] density matrices.
Let me just give this quoted density number using their node name [2]; in Million Transistors per mm2.
2015 - Intel 14nm - 44.67 [3]
2017 - TSMC 10nm - 52.51
2018 - TSMC 7nm - 91.20
2019 - 178.68 ( Hypothetical of Intel 14nm lineage at 2x / 2 year )
2020 - TSMC 5nm - 171.30
2023 - TSMC 3nm - 292.21 ( EST )
2025 - IBM Research GAA 2nm - 333.33 ( EST ), TSMC GAA 2nm - ~500 ( EST ),
2025 - 1430 ( Hypothetical of Intel 14nm lineage at 2x / 2 year )
Notice where the trend starts to break? 2019 - 2020. [4]
And unless Intel or TSMC could adjust their 2025 - 2030 roadmap to somehow increase transistor density by 2.9x every 2 years, they would not be back in the same trajectory as the original trend. So it either follows Power Law, or the next 5 - 10 years will be a beep / outliner in Moore's law history.
And it is not only Jensen, CEO of AMD Dr. Lisa Su made similar comments on Moore's Law. And they are not wrong ( or uninformed, in fact they are too well informed ). In order to achieve 2x performance increase or 50% reduction in die size. Their Cost of Die, Cost of R&D purely in terms of design and fabrication are increasing. Their total unit cost are increasing. GPU vendors are much more sensitive to this since their performance scales extremely well with transistor count. That is why chiplet and packaging has become important to solve this cost issues. ( They are not silver bullet )
There are also problems with 3D Stacking and layering. Which I have seen far too many people being completely dismissive of it, is thermal dynamics. You cant have a hundred layer of compute with each layer using 10W if not higher. It wasn't until AMD made it absolutely clear with their V-Cache implementation, you cant have your SRAM layer on top of your compute layer due to heat issue did people start to realise their dream of a hundred layered GPU might not actually work. At least not between now and 2030.
Not only are DRAM not getting any price / bit reduction in the past 10 years. NAND may see similar fate. We are getting faster, and lower power DRAM, but we are certainly not getting any cheaper DRAM [5]. And that is ignoring a majority of DRAM revenues comes from LPDDR and not normal DRAM. Which has a higher price per GB. NAND may have one or two generation to go in terms of cost reduction. ( Also worth looking at HDD Cost / GB with similar trend. ) But those hundred layers of NAND are done by string stacking. Stacking up multiple of 60 / 70 layers of stacks which has higher yield. Currently Samsung is the only one doing 128 layer single stack. Cell sizes hasn't shrink much either due to error rate and cost until they moved to EUV. Moving from TLC to QLC and later PLC has diminishing returns. It may be worth pointing out the obvious, DRAM and NAND are commodities, and follows the rules of any commodity market. It is not that we are not getting any more cost / performance or IO or storage improvements. It's just their rate are slowing.
[0] Originally scheduled for 2022 iPhone launch ( as usual ) but TSMC announced they had a three month ( one quarter ) delay. Assuming yield were good enough and no contractual obliteration for Apple to be the first using their 3nm you might see other vendor launch using 3nm in early 2023.
[1] Peak Quoted Transistor Densities - Different Fabs may h...