Spoiler: the “Huang's law” this author is trying to coin is that AI processors (surely this means ML) are still following something like Moore's Law in terms of rate of improvement of processing speed per year.
Which is cool I suppose but comparing it to Moore's Law seems a bit unfair.
Moore's law benefits all applications, by increasing performance across the board.
This new "Huang's law" increases performance for certain critical applications, while presumably decreasing performance of other applications (at fixed cost).
The difference is generally increasing transistor density vs. specializing hardware to perform critical tasks (like graphics rendering or tensor algebra).
That's not a generalization, it's a misunderstanding. Not every program can benefit from doubling transistors because not every program is written to minimize serial computations and the impact of memory latency.
The reason Moore's law was insightful was because he didn't make the simplistic generalization of computing power (which is more difficult to quantify anyway) and looked at what was really happening on a fundamental level from an engineering perspective.
Yes, the competition for attention used to be at the level of the whole newspaper, or the whole magazine.
Now the attention competition is at the article level, and the headline is near the top of the funnel. To not be subjected to (as much of) this insanity, you can use a smart aggregator like HN or a well-moderated subreddit.
Whether you call it Huang's Law or recast Moore's Law to mean continuing performance improvements on the workloads where it most matters, there's definitely a widespread industry interest in promoting "Moore's Law is continuing." You saw it in full display at Hot Chips this year.
And it's not even unreasonable. We collectively used transistors in general purpose CPUs in a way that wasn't most efficient but made it easy from a software and operational perspective. And that's changing now.
Do you mean that we clung to single threaded compute for too long? Like we spent too many transistors on just making one fast core because it's easy to write code to use one core?
Intel at least absolutely clung to clock speed and one core for software reasons--in large part because of pressure from a certain very important software partner.
But multicore on a single (mostly) general purpose architecture (leaving aside vector instructions) is still easier from a software perspective than the very heterogeneous architecture--vector, array, FPGA, etc.--we're increasingly adopting.
Yes. A senior Intel exec told me around the time multicore was coming in something to the effect of "Of course we knew we had to switch from frequency to multicore but Microsoft was terrified of the software implications of moving away from pure instruction level parallelism."
This was of course somewhat self-serving given all the criticism Intel caught for being slow to adopt multicore. But there were certainly concerns about software support at the time, especially on the desktop.
(In the end, it ended up not being as big a problem as some were fearing for various reasons. But it was a real fear at the time.)
>Why were either allowed to become laws. Moore's law was a little confusing in high school
Nobody allowed Moore's Law to become a law.
The so-called Moore's Law is really just Gordon Moore's Extrapolation of an observation. In 1965, he observed that transistor count was doubling every ~12-18 months. He extrapolated that this trend could continue for ~10 more years to 1975.
But journalists and readers like snappy poetic words. "Extrapolation" is a long-winded 5-syllable noun so it's no surprise that the 1-syllable "law" was used instead. Wikipedia says: shortly after 1975, Caltech professor Carver Mead popularized the term "Moore's law".[1]
Although calling it "Moore's Extrapolation" would be more accurate, it just doesn't roll off the tongue as nicely. Sometimes humans value the sing-song poetry of a meme more than its dictionary accuracy.
I don't know for sure but Moore's Law was part of a certain expectation that was set for the industry and the market. For better or worse, this expectation tended to force various decisions that kept advances close to the predicted line.
The business application of it meant profits from each generation could fund the higher costs of each succeeding generation.
Now we are in the tens of $billions for a new generation, with much smaller increases in performance, and it is faltering. Will we spend hundreds of $billions on factories for smaller transistors? Or will we find a more cost-effective way to get more performance?
Carver Mead often gets credit for Lynn Conway's work.
If it's about device physics or transistor and single-gate layout, it is likely Mead. If it's about organizing logic functions on a chip, it is more likely Conway.
She relied on Mead for a lot of the publicizing of her design methodology, because he was better on stage.
This is silly. Just because you can build bigger chips doesn't mean you have anything similar to Moore's law, which was more about the economics of semiconductor manufacturing.
Yes, except the important parts. In Moore’s law scaling, you reduce the cost per transistor, and because of that there is money for capex and the consumer benefits.
In Huangs law, si is commoditized, so the benefit accrues to the designer and software makers, and really only in narrow domains.
It's interesting that they picked Amazon Go as an example of the power of AI (or ML) when in actuality most of the grunt work in those stores is done by "dumb" sensors AFAIK. The store "knows" that an item has been picked up from a certain shelf and thus it assumes that the item is what is usually on that shelf, it has little to do with either CV or AI in general, just careful placement of the items and weight sensing. I think that if you let a few kids roam free in there picking stuff up and setting it down in an unexpected place the magic of AI would dissolve into thin air.
I've been a couple times to one of the SF locations with coworkers. We definitely tried to trick system by passing items between us, and putting items down in random locations and then picking them back up later. When we left the store everyone got charged for the items they had on them.
I think you are too dismissive in saying its "just careful placement of the items and weight sensing".
I went to one of these stores in Seattle on a weekday afternoon with a couple friends. It was literally packed with people. We tried breaking the system by leaving items in random places and exchanging them between us. It still charged me accurately for a couple bottled waters, a sandwich, and some other snacks.
Except that cameras in stores have been used for anti-shoplifting for decades. They've been used for more subtle things like monitoring tables at casinos.
I mean, I'm just asking the question how people know for sure Go isn't (at least partly) a Turk. It seems like an obvious thing to ask? But I first came to wonder if there's manual component involved after shopping a few times at Go. When it was busy, it took sometimes well over an hour to get a receipt. When it was empty, it was immediate. How come?
Also, the accuracy is completely anecdotal both ways... at least one person I visited the store with never got billed.
Did you forget about the aspect of the store that knows who picked up what? You focused completely on the irrelevant aspect. The store tracks every customer to those dumb sensors and then keeps a running tally of their cart. The dumb sensors are used as double confirmation to handle occlusion.
Carmack was actually mentioning he asked Jim Keller about Moore's Law. For more details about Jim's thoughts I recommend this interview https://www.youtube.com/watch?v=Nb2tebYAaOA
I read the whole article a few times. But nothing in there explains why Nvidia wants ARM.
Is this one of those PR articles [1] that try to convince public about the ARM-Nvidia deal being great for the industry? ( While I prefer ARM to stay independent, I am not exactly against Nvidia acquiring it )
Where's the data that says Moore's law no longer holds? I see comments and articles asserting this but everytime with evidence. The data that I do find certainly still suggests Moore's law is doing fine.
I'm writing this comment on an 11 year old machine. It doesn't feel much slower than the 1 year old machine that I use at work. The number of transistors in its processor is also not 1/32-th or whatever of the office-machine's (especially if you don't count cache). It's processor has 45nm feature size, the modern machine's CPU is 14nm. In what way do you think does Moore's law still hold?
first generation core i7 (960), 45nm, october 2009, 263mm^2 die area, 4 cores, 4 threads, 3.2-3.46ghz 731 million transistors, 130W TDP,
tenth generation core mobile i7 (i7-10875H) 14nm, Q2-2020. 25mm^2 per set of two cores[0], 8 cores, 16 threads, 2.3-5.1ghz. 43 million transistors per square mm[1] -> over 4.3 billion transistors, 45W TDP.
So you have 4 times the threads, at a hair under twice the speed (without stressing turbo boost that much) at 1/3rd of the power at the wall. from top of the line desktop CPU in october of 2009 to "it's ok" mobile CPU from april of this year.
Intel stopped publishing (where i can find it) die size or transistor counts, i found the [0] and [1] information on a comparison between TSMC's 7nm fab runs and Intel's claimed 14nm fab runs; and i was only able to very briefly confirm this for the mobile i7 tenth gen and not the desktops. The desktop CPUs are like 400mm^2 larger in physical size, no idea about die size. Sorry. Intel stopped publishing those i guess, seems around the time that ryzen came out or slightly before.
edit: 11 years is 5.5 doublings of transistors IIRC, so you should really compare like to like, but i don't have that much free time. My laptop's CPU has 7.8 billion transistors just for the cores, and an additional 2 billion for I/O.
sorry. this information isn't in a precise place and i am unused to HN forms.
Moore's law is still on pace but Dennard scaling has broken down for ~15 years now. That means we're not getting the same speedups as we were used to and that gets explained by "Moore's law has broken down" as a title.
Dennards law held until the approximately linear relations that made it work ended. Delay stopped being entirely gage, and more importantly, voltage couldn’t drop forever, due to material limits and intrinsic silicon limits.
the trick is to look at transistor density. people will take transistor counts of say, the eypc cpu but that is 9 chiplets for a huge total die area. transistor counts doubling via doubling die area isn’t moores law
They're both really Wright's law. Theodore Wright published a paper in 1936 describing how aircraft production costs decline at a constant percentage with each doubling of production. The formal term is manufacturing learning curve.
Well, to be specific that seems more about manufacturing efficiencies (optimizing waste, batches, waiting time, etc as a line learns how to be more efficient) while this law and Moore's law are more about the intrinsic design of the chips (hardware and software).
Plus Moore's law has a time based element. I don't think anyone could have got to a 5nm process in 1970 say simply by deciding to up the volume of transistors being produced.
Generallys Moore’s law is time based, Wright’s law is production based. Both are great models, but Ark Invest’s researcher found Wright’s law to be more precise.
1. Huang was seen sitting at the wheel of a Mercedes Benz S class, which has a stupendous amount of NVIDIA silicon. That's great, but the bulk of the market is for the electronics that go into a Chevy.
NVIDIA owns the highly profitable top end, but competitors (for gfx, robot brains, ...) are so disorganized that NVIDIA should move downmarket quickly to capture that terrain even if the profit margin is less.
2. A cynical take is that of "Patronage Networks". That is if Softbank can't manage "Return of Capital" to its investors, Masayoshi Son might come to his room and find a katana to cut himself with. The $40B from ARM goes a good way to "saving his ass" and might mean Huang owes him one in the same sense that politician owes something to voters.
The author fundamentally misunderstands how the industry functions.
Money wise, the ARM acquisition for this amount of money makes no sense if they want to save on core, or architectural licenses.
With the amount of money, and engineers Nvidia has, they can easily bootstrap a CPU ISA on their own.
It was by far not their top priority.
And in their function relevant to the "AI" trend as giant linear algebra crunchers, GPUs don't need much help from anything else.
And in longterm, basically anything you do pales to logarithmic scaling of PPW/relative performance. A s..ttier, and cheaper chip few years down the line invariably beats current best.
In practical terms, the winner in the GPU/CPU/WhateverPU is decided by who gets the better fab contract.
I still cannot comprehend the significance of Moore's law. Although the hardware keeps getting more sophisticated, I cannot perceive that today's computer is 100x faster or more than 20-30 years ago. Perhaps the software has been getting bloated to counteract or the speed increase is not real.
For AI chip, will we fall into the same situation? will AI algorithm still focus on efficiency?
For games, which many are pretty run entirely locally, things are noticeably faster and higher quality.
For networked applications, I feel like the much bigger limit is the network. The Tidal streaming service ran into this with its business model; streaming lossless, high quality audio is a bit of a moot point if the network doesn't allow you to download all of it without noticeable buffering when listening to music.
While processors have become more powerful, we're now coding our apps in Javascript, which negates any progress described by Moore's Law. Of course I'm being hyperbolic but you get my point.
Thanks, finally someone agree with my observation. :-) I got deduction point for saying that statement. For me, it is not right to use 100x speed increase for eye candy feature or using bloated Javascript libraries.
In certain limited domains, increased hardware capability shows up in proportionally more useful or meaningful results. In most uses the difference is squandered on eye candy or abstraction overhead.
We see it in video game rendering and weather prediction, but not in web page presentation, where it is squandered on pointless animation.
The usefulness of a computer isn't linear with speed. Usefulness is application-specific, of course, but in most applications usefulness is somewhere between log(speed) and sqrt(speed).
For example:
If you're solving NP problems (common in optimization), the size of the problem you can handle in a given amount of time is proportional to log(speed).
If you're adding features to software, then any given feature requires some amount of compute power. Experience shows these to be power-law distributed. Early software versions started with the cheapest features, so now we're adding things on the right side of the distribution. So each new feature is likely to require additional compute power increasing like a power law.
For a concrete example: when OSX added transparency and shadows to windows, it required every pixel in a window to be operated on twice. A doubling of a major component of the screen update time for a feature which only slightly increases user happiness/productivity. It had to wait until GPUs were fast enough to support it.
In machine learning, the accuracy of results increases much less than linearly, somewhere between log(# parameters) and sqrt(# parameters). And the inference time is linear in the number of parameters.
So it's a good thing that compute power is increasing exponentially or else progress would be really slow.
Even trivially a 4k@120hz monitor is 108x the pixels as a 630x480@30hz monitor and you can clearly render much more complex scenes per pixel. It seems trivial to me to see 100x the speed without relying on benchmarks.
That said, speed as a derivative of how long the user waits has not changed simply because humans have not changed what they consider acceptable wait times.
For a concrete example of what 100x feels like, importing digital camera imagery into a gallery in 1999-2000 was on the order of 10-15 seconds per image (for a ~2 MP image, which was current state-of-the-art for resolution). Multi-core and hyperthreading was still years away.
PhotoStructure, a self-hosted photo and video manager, uses libvips (a wonderfully performant image library) and is multi-processed and multi-threaded. Typical images are >10x the resolution (24-40MP images are standard from DSLR) yet several are processed in parallel every second.
I’m wondering if there are any firewall/router/load balancer type of devices that use GPUs for their parallel processing capabilities (or whether that’s even possible). Anyone familiar with this ecosystem to comment?
One of the main system challenges has been efficient high-bandwidth and low-latency delivery of packet flows between NICs and GPUs. NVIDIA has their cuVNF library (part of the Aerial SDK) that works with GPU Direct NICs and extends the DPDK toolkit to accomplish this: https://developer.nvidia.com/aerial-sdk
The rationale put forward in this piece for Nvidia wanting to acquire Arm is essentially that AI compute will move to the 'edge' and that as Arm dominates compute in the edge Nvidia needs to defend its position and to follow that compute. So in one sense it's defensive of Nvidia's existing position.
Once again though did Nvidia need to buy Arm to do this? It could license its own IP and /or compete with SoC's of its own for this market. As in the server market the advantage of ownership (and what is worth $40bn) is the advantage it gives Nvidia in information and control of what is done with Arm IP.
79 comments
[ 4.8 ms ] story [ 144 ms ] threadAfter CUDA we had to use Tensor cores, later float16, now sparce matrix multiplications that work because deep learning uses lots of RELUs.
[1] https://youtu.be/fBRMaEAFLE0?t=651 "Free with every CD, a Cray supercomputer"
Which is cool I suppose but comparing it to Moore's Law seems a bit unfair.
Also it is extra silly because Moore's law is about transistor density and transistor cost.
Moore's law benefits all applications, by increasing performance across the board.
This new "Huang's law" increases performance for certain critical applications, while presumably decreasing performance of other applications (at fixed cost).
The difference is generally increasing transistor density vs. specializing hardware to perform critical tasks (like graphics rendering or tensor algebra).
The reason Moore's law was insightful was because he didn't make the simplistic generalization of computing power (which is more difficult to quantify anyway) and looked at what was really happening on a fundamental level from an engineering perspective.
Now the attention competition is at the article level, and the headline is near the top of the funnel. To not be subjected to (as much of) this insanity, you can use a smart aggregator like HN or a well-moderated subreddit.
And it's not even unreasonable. We collectively used transistors in general purpose CPUs in a way that wasn't most efficient but made it easy from a software and operational perspective. And that's changing now.
But multicore on a single (mostly) general purpose architecture (leaving aside vector instructions) is still easier from a software perspective than the very heterogeneous architecture--vector, array, FPGA, etc.--we're increasingly adopting.
are you referring to microsoft?
This was of course somewhat self-serving given all the criticism Intel caught for being slow to adopt multicore. But there were certainly concerns about software support at the time, especially on the desktop.
(In the end, it ended up not being as big a problem as some were fearing for various reasons. But it was a real fear at the time.)
Nobody allowed Moore's Law to become a law.
The so-called Moore's Law is really just Gordon Moore's Extrapolation of an observation. In 1965, he observed that transistor count was doubling every ~12-18 months. He extrapolated that this trend could continue for ~10 more years to 1975.
But journalists and readers like snappy poetic words. "Extrapolation" is a long-winded 5-syllable noun so it's no surprise that the 1-syllable "law" was used instead. Wikipedia says: shortly after 1975, Caltech professor Carver Mead popularized the term "Moore's law".[1]
Although calling it "Moore's Extrapolation" would be more accurate, it just doesn't roll off the tongue as nicely. Sometimes humans value the sing-song poetry of a meme more than its dictionary accuracy.
[1] https://en.wikipedia.org/wiki/Moore%27s_law#History_of_the_c...
The business application of it meant profits from each generation could fund the higher costs of each succeeding generation.
Now we are in the tens of $billions for a new generation, with much smaller increases in performance, and it is faltering. Will we spend hundreds of $billions on factories for smaller transistors? Or will we find a more cost-effective way to get more performance?
But the expectation is that a lot of continued performance gains will have to come through other means. These are some of the things that were talked about at the Hot Chips conference: https://enterprisersproject.com/article/2020/9/moores-law-wh...
If it's about device physics or transistor and single-gate layout, it is likely Mead. If it's about organizing logic functions on a chip, it is more likely Conway.
She relied on Mead for a lot of the publicizing of her design methodology, because he was better on stage.
In Huangs law, si is commoditized, so the benefit accrues to the designer and software makers, and really only in narrow domains.
I've been a couple times to one of the SF locations with coworkers. We definitely tried to trick system by passing items between us, and putting items down in random locations and then picking them back up later. When we left the store everyone got charged for the items they had on them.
I think you are too dismissive in saying its "just careful placement of the items and weight sensing".
Go has relatively few stores. Seems like the sort of thing you can still do manually, especially if the model expresses doubt.
At the scale they're operating at and the accuracy you point out, to me it's more plausible they're having humans assist.
I mean, I'm just asking the question how people know for sure Go isn't (at least partly) a Turk. It seems like an obvious thing to ask? But I first came to wonder if there's manual component involved after shopping a few times at Go. When it was busy, it took sometimes well over an hour to get a receipt. When it was empty, it was immediate. How come?
Also, the accuracy is completely anecdotal both ways... at least one person I visited the store with never got billed.
- the exact same article for those who don't have a wsj-account
Is this one of those PR articles [1] that try to convince public about the ARM-Nvidia deal being great for the industry? ( While I prefer ARM to stay independent, I am not exactly against Nvidia acquiring it )
[1] http://www.paulgraham.com/submarine.html
tenth generation core mobile i7 (i7-10875H) 14nm, Q2-2020. 25mm^2 per set of two cores[0], 8 cores, 16 threads, 2.3-5.1ghz. 43 million transistors per square mm[1] -> over 4.3 billion transistors, 45W TDP.
So you have 4 times the threads, at a hair under twice the speed (without stressing turbo boost that much) at 1/3rd of the power at the wall. from top of the line desktop CPU in october of 2009 to "it's ok" mobile CPU from april of this year.
Intel stopped publishing (where i can find it) die size or transistor counts, i found the [0] and [1] information on a comparison between TSMC's 7nm fab runs and Intel's claimed 14nm fab runs; and i was only able to very briefly confirm this for the mobile i7 tenth gen and not the desktops. The desktop CPUs are like 400mm^2 larger in physical size, no idea about die size. Sorry. Intel stopped publishing those i guess, seems around the time that ryzen came out or slightly before.
edit: 11 years is 5.5 doublings of transistors IIRC, so you should really compare like to like, but i don't have that much free time. My laptop's CPU has 7.8 billion transistors just for the cores, and an additional 2 billion for I/O.
sorry. this information isn't in a precise place and i am unused to HN forms.
https://en.wikipedia.org/wiki/Dennard_scaling
If i could swap out a chiplet i might be inclined to agree with you.
https://ark-invest.com/wrights-law/
1. Huang was seen sitting at the wheel of a Mercedes Benz S class, which has a stupendous amount of NVIDIA silicon. That's great, but the bulk of the market is for the electronics that go into a Chevy.
NVIDIA owns the highly profitable top end, but competitors (for gfx, robot brains, ...) are so disorganized that NVIDIA should move downmarket quickly to capture that terrain even if the profit margin is less.
2. A cynical take is that of "Patronage Networks". That is if Softbank can't manage "Return of Capital" to its investors, Masayoshi Son might come to his room and find a katana to cut himself with. The $40B from ARM goes a good way to "saving his ass" and might mean Huang owes him one in the same sense that politician owes something to voters.
Money wise, the ARM acquisition for this amount of money makes no sense if they want to save on core, or architectural licenses.
With the amount of money, and engineers Nvidia has, they can easily bootstrap a CPU ISA on their own.
It was by far not their top priority.
And in their function relevant to the "AI" trend as giant linear algebra crunchers, GPUs don't need much help from anything else.
And in longterm, basically anything you do pales to logarithmic scaling of PPW/relative performance. A s..ttier, and cheaper chip few years down the line invariably beats current best.
In practical terms, the winner in the GPU/CPU/WhateverPU is decided by who gets the better fab contract.
For AI chip, will we fall into the same situation? will AI algorithm still focus on efficiency?
For networked applications, I feel like the much bigger limit is the network. The Tidal streaming service ran into this with its business model; streaming lossless, high quality audio is a bit of a moot point if the network doesn't allow you to download all of it without noticeable buffering when listening to music.
Seems pretty obvious when comparing, say, Duke Nukem 3D with Far Cry 5. Or Wolfenstein 3D with Assassin's Creed Odyssey.
We see it in video game rendering and weather prediction, but not in web page presentation, where it is squandered on pointless animation.
For example:
If you're solving NP problems (common in optimization), the size of the problem you can handle in a given amount of time is proportional to log(speed).
If you're adding features to software, then any given feature requires some amount of compute power. Experience shows these to be power-law distributed. Early software versions started with the cheapest features, so now we're adding things on the right side of the distribution. So each new feature is likely to require additional compute power increasing like a power law.
For a concrete example: when OSX added transparency and shadows to windows, it required every pixel in a window to be operated on twice. A doubling of a major component of the screen update time for a feature which only slightly increases user happiness/productivity. It had to wait until GPUs were fast enough to support it.
In machine learning, the accuracy of results increases much less than linearly, somewhere between log(# parameters) and sqrt(# parameters). And the inference time is linear in the number of parameters.
So it's a good thing that compute power is increasing exponentially or else progress would be really slow.
That said, speed as a derivative of how long the user waits has not changed simply because humans have not changed what they consider acceptable wait times.
PhotoStructure, a self-hosted photo and video manager, uses libvips (a wonderfully performant image library) and is multi-processed and multi-threaded. Typical images are >10x the resolution (24-40MP images are standard from DSLR) yet several are processed in parallel every second.
This is a developing area with a number of research and development efforts ongoing looking at basic packet filtering flows, for example: https://ieeexplore.ieee.org/document/7019193 https://arxiv.org/pdf/1312.4188.pdf
and AI-driven approaches like CyBERT showing promise: https://medium.com/rapids-ai/cybert-28b35a4c81c4
One of the main system challenges has been efficient high-bandwidth and low-latency delivery of packet flows between NICs and GPUs. NVIDIA has their cuVNF library (part of the Aerial SDK) that works with GPU Direct NICs and extends the DPDK toolkit to accomplish this: https://developer.nvidia.com/aerial-sdk
However, as for integrated devices, I'm familiar with at least one that claims to be on the market: http://www.h3c.com/en/About_Us/News___Events/News/201907/121...
Once again though did Nvidia need to buy Arm to do this? It could license its own IP and /or compete with SoC's of its own for this market. As in the server market the advantage of ownership (and what is worth $40bn) is the advantage it gives Nvidia in information and control of what is done with Arm IP.