> Already you can see this: whatever language-learning system we’re all meant to worship at the feet of ...
This is a ridiculous strawman.
> has now been trained on all the natural language that exists on the internet, in order to produce results which are not, in fact, acceptable. And there’s nowhere to go from here: there is no more training data.
Better algorithms and hardware? Also, no one has ever trained anything on "all the natural language which exists on the internet". Only a small (curated) fraction of that is captured in datasets. 45TB of text data was used to train GPT-3.
A kilobyte of text is roughly half a page of text. If the 45TB does not include the size of metadata, then that comes out to roughly 3,000 pages of text per living person on the planet. Especially when you consider how little of the world's population is online, and how much writing isn't preserved, that would probably be a double-digit percentage of the total extant digital written record of humans.
(This makes me think that the size does include metadata size or markup. But even if it includes those things, you're still probably looking at 0.1-10% of the extant digital written record. The only way it's not a significant fraction of all extant human writing is if the size is also including copious amounts of non-text data, such as embedded images or videos.)
You're also incorrect, although less wrong than OP. 45TB was the starting point (the Common Crawl snapshots, mostly). The actual amount of plain text fed in is much smaller, they filtered it down to like 500GBish and then didn't even train a full epoch. (The data scaling law is also favorable, so the dataset doesn't need to grow much even for much larger smarter models.)
> no-one wanted to run anything other than rather specialised programs on a Cray 1 or any of its descendants because it was just not very fast for that
That’s not at all true. Cray-1 and immediate descendants were just as good at scalar workloads as vector (their killer advantage over the likes of STAR-100 etc). Their major issues came from having crummy early 1960s-style memory management - never really fixed - making them pretty awful Unix (or even just shared) machines. As effectively single threaded workload batch targets under COS (regurgitated CDC yuck) they were the awesome Fortran machines they were designed to be.
But users were always hitting that device making it a bottleneck as job sizes grew, at least in my experience. Swap isn’t partial either, just roll out, roll it. Lustre btw was originally designed as a networked substitute HPC swap - the source of many original issues.
I didn’t say Unicos was crummy - it was the feel of the system with more than a small few on it.
It had amazing bandwidth to the swap device, so no, I've never seen interactive use be slow. Of course that's just my actual usage (NCSA, SDSC, Stennis, etc) and not the only possible experience.
RISC-V appears to be implementing vector in preference to SIMD.
Vector math units don't seem to be a fad from the age of Cray, and the RISC-V criticism appears to be that SIMD is too much of a niche specialization for a general purpose processor.
General Purpose computing is a series of nonlinear transformation. Batching vector operations scales the general case horizontally. It's lookups all the ways down.
What's the actual conclusion of this article? Is it that "general purpose computing" (which is defined as...what exactly? This is an enormous space of programs!) is not as fast as specific, fully hardware accelerated computations? This is hardly surprising, no? Of course specific things can be optimized better than trying to simultaneously optimize every program under the sun
I once interned at a Navy lab that worked on what amounted to ray-tracing in non-visible and radio spectrum. They had just bought a ~$2M Convex C3800 vector supercomputer and I had to benchmark it with the Lawrence Livermore Loops - a set of benchmarks of varying vectorizability.
I also somehow lucked into having a SGI $50k Indigo2 as my desktop because no one else there wanted it. It was bought for visualizations that no one worked on. So I built and ran the LLL there too.
Well, our team's code was about 75-80% vectorizable, and on the similar LLL benchmarks the Indigo2 just crushed the Convex. The Convex didn't start winning until loops were >90% vectorizable.
People were pissed, but that led to a lot of SGIs being bought.
Article flaws:
- ascribes demise of vector supercomputers to AI winter; false, supercomputers did just fine post-AI winter. It was Intel’s economies of scale and growing availability of open source software for managing clusters that eventually killed them.
- doesn’t discuss the M1 in any depth, just asserts it’s a vector supercomputer; false, not only does the M1 have a number of specialized units (beyond generic vector units) accelerating demanding tasks, it also has monster ILP, with 8-way integer dispatch and a huge reorder buffer.
- only discusses machine learning and implies it’s the primary workload; false, a lot of workload on a typical end user machine is video.
With 128 bit vectors, the M1 is less of a vector computer than the x86 competition, which all have 256-bit or 512-bit vectors.
As you note, the firestorm core's real strength is in its scalar execution (and when it comes to the vector units, it can do twice the operations/cycle as the x86 competition, giving it comparable throughput even for vector workloads).
So the M1 generally impressing people in its performance seems like it'd support the author's point, and anecdotes about it underwhelming would contradict it?
Unless the author was referring to the specialized matrix multiply unit and neural engine?
> There are apocryphal reports that Apple M1 systems are not as fast as people have been led to believe for general-purpose programs. That’s unsurprising.
I’m surprised, to be honest. In my testing M1 has done extremely well on general-purpose workloads, so I’m curious what this refers to.
If anyone says the AI revolution has failed because it hasn't given us strong AI roll your eyes and go the other way.
Or if you want to engage, ask them what their definition of intelligence is and how would they benchmark it.
And once they come up with some kind of exotic benchmark they are sure a computer couldn't pass, ask how many humans would it also exclude.
The best example of this was someone I talked to who thought embodiment was the key to Strong AI. Their definition of intelligence excluded Stephen Hawking.
22 comments
[ 3.9 ms ] story [ 58.0 ms ] threadThis is a ridiculous strawman.
> has now been trained on all the natural language that exists on the internet, in order to produce results which are not, in fact, acceptable. And there’s nowhere to go from here: there is no more training data.
Better algorithms and hardware? Also, no one has ever trained anything on "all the natural language which exists on the internet". Only a small (curated) fraction of that is captured in datasets. 45TB of text data was used to train GPT-3.
A kilobyte of text is roughly half a page of text. If the 45TB does not include the size of metadata, then that comes out to roughly 3,000 pages of text per living person on the planet. Especially when you consider how little of the world's population is online, and how much writing isn't preserved, that would probably be a double-digit percentage of the total extant digital written record of humans.
(This makes me think that the size does include metadata size or markup. But even if it includes those things, you're still probably looking at 0.1-10% of the extant digital written record. The only way it's not a significant fraction of all extant human writing is if the size is also including copious amounts of non-text data, such as embedded images or videos.)
That comes out to about 3 pages by your measure.
(So much for decimal-based systems making it easy to do calculations in your head...)
That’s not at all true. Cray-1 and immediate descendants were just as good at scalar workloads as vector (their killer advantage over the likes of STAR-100 etc). Their major issues came from having crummy early 1960s-style memory management - never really fixed - making them pretty awful Unix (or even just shared) machines. As effectively single threaded workload batch targets under COS (regurgitated CDC yuck) they were the awesome Fortran machines they were designed to be.
Even though the memory management was antiquated, there was an enormous, fast swap device available on every vector Cray.
I didn’t say Unicos was crummy - it was the feel of the system with more than a small few on it.
Vector math units don't seem to be a fad from the age of Cray, and the RISC-V criticism appears to be that SIMD is too much of a niche specialization for a general purpose processor.
https://medium.com/swlh/risc-v-vector-instructions-vs-arm-an...
I also somehow lucked into having a SGI $50k Indigo2 as my desktop because no one else there wanted it. It was bought for visualizations that no one worked on. So I built and ran the LLL there too.
Well, our team's code was about 75-80% vectorizable, and on the similar LLL benchmarks the Indigo2 just crushed the Convex. The Convex didn't start winning until loops were >90% vectorizable.
People were pissed, but that led to a lot of SGIs being bought.
So the M1 generally impressing people in its performance seems like it'd support the author's point, and anecdotes about it underwhelming would contradict it?
Unless the author was referring to the specialized matrix multiply unit and neural engine?
I’m surprised, to be honest. In my testing M1 has done extremely well on general-purpose workloads, so I’m curious what this refers to.
Or if you want to engage, ask them what their definition of intelligence is and how would they benchmark it.
And once they come up with some kind of exotic benchmark they are sure a computer couldn't pass, ask how many humans would it also exclude.
The best example of this was someone I talked to who thought embodiment was the key to Strong AI. Their definition of intelligence excluded Stephen Hawking.