These large models are going to be the operating system of the future. When they are going to run efficiently on edge we are going to see an explosion of automation. Being so large they are not only more accurate, but also quicker learners. So less initial effort to start a project.
Google is teasing us with leaks about Pathways[1], but not releasing a paper or demo yet.
I call them multi-multi-models: multi-modality, multi-task, multi-language, multi-memory (besides the processing memory it can have search engine in the loop and episodic memory). Somebody's going to make it in the next couple of years.
To start an AI company you generally don't need much. A few thousands in GPUs and you are good to go. If AI researchers feel they are underpaid there is literally nothing stopping them from starting a company themselves.
that is absolutely not true. I'm not sure where you are getting your information, but read anything from an angel investor.
most AI start ups fail because they go into huge debt getting correct data and training models, which end up bankrupting them in the process with nothing to show for it
Sorry, are you claiming that the French Revolution occurred due to bad labor practices?
Or did trust busting and unionization in the late 19th century US utilize guillotines?
Either of those would certainly be a.. uh.. let's go with "novel" retelling of the history!
Or did you mean Red October? Because I'm pretty sure /that/ revolution didn't actually solve any problems for the worker in the way things actually panned out
You’re conflating two things. Feudalism is like slavery: taking credit for someone else’s labor. Automation removes the need for labor. Automation has almost universally led to quality of life improvements, not revolution.
Nowadays startups often delay going to the stock market, and remain private while they're growing fast. It's only when they are growing more slowly and the VCs want to get out that they go public.
This way, the average investor usually does not reap outsized benefits from the ride.
The nice part about it is that everyone will have access to great models for direct use, fine-tuning and adaptation. But if people don't take this freedom and are content with simply consuming applications, then they depend on those who deploy the apps.
IMHO (as a AI researcher) 'multi-modality, multi-task, multi-language, multi-memory' may take longer than you think. Doing multi modal learning is quite hard due to lack of labeled data, a lot of tasks either don't offer clean plentiful datasets or require RL / human in the loop learning and not just supervised or self supervised learning, and on and on. Certainly a lot of progress will be made and partial automation is likely to speed up (as has happened with Copilot), but full automation seems to be a good while away.
Plug - I expand on this a bit in this blog post 'The Inherent Limitations of GPT-3' (which was on front page of HN a few weeks back and led to a good deal of discussion ) https://lastweekin.ai/p/the-inherent-limitations-of-gpt-3
Define "accessible". Very modest hardware (sub $100 and certainly sub $1000) can train random forests very quickly. This is going to be close to SOTA on basically any classification task.
Are random forests new? no. SOTA in what? Image classification, obviously no. But they are still going to perform very well in the vast majority of business problems in classification.
> speedup we need for ML to be truly accessible for the average developer. 50x? 100x?
The data scale (and access) is often more important than the modelling, and that doesn't have a technological fix, or at least not a purely technological fix.
What I'm saying is the hardware is less of a problem than access to data, for most problems that would get you out of what you can reasonably do on a consumer accessible (cost) machine.
Unless we have widely available pre-trained models with generic knowledge that is easily transferable. Such a thing is obviously difficult, but it isn't impossible if we're talking about the future.
The trick for ML accessibility is minor finetuning the existing models that have been trained on these super training models so that users can get similar performance, and lately companies have been more open about releasing models open source.
Google is all over this -- there are at least two science-fiction level videos on YouTube of actual field experiments using ML, each with a very non-USA/third world person narrating.. the view count is in the low four digits both times
one of them is related to a paper in Nature this year on "democratizing ML" .. the MOSAIK project iir
edit- hilariously, Google Search wont let me look for MOSAIK instead correcting the spelling.. and when forced, many similarly named projects that are not it, are ahead.. the one referenced here is
A generalizable and accessible approach to machine learning with global satellite imagery
The article is a bit lacking in details. Here are some changes in ML performance that are happening today and don't require increased transistor counts:
You can overfit your data by coming up w/ a model that is very specific to it. Consider an n dimensional data set where one variable is a boolean.
A researcher might notice that the True and False cases are different enough to warrant their own algorithms. Say they now fit a lower dimensional line to the True and False cases.
They've just performed gradient descent but the grad student was the gradient. It's a common joke in CS/ML departments.
(This is an obviously simple example, but the point is that hyperparameter tuning and algorithm selection _are a part of your algorithm_ and the data you're looking at while doing so is part of your training set)
> companies employ grad-student descent (hire a bunch of PhDs and task them with improving MLPerf results)
No joke, always cheaper than massive optimization tasks. I’ve managed a team working on automl for years and I can say with certainty, the cheapest option is always hiring interns (grad or bachelor students) or a hackathon
Neural architecture search is finding more efficient architectural building blocks which translate to better quality models with fewer overall parameters.
There's a lot of work going on right now to improve transformer architectures in particular; I think once the dust settles they will be massively more efficient than the current models. Efficient transformers often also introduce an inductive bias, which ends up improving model quality. (similar to replacing fully connected networks with convolutional networks; more efficient, but also carrying an inductive bias that improves model quality.)
We're also realising we don't need full transformers all the time. Token mixing might suffice, there are quite a few papers replacing the core of the transformer with something simpler.
"The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin. The ultimate reason for this is Moore's law, or rather its generalization of continued exponentially falling cost per unit of computation."
You're right about literal Moores Law, but its benefits are much diminished, of performance, power usage and cost. My understanding is the main problem is heat dissipation.
I've long predicted macrochips to solve this, brute force of huge quantities of silicon, but it hasn't really happened.
Graphics cards and neural accelerators have continued an exponential improvement in performance per watt dollar, which is the intuitive thing that people often mean when they say Moore's law.
I guess it's inevitable that we'll reach a "hard" limit at some point, with exponential increase in density (electrons and photons have a "size"). But I don't know how that translates to current densities (how many atoms in a gate on the Apple m1 chips, and what is the theoretical minimum?).
Did he make a clear statement on cost? On a skim of: https://newsroom.intel.com/wp-content/uploads/sites/11/2018/... I see costs mentioned - but I can't immediately see a clear statement - to tie it in with transistor increase - but I might be overlooking something?
On page two he charts cost per transistor dropping roughly an order of magnitude every 5 years. Since the the "law" of doubling transistors is on a two year time span, this roughly matches.
If you're going strictly on number of transistors, I think we're still going up up and away. If you're talking about performance, which is what most folks really care about, we really are hitting thermal limits especially with consumer hardware.
AMD, Nvidia and intel are all hitting the 'heat wall' with their hardware just trying to cram more into smaller nodes and boost clocks. Nvidia's 4000 series cards are rumored to be 500W cards which is well into 'space heater' territory.
Have we considered yet whether this is, to quote Stephen Diel, another handwavy technobabble nothingburger? I have very serious misgivings about all this what we currently call AI. Also, even if it is something, is it beneficial to all? Sure it can be beneficial to some corporates/VC but we didn't stop at the dawn of social media to think on the societal impact and we already know there's a lot of algorithmic bias being cooked here.
If I'm reading the plot from the article right, the most impressive gains have happened in the first half of 2019. Since then, the progress has been notably slower, with year-on-year performance increases for some network architectures falling behind Moore's law already.
48 comments
[ 7.0 ms ] story [ 120 ms ] threadGoogle is teasing us with leaks about Pathways[1], but not releasing a paper or demo yet.
[1] https://blog.google/technology/ai/introducing-pathways-next-...
I call them multi-multi-models: multi-modality, multi-task, multi-language, multi-memory (besides the processing memory it can have search engine in the loop and episodic memory). Somebody's going to make it in the next couple of years.
When the working class had this problem a couple centuries ago, it was solved by building guillotines. I hope we can do better this time.
Or did trust busting and unionization in the late 19th century US utilize guillotines?
Either of those would certainly be a.. uh.. let's go with "novel" retelling of the history!
Or did you mean Red October? Because I'm pretty sure /that/ revolution didn't actually solve any problems for the worker in the way things actually panned out
This way, the average investor usually does not reap outsized benefits from the ride.
The nice part about it is that everyone will have access to great models for direct use, fine-tuning and adaptation. But if people don't take this freedom and are content with simply consuming applications, then they depend on those who deploy the apps.
Plug - I expand on this a bit in this blog post 'The Inherent Limitations of GPT-3' (which was on front page of HN a few weeks back and led to a good deal of discussion ) https://lastweekin.ai/p/the-inherent-limitations-of-gpt-3
Right now we seem to be into a kind of "mainframe phase", where running big models is super expensive.
I wonder what kind of speedup we need for ML to be truly accessible for the average developer. 50x? 100x?
The data scale (and access) is often more important than the modelling, and that doesn't have a technological fix, or at least not a purely technological fix.
one of them is related to a paper in Nature this year on "democratizing ML" .. the MOSAIK project iir
edit- hilariously, Google Search wont let me look for MOSAIK instead correcting the spelling.. and when forced, many similarly named projects that are not it, are ahead.. the one referenced here is
1. newer optimizations algorithms converge faster (Adam, adagrad)
2. lower-precision numerics can be used (TF32 is 19bit, bf16 is popular)
3. matrix multiplications are implemented in hardware (tensor cores, AMX)
4. neural networks are compiled directly (TVM, halide)
5. multi-device network topologies are tuned for training (e.g. nvlink improvements)
6. companies employ grad-student descent (hire a bunch of PhDs and task them with improving MLPerf results)
A researcher might notice that the True and False cases are different enough to warrant their own algorithms. Say they now fit a lower dimensional line to the True and False cases.
They've just performed gradient descent but the grad student was the gradient. It's a common joke in CS/ML departments.
(This is an obviously simple example, but the point is that hyperparameter tuning and algorithm selection _are a part of your algorithm_ and the data you're looking at while doing so is part of your training set)
No joke, always cheaper than massive optimization tasks. I’ve managed a team working on automl for years and I can say with certainty, the cheapest option is always hiring interns (grad or bachelor students) or a hackathon
Neural architecture search is finding more efficient architectural building blocks which translate to better quality models with fewer overall parameters.
There's a lot of work going on right now to improve transformer architectures in particular; I think once the dust settles they will be massively more efficient than the current models. Efficient transformers often also introduce an inductive bias, which ends up improving model quality. (similar to replacing fully connected networks with convolutional networks; more efficient, but also carrying an inductive bias that improves model quality.)
I wonder if it would be worth starting with low precision and then over time raise precision to zero in on a local min / max.
http://www.incompleteideas.net/IncIdeas/BitterLesson.html
"The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective, and by a large margin. The ultimate reason for this is Moore's law, or rather its generalization of continued exponentially falling cost per unit of computation."
And now we are beating out Moore's law!
http://www.transistorcount.com/
Slope seem a bit more gentle - but certainly not gone?
I've long predicted macrochips to solve this, brute force of huge quantities of silicon, but it hasn't really happened.
https://www.quora.com/If-the-Moores-law-is-slowing-down-can-...
AMD, Nvidia and intel are all hitting the 'heat wall' with their hardware just trying to cram more into smaller nodes and boost clocks. Nvidia's 4000 series cards are rumored to be 500W cards which is well into 'space heater' territory.
Not yet at least.