They want to join all models into one, but I fear it's going to be too large/expensive to use. GPT-3 size models are not economically viable for general use.
This is certainly the way things are headed, and they're smart to clarify that vision, but I feel like XLA needs better support for sparsity in order for this to really take off. And I reckon Google knows it, too. Everybody talks a big game about more efficient models but the tooling around sparse NNs is way behind that for dense ones
As others have said, this is the way ML is headed and it has been clear for the last couple of years.
Assuming a multi-task and multi-modal architecture is achievable, won't we reach a point in the future where preprocessing data is just as difficult as the machine learning component? Today, the input to ML models is almost always carefully curated and processed.
You can make sensor type components in NNs which reshape arbitrary arrays to a fixed size and shape (or a ragged shape which is consistent in 1 dimension and works with sequence models)
It seems to me that there are two ways to learn well. One is to have a carefully curated curriculum and the other is to have enough experience to parse the world by yourself. In humans we might describe these as knowledge/education and wisdom. I see data preparation as improved knowledge transfer and more training data as the path to wisdom.
Wisdom is usually heavily discounted by smart young people so I expect engineers will double down on better data prep than better data acquisition. Also which looks better on a resume?
6 comments
[ 5.1 ms ] story [ 22.9 ms ] threadAssuming a multi-task and multi-modal architecture is achievable, won't we reach a point in the future where preprocessing data is just as difficult as the machine learning component? Today, the input to ML models is almost always carefully curated and processed.
Wisdom is usually heavily discounted by smart young people so I expect engineers will double down on better data prep than better data acquisition. Also which looks better on a resume?