You are probably right that there is massive disparity between the two re: compute. With that said, it is conceivably possible to use a larger number of weaker chips rather than fewer bigger ones, and come out ahead per unit-time. Also, given their strategy seems to be trying to do as much on-device as possible, they are targeting smaller models, so they likely have less of a packing problem with smaller chips.
So in the goal of producing a strong model, it could go either way; especially for smaller models, data seems to be much more important than compute, as per the “Textbooks are all you need” paper (https://arxiv.org/abs/2306.11644), especially if you are not looking for a fully generalist model.
There is no connection whatsoever between hardware used to train models and performance of the models if hardware permits truly large models. Google's hardware does permit it.
As I understand, many people in Apple’s ML org come from Google. So there’s a bias to use Google software and hardware. Moreover, there’s probably a deal that Apple extracted to do training on Google hardware.
The important thing for Apple is the endpoint inference which will be done on Apple Silicon.
6 comments
[ 3.0 ms ] story [ 24.5 ms ] threadSo in the goal of producing a strong model, it could go either way; especially for smaller models, data seems to be much more important than compute, as per the “Textbooks are all you need” paper (https://arxiv.org/abs/2306.11644), especially if you are not looking for a fully generalist model.
The important thing for Apple is the endpoint inference which will be done on Apple Silicon.
Apple intends to offload most complex requests to OpenAI, which does not use any Apple hardware whatsoever in the inferencing stage.