It's Known as 'The List'–and It's a Secret File of AI Geniuses (wsj.com)

This feels like one of those bell-curve memes with the idiot and savant on the opposite sides saying “building LLMs is magical work undertaken by wizards”, where in the middle someone is saying “it’s just high school math lol”

I'm curious about that syllabus.

There is obviously a difference between understanding what smarter people have already done, and actually innovating and building on what they have done. The latter requires a much deeper understanding.

Gradient descent is to AI, what a loop is to understand programming. However, understanding a loop doesn't mean you can program a full video game from scratch. Organizing hundreds of layers in an efficient way is pretty complex, even if the work today has been simplified thanks to PyTorch or Tensorflow, it remains pretty complicated. You have to understand how to organize your data, how to size your batches, how to make your code resilient enough to survive GPU cards crashing. Train a model over hundreds of GPU is really really complicated. New algorithms are proposed all the time, because we have no idea how to handle these interconnected layers in an efficient way, but with cumbersome heuristics. However, salary inflation is never a good thing, because it will create a gap between decent engineers and other people pronounced geniuses. The AI teams will suffer from these decisions... Badly. It will be like these samouraïs who would kill peasants after a battle to increase their head count, because this was how people were rewarded after a battle. Some of these people, in order to justify their salaries, will feel pressure to poach other people's ideas...

[dead]

[dead]