This would have been bigger news except for Gemini 1.5, Sora, and the Magic investment all happening at the same time. Gemini can do needle in a haystack reliably in the 3hrs of video they tested against.
Magic.dev got ~100MM investment because the ex CEO of github was super impressed by their progress towards super long context, code optimized LLMs to replace developers...
It started off as an improvised test for longer context windows I think Anthropic/Claud did some early on but Greg Kamradt seems to have got it established as a benchmark [1]. Basically they hide a phrase (in this Gemini case it was text flashed on the screen) at various points in a document and then compare performance across LLMs by retrieval accuracy as the size of the document increases.
To expand on this, it was so perfect at the haystack benchmark that they invented a new one, basically a hundred needle in the haystack benchmark. they had to invent new benchmarks for the model because it blew all existing ones out of the water. [0]
Look at how Alpha Go started with human data, and then they found a way to train it without that. I've been wondering if it might be possible to do a similar thing with LLMs by grounding them on real world video by having them predict what happens in the video. I suppose you'd still need some minimal language ability to bootstrap it from, but imagine it learning the laws of physics and mathematics from the ground up.
7 comments
[ 2.9 ms ] story [ 27.3 ms ] thread[1] https://x.com/GregKamradt/status/1758149501188210932
[0]https://youtu.be/Cs6pe8o7XY8