HN Explorer

11 comments

[ 3.6 ms ] story [ 47.3 ms ] thread
gavindean90
Seems like this is a modded NanoGPT not the original.
munchler
Yes. It’s literally called “Modded-NanoGPT”.
Scene_Cast2
I wonder how much improvement is owed to which changes. I've also never heard of "Muon - Momentum Orthogonalized by Newton-schulz" being used.

EDIT: there's a bit more info on his twitter - https://x.com/kellerjordan0

It looks like he created this optimizer. Works on 2D matrices only.

molticrystal
Just needs a Zero To Hero series episode offering line by line commentary to follow along on why each choice was made over alternatives.
m3kw9
So it compresses info better.
pyinstallwoes
That is literally intelligence.
parineum
It's not.
nikonyrh
I suppose you aren't a fan of the https://en.wikipedia.org/wiki/Hutter_Prize .

> The goal of the Hutter Prize is to encourage research in artificial intelligence (AI). The organizers believe that text compression and AI are equivalent problems.

parineum
I believe that they believe that and that it _could_ be true. That's far from declaratively stating that they are the same thing, as if there was some sort of evidence and consensus of such a claim.
byyoung3
do you have a baseline of the regular implementation with 3x learning rate?