HN Explorer

Show HN: Microgpt is a GPT you can visualize in the browser (microgpt.boratto.ca)

282 points by b44 ↗ HN
very much inspired by karpathy's microgpt of the same name. it's (by default) a 4000 param GPT/LLM/NN that learns to generate names. this is sorta an educational tool in that you can visualize the activations as they pass through the network, and click on things to get an explanation of them.

17 comments

[ 5.1 ms ] story [ 38.8 ms ] thread
kfsone
Minor nit: In familiarity, you gloss over the fact that it's character rather than token based which might be worth a shout out:

"Microgpt's larger cousins using building blocks called tokens representing one or more letters. That's hard to reason about, but essential for building sentences and conversations.

"So we'll just deal with spelling names using the English alphabet. That gives us 26 tokens, one for each letter."

msla
About how many training steps are required to get good output?
alansaber
Depends on the model size, batch size, input sequence length, ... etc. With a small model like this you'll never get a 'good' output but you can maximise its potential.
krackers
There used to be this page that showed the activations/residual stream from gpt-2 visualized as a black-white image. I remember it being neat how you could slowly see order forming from seemingly random activations as it progressed through the layers.

Can't find it now though (maybe the link rotted?), anyone happen to know what that was?

stevage
I'd love to understand how LLMs work, but this site assumed a bit too much knowledge for me to get much from it. Looks cool though.
GaggiX
Wtok and Wpos should be 26-dim along one of the axis but it shows a 16x16 matrix be default, fc1 instead 16x64 with the default settings (not 16x16).
kengoa
Amazing work! Reminded me of LLM Visualization (https://bbycroft.net/llm) except this is a lot easier to wrap my head around and that I can actually run the training loops, which makes sense given the simplicity of the original microgpt.

To give a sense of what the loss value means, maybe you can add a small explainer section as a question and add this explanation from Karpathy’s blog:

> Over 1,000 steps the loss decreases from around 3.3 (random guessing among 27 tokens: −log(1/27)≈3.3) down to around 2.37.

to reiterate that the model is being trained to predict the next token out of 27 possible tokens and is now doing better than the baseline of random guess.

RugnirViking
I was a little confused by "see, its much better" when the output is stuff like isovrak and kucey. What is it supposed to be generating?
ramon156
My Android phone was not a fan of this site, but on my desktop it works great! Cool stuff
youio
really well done
keepamovin
I can't help but think there has to be a cheaper way to LLM.
prakashdep
It reminds me the anything+GPT era of 2022-2024
armcat
Really nicely presented, well done!