The overhead really isn't that bad is it? Since the the python code is mostly about saying multiply matrix A with matrix B, and then that actual computation is done by optimized low level code.
I suspect that this has a high chance of running afoul of Ahmdal’s Law. Even if you can parallelise the bulk of the computation, the serial parts remain single-threaded and start to dominate the total runtime.
What I've seen is issues with the implementation of those libraries in a project.
I don't remember exactly, but I was playing with someone's wrapper for some kind of machine learning snake game and it was taking way longer than it should have on back of the napkin math.
The issue was using either a dict or a list in a hot loop and changing it to the other sped it up like 1000x.
So it's easy to think "yeah this library is optimized" but then you build something on top of it that is not obviously going to slow it down.
But then again if your program have places where choosing the right Python primitive is important for performance, then using python is affecting performance here since even the best algorithm in Python would be slower than the equivalent C.
Most of the time it doesn't matter because there's nothing hoy on the Python side, but if there is, then Python is going to be slowing your stuff down.
Kinda. I guess my native tongue is C/C++ and I wouldn't expect such a huge performance difference when using an array vs a linked list or something.
It's not like I had millions of items in that structure either, it was like 100. I think it contained the batch training data from each round. I tried to find the project but couldn't.
I was just shocked that there was such a huge difference between primitive data structures. In that situation, I wouldn't have guessed it would make a difference.
It depends on how you define overhead. Runtime overhead and memory usage is absolutely marginal, and the tightest, most perfect implementation will have trouble beating it.
Instead people are trying to optimize install size of dependencies, which while maybe a fun hacking project...who really cares?
Taking a peek inside the package it seems to mostly be the libraries - CuFFT alone is about 350MB for example, twice over for the debug and release versions. I'm guessing those are probably fat binaries pre-compiled for every generation of Nvidia hardware rather than just the PTX bytecode, which would help to speed up fresh builds, at the expense of being huge.
I don't think it's about the byte size, but the inherent complexity of the implementation. 1000 lines of C code is extremely simple by any standard. Whereas a sundry collection of Python and PyTorch libraries is anything but.
The most interesting one IMO is OLMo from AI2, which is truly open. You can read their blog post about it (https://blog.allenai.org/hello-olmo-a-truly-open-llm-43f7e73...) but basically it is open everything - they released everything you need to reproduce their weights (training data, training code, evaluation code, and weights) with a friendly (Apache) license.
But in this particular application, if you look e.g. through the training code, there's very little going on in terms of resource management. A handful of mallocs and some file handling.
That doesn't mean that you can have bugs even in a small number of instances, but if automatic resource management is your main argument, perhaps llm.c isn't the most prominent case that could benefit from it.
RAII sucks the way C++ does it. Magical background BS with massive unintended complexity consequences requiring obtuse intricate crap like the copy-and-swap idiom, a mudball of pointer and reference types, etc. They should have added a defer/scope-exit statement and been done with it.
OT but question from someone curious..... is Cuda still entrenched as the only option for doing AI or is there growing support for AMD/Intel/Other ways of doing AI?
He loudly gave up on AMD after they did not fix a blocker he had for 5+ months and gave him the runaround the entire time when he asked for the code to fix it himself. He is still shipping the AMD tinybox with huge warning labels.
Didn't they recently announce that everything was open sourced? Would be cool if he took another look at it once all of the souce code is available (if not already).
Randomly stumbled over this[1] post with another fed up open source contributor, due to several serious issues with AMDs GPU drivers and firmware that remain unresolved for years. It also references the geohot decision you mention.
Some quotes:
I find it incredible that these companies that have large support contracts with you and have invested hundreds of thousands of dollars into your products, have been forced to turn to me, a mostly unknown self-employed hacker with very limited resources to try to work around these bugs (design faults?) in your hardware.
In the VFIO space we no longer recommend AMD GPUs at all, in every instance where people ask for which GPU to use for their new build, the advise is to use NVidia.
I'm not sure if he's "attempting to solve it" so much as he's looking for yet another way to keep himself famous.
The guy did one good jailbreak for the iPhone, and as near as I can tell, the rest of his work has been a lot of boasting, half-assed hyped-up implementations (e.g: his self-driving car), and trying to befriend other powerful people in tech (see: his promise to single-handedly fix Musk's Twitter). He might be a smart dude, but he vastly overrates his own accomplishments, and doesn't finish near anything he starts.
You can run inference today on pretty much any card.
Download Ollama on a modern MacBook and can run 13B and even higher (if your RAM allows) at fast speeds. People run smaller models locally on their phones
Google has trained their latest models on their own TPUs... not using Nvidia to my knowledge.
So, no, there are alternatives. CUDA has the largest mindshare on the training side though.
there are obv alternatives from both intel and amd, performant blas/dnn packages, but small teams don’t use them bc cuda is easier to use and has more support, and larger teams don’t use them bc they have deals w/ nvidia or not enough GPUs are available or they’re after the absolute best performance (which is still nvidia) or bc of other stuff like unstable drivers or smth
Yes. CUDA is still entrenched due to its ease of use when compared to specialised AI hardware SDKs.
OpenMP is still a thing in 2024, but I presume that is not the kind of scale you are asking about.
If you have MI accelerators, you'll be using ROCm anyway (although AMD contracted Andrzej Janik in 2022 to make ZLUDA run on AMD GPUs. I have no idea what the practical applications of it are at the moment.)
The only other serious challenger (apart from the existing GPU manufactures like AMD) who is trying to give Nvidia a run for its money is Tenstorrent and their TT-Buda software kit.
When Lex recently talked to Andre, Andre said that he gets positively obsessed with a problem and says "this must exist". I imagine this must be one of those outputs.
I've seen his nano GPT implemented using JAX, now we have C/CUDA. I'd love to see if nano GPT could be doable in Mojo. I took a stab at a Mojo conversion of his Wavenet project (Andrej's zero to hero course) and I gotta say... python has so many nice features lol. Stating the obvious I know but what you see done in 6 lines of python takes so much more work in other languages.
For a prior generation of karpathy-splaining this is this Nim port: https://github.com/Vindaar/llama2nim - maybe of interest if you are interested in Mojo.
If you want CUDA up front go write PyTorch. No one is stopping you. Modular’s goal was to leverage MLIR first and bring GPUs in later. They’re barely a year old company.
It would be great if someone created a tutorial around this explaining exactly how it works and how to do a test training run. I’m aware it’s not feasible to train a “real” model on personal hardware but it would be nice to have a practical learning experience. I’m not sure if there are good alternatives for that.
The author has a whole series where he does exactly that. YouTube videos, code examples, documentation, everything. Explains the math, explains how to code it, explains the architecture. Everything.
> direct CUDA implementation, which will be significantly faster and probably come close to PyTorch.
It almost hurts, to read that PyTorch is faster.
But then again, with these GPU-RAM-prices, let's see how it speeds up the CPU.
We really need SO-DIMM slots on the RTX series (or AMD/Intel equivalent) so that we can expand the RAM as we need it to. Is there a technical problem to it?
> We really need SO-DIMM slots on the RTX series (or AMD/Intel equivalent) so that we can expand the RAM as we need it to. Is there a technical problem to it?
I imagine it would incur a non trivial latency and cost penalty. The memory modules are placed pretty close to the compute die right now. Cooling would also have to change (the memory modules produce a lot of heat).
But there is also no reason for any of the GPU manufacturers to do this. A skew with twice as much memory can go for a lot more than the difference in memory cost alone
And especially doing "interesting" combinations of gpu and memory.
Like lower end gpu with 16 GB of VRAM, but offering just 8 / 12 GB of VRAM in the middle class and then again 16 GB in the upper class of gpu selection.
I don't disagree but (I know nothing about this btw...) would it not benefit in terms of, say, a L3 cache kind of thing?
Imagine you could stick 2 x 64GB DDR5 DIMMS on the GPU in sockets, would that not be faster to access than the motherboard DIMMS? It won't be as fast as on-die memory of course but could it not act like a sort of halfway house?
Memory speed is more or less directly proportional to how close the memory is to the processor, with the fastest memory being literally inside the processor (SRAM cache), followed by memory on the same package as the processor (HBM GPUs, Apple M-series), followed by soldered down discrete memory chips (regular GPUs, games consoles), followed by socketed DIMMs in distant last place. There's not really any getting around it, the bandwidth that GPUs crave just isn't compatible with modularity.
Even CPUs are starting to move their memory closer to the core in the name of performance, as mentioned Apple is already doing it, Intel is making Xeons with on-chip memory now, and they have a version aimed at consumers on their roadmap.
FYI, most discrete GPUs with discrete memory packages soldered to the board near the GPU are running at substantially higher memory frequencies than the on-package DRAM in Apple's chips. But running GDDR at those speeds costs a lot of power.
I watched a presentation on this today. The presenter focused on the soldering and proximity as well. Is this really the only difference or is this transistor based memory (like L1, L2, etc.)? I get the proximity factor of course (1ft / ns EE rule of thumb). In any case, soldering and proximity don't seem like breakthrough innovations (but maybe I am wrong).
Gpu ram is typically gddr6 or gddr6x which is a different standard to the chips used in ddr5 for example. GPUs have terrible latency to ram, but enormous throughput, and I assume the chips are internally optimized for that. Many aspects of a design change when you choose different latency or clockspeed targets translating into different power / area calculations.
For data rates, as in bandwidth per IO pin, distance is really only a secondary factor. HBM memory, for example, runs at substantially lower data rates than GDDR, yet it sits right next to the GPU die compared to centimeters for the GDDR. And high-speed serial links run at speeds that are an order of magnitude higher than even the internal register files of a CPU.
That's true it's got an impact, but I think there's still space available for "slightly slower with 2x memory" models. For many local uses, new cards are way past the "fast enough" line, but having 64gb on them would be really beneficial.
It's love to see some experiments / different SKUs in this area, given people are already diy-ing extra memory on NVIDIA. (https://hackaday.com/2021/01/29/add-an-extra-8gb-of-vram-to-... there were stable experiments later on, but I don't have a link now)
Graphics card manufacturers believe that selling high-memory consumer graphics cards will affect the market for commercial computing cards, so they will not do so, that's all.
Problem is, making a board design using an existing GPU chip and sticking more RAM into it is (relatively) simple but of course none of the GPU chip makers would allow partners to do that. Making your own GPU chip that’s competitive with Nvidia or AMD’s current offerings is a massive undertaking and pretty much impossible for a newcomer.
Just look at how much trouble Intel has had breaking into the discrete GPU market or even just how hard it’s been for AMD to compete with Nvidia even with decades of experience in the market.
And if some newcomer could make a competitive GPU with large memory capacity they’d be crazy not to sell it at datacenter prices, maybe just undercutting the others but a few grand but still way more expensive than any consumer GPU you can buy today, even a 4090.
Check out PCB back drilling. It's a process where you remove a few hundred microns from the vias that are used to connect GDDR RAMs to the GPUs, to avoid reflections due to the impedance mismatch that's caused by the stub.
When you have a pulse coded signal traveling at close to 10GHz, everything becomes an antenna. The technical problem is that you can't do this with a flimsy connector like the ones used for DIMMs. The reason GDDR can have a bandwidth per pin that is 4 times higher than regular DDR is because they are soldered down on the PCB.
If I was starting from scratch, what resources should I start with to build up an understanding of what this code does and how to read it? It's quite dense and my knowledge of LLMs is quite minimal. Are these terse variable names standard in LLM-land?
“What resources would I need” -> you’re literally commenting on a teachers content. Karpathy (the author) has a very informative YouTube channel where he goes step by step through everything. He has a ton of repos and tutorials. Dig a little.
You’re not supposed to know that. You asked a question, and this is you being told the answer.
It’s very convenient that the author of the post is quite literally the world’s most prolific teacher on this topic. Makes it easy to find Karpathy. You shouldn’t be expected to otherwise know that (or else why ask if you knew).
> I didn't realize variables had to be so short in C. Glad I write C++ professionally where they've added support for longer variable names.
This feels like a joke but old C compilers did have variable length limits. This is part of why C historically had shorter variables than other more modern languages.
Sorry if it came off rude, the internet is hard to communicate over.
As siblings have said, his video series are quite good. But if you're just looking at this repo only, you probably want to look at the python reference implementation. (The C is designed to exactly replicate its functionality.)
Question, apologize if slightly off-topic, it's something I'd like to use this project for: Is there an example of how to train GPT-2 on time series, in particular with covariates?
As my understanding of LLM goes at a basic level it's predicting the next token from previous tokens, which sounds directionally similar to time series (perhaps letting aside periodicity).
These kind of papers often talk the world, but often lack a proper baseline model. They only compare against very simple (naive forecast), or non tuned models. In my experience a gradient boosting model will probably solve 95% of your forecasting problems, and trying to get fancy with a transformer (or even just a simple neural net) is more trouble then it is worth.
On a related note, IMO it would be pretty cool if we could get an LLM implementation that provides and RCI interface like all the old computational codes used to.
That's a good question. Unfortunately I think you're asking to compute the Kolmogorov complexity of every interesting concept we have that doesn't yet have an implementation less than n=1000 lines, which is equivalent to the halting problem (modulo unbounded memory).
If you could exhaustively list all the interesting algorithms (hard but feasible) you could potentially prove a lower bound for each one's complexity by writing a shorter than n implementation (hard, probably infeasiblel and show positively that GP's prop isn't true. On the other hand showing that it was true would require either some very clever proof which can't apply to all programs, but somehow only these interesting ones (very likely impossible) or enumerate all C^n programs where C is the number of possible lines (something like 64^80) and show that none of them implements at least one of the interesting algorithms (absurdly impossible).
You are right but I think that there's a more interesting question: do humans stumble upon those large interesting/great algorithms in practice?
The key point here is that we are looking at algorithms already discovered in human history rather than enumerating all possible interesting algorithms. Of course there is an interesting algorithm that is very large, but humans don't discover them in practice. If you look up a list of greatest algorithms in history, they will be rather small in length. Many of them can be sketched in a whiteboard
I think that what is happening here is that our minds just can't hold billions of concepts at once. So if you have an algorithm with billions of things, it was most likely produced by a machine. Handcrafted things, on the other hand, are smaller in comparison
Another thing is that our minds like conceptual simplicity and view simplicity as a kind of beauty. So if we have a great algorithm but it is too large, we look for ways to express them in succinct ways (the right abstractions can help with that, and also help with understanding the algorithm better). We end up succeeding because the algorithms themselves had low Kolmogorov complexity (and thus, if they are too large they probably can be further compressed)
Most modern A/V codecs won't fit in that limit by several orders of magnitude.
Even standard-compliant JPEG decoder would be hard to squeeze without some serious codegolfing. Discarding some barely used features gets you close to that limit, though [1].
A JPEG decoder or TCP stack are very clearly not individual concepts though. There's obviously some subjectivity as to what constitutes a single "concept" or "algorithm", but I'm not sure either of those two examples are in a gray area.
A single concept might be implementing just ARP or a discrete cosine transform. If you wanted to do a full TCP stack or JPEG decoder, that would make a lot more sense after building their internal components one by one.
This gave me an interesting thought... How many algorithms can be expressed in 1000 lines (say 5 - 80 chars per line) of correct C code. Obviously the number is huge if just look at char combinations (62^80E3 maybe) but if reduced to compiling / correct and unique (i.e. not including the same algorithm with differing variable names for example) the number would obviously be much less. If in the range of < 1E12 it could be practical to have AI further filter by recognizing useful algorithms.
Who knows, there might be a warp drive in there somewhere!
Nah, recent research shows that we don't need mountains of data to have very competent models. I don't have any links on hand but quick web browsing should return some relevant information
I think 6 months ago there was a old video from theate 80s/ beginning 90s at a US university where a professor experimented with a llm from scratch. If I find that one I will post it.
But definitely they knew the principles back then.
Wow, and this is done after a recent trip to Bhutan to clear his head! I follow karpathy on twitter and he posted that 2 weeks without constantly looking and checking his phone kind of turns off the constantly on radio in his head.
Candle dev here, we also support training/backdrop! We certainly focus on optimizing inference performance but hopefully that should improve the training efficiency too.
Inference means using the neural net, as opposed to training it.
During inference you feed an input into the NN and it passes through it in "forwards" direction (i.e. from input to output), being modified according to the "weights" that were learnt during training, to derive the output.
During training, each training sample is first fed forwards through the NN, the same way as for inference, but then the output of the model (which at the beginning of training will be random/wrong) is compared to the correct/desired output for that training sample, and a corresponding error value will then be fed backwards (from output to input) through the NN according to the "backpropagation" mechanism to update the weights.
Training is a lot more involved than inference since it involves this backpropagation step.
Or rather GLSL... The C++ code looks like it's mostly just scaffolding to kick off the actually important GPU work, and for that it's a surprising amount of code. Quite typical both for Vulkan and C++ though ;)
188 comments
[ 3.2 ms ] story [ 242 ms ] threadWhat I've seen is issues with the implementation of those libraries in a project.
I don't remember exactly, but I was playing with someone's wrapper for some kind of machine learning snake game and it was taking way longer than it should have on back of the napkin math.
The issue was using either a dict or a list in a hot loop and changing it to the other sped it up like 1000x.
So it's easy to think "yeah this library is optimized" but then you build something on top of it that is not obviously going to slow it down.
But, that's the Python tradeoff.
Most of the time it doesn't matter because there's nothing hoy on the Python side, but if there is, then Python is going to be slowing your stuff down.
The programmer using the wrong data structure is not a problem with the language.
Go kind of cheats and has maps play double duty as sets.
It's not like I had millions of items in that structure either, it was like 100. I think it contained the batch training data from each round. I tried to find the project but couldn't.
I was just shocked that there was such a huge difference between primitive data structures. In that situation, I wouldn't have guessed it would make a difference.
Instead people are trying to optimize install size of dependencies, which while maybe a fun hacking project...who really cares?
Go to try for self
Step 1 download 2.4GB of CUDA
As others have said, George Hotz is doing his best in reverse-engineering and skipping layers.
https://carlpearson.net/post/20231023-cuda-releases/
Although I wonder if it would work well with GCC PTX OMP offloading.
"Currently, I am working on [...] direct CUDA implementation, which will be significantly faster and probably come close to PyTorch."
The most interesting one IMO is OLMo from AI2, which is truly open. You can read their blog post about it (https://blog.allenai.org/hello-olmo-a-truly-open-llm-43f7e73...) but basically it is open everything - they released everything you need to reproduce their weights (training data, training code, evaluation code, and weights) with a friendly (Apache) license.
[1] https://github.com/huggingface/candle
That doesn't mean that you can have bugs even in a small number of instances, but if automatic resource management is your main argument, perhaps llm.c isn't the most prominent case that could benefit from it.
[1@2024-04-06] https://www.youtube.com/watch?v=j7MRj4N2Cyk&t=429s
[Twitch] https://twitch.tv/georgehotz
Some quotes:
I find it incredible that these companies that have large support contracts with you and have invested hundreds of thousands of dollars into your products, have been forced to turn to me, a mostly unknown self-employed hacker with very limited resources to try to work around these bugs (design faults?) in your hardware.
In the VFIO space we no longer recommend AMD GPUs at all, in every instance where people ask for which GPU to use for their new build, the advise is to use NVidia.
[1]: https://www.reddit.com/r/Amd/comments/1bsjm5a/letter_to_amd_...
The guy did one good jailbreak for the iPhone, and as near as I can tell, the rest of his work has been a lot of boasting, half-assed hyped-up implementations (e.g: his self-driving car), and trying to befriend other powerful people in tech (see: his promise to single-handedly fix Musk's Twitter). He might be a smart dude, but he vastly overrates his own accomplishments, and doesn't finish near anything he starts.
Download Ollama on a modern MacBook and can run 13B and even higher (if your RAM allows) at fast speeds. People run smaller models locally on their phones
Google has trained their latest models on their own TPUs... not using Nvidia to my knowledge.
So, no, there are alternatives. CUDA has the largest mindshare on the training side though.
[1] https://github.com/hidet-org/hidet
OpenMP is still a thing in 2024, but I presume that is not the kind of scale you are asking about.
If you have MI accelerators, you'll be using ROCm anyway (although AMD contracted Andrzej Janik in 2022 to make ZLUDA run on AMD GPUs. I have no idea what the practical applications of it are at the moment.)
The only other serious challenger (apart from the existing GPU manufactures like AMD) who is trying to give Nvidia a run for its money is Tenstorrent and their TT-Buda software kit.
I love the attitude of considering 0.x languages production ready for all imaginable kinds of workloads.
[1] https://github.com/karpathy/nanoGPT
But if you don't have the background, I'd recommend my YouTube videos, see the Zero To Hero playlist: https://www.youtube.com/watch?v=VMj-3S1tku0&list=PLAqhIrjkxb...
It almost hurts, to read that PyTorch is faster.
But then again, with these GPU-RAM-prices, let's see how it speeds up the CPU.
We really need SO-DIMM slots on the RTX series (or AMD/Intel equivalent) so that we can expand the RAM as we need it to. Is there a technical problem to it?
I imagine it would incur a non trivial latency and cost penalty. The memory modules are placed pretty close to the compute die right now. Cooling would also have to change (the memory modules produce a lot of heat).
But there is also no reason for any of the GPU manufacturers to do this. A skew with twice as much memory can go for a lot more than the difference in memory cost alone
Like lower end gpu with 16 GB of VRAM, but offering just 8 / 12 GB of VRAM in the middle class and then again 16 GB in the upper class of gpu selection.
Imagine you could stick 2 x 64GB DDR5 DIMMS on the GPU in sockets, would that not be faster to access than the motherboard DIMMS? It won't be as fast as on-die memory of course but could it not act like a sort of halfway house?
Even CPUs are starting to move their memory closer to the core in the name of performance, as mentioned Apple is already doing it, Intel is making Xeons with on-chip memory now, and they have a version aimed at consumers on their roadmap.
It's love to see some experiments / different SKUs in this area, given people are already diy-ing extra memory on NVIDIA. (https://hackaday.com/2021/01/29/add-an-extra-8gb-of-vram-to-... there were stable experiments later on, but I don't have a link now)
Just look at how much trouble Intel has had breaking into the discrete GPU market or even just how hard it’s been for AMD to compete with Nvidia even with decades of experience in the market.
And if some newcomer could make a competitive GPU with large memory capacity they’d be crazy not to sell it at datacenter prices, maybe just undercutting the others but a few grand but still way more expensive than any consumer GPU you can buy today, even a 4090.
When you have a pulse coded signal traveling at close to 10GHz, everything becomes an antenna. The technical problem is that you can't do this with a flimsy connector like the ones used for DIMMs. The reason GDDR can have a bandwidth per pin that is 4 times higher than regular DDR is because they are soldered down on the PCB.
Why?
> And once this is a in a bit more stable state: videos on building this in more detail and from scratch.
Looking forward to watching the videos.
“What resources would I need” -> you’re literally commenting on a teachers content. Karpathy (the author) has a very informative YouTube channel where he goes step by step through everything. He has a ton of repos and tutorials. Dig a little.
If all else fails… Google it.
They're a math / toy code thing. Large C projects have long descriptive names just like other languages.
How am I supposed to know that?
> Karpathy (the author) has a very informative YouTube channel where he goes step by step through everything.
Or that, without knowing that he's a teacher?
> Terse variables are a C thing.
I didn't realize variables had to be so short in C. Glad I write C++ professionally where they've added support for longer variable names.
> If all else fails… Google it.
There's a lot of LLM garbage out there. I got an answer here in a few minutes pointing to Karpathy's course which seems very high quality.
Be kinder.
You’re not supposed to know that. You asked a question, and this is you being told the answer.
It’s very convenient that the author of the post is quite literally the world’s most prolific teacher on this topic. Makes it easy to find Karpathy. You shouldn’t be expected to otherwise know that (or else why ask if you knew).
> I didn't realize variables had to be so short in C. Glad I write C++ professionally where they've added support for longer variable names.
This feels like a joke but old C compilers did have variable length limits. This is part of why C historically had shorter variables than other more modern languages.
Sorry if it came off rude, the internet is hard to communicate over.
https://publications.gbdirect.co.uk/c_book/chapter2/keywords...
https://karpathy.ai/zero-to-hero.html
Do those, then come back to this and it will make more sense.
As my understanding of LLM goes at a basic level it's predicting the next token from previous tokens, which sounds directionally similar to time series (perhaps letting aside periodicity).
https://github.com/KimMeen/Time-LLM
These kind of papers often talk the world, but often lack a proper baseline model. They only compare against very simple (naive forecast), or non tuned models. In my experience a gradient boosting model will probably solve 95% of your forecasting problems, and trying to get fancy with a transformer (or even just a simple neural net) is more trouble then it is worth.
It was originally based off of Karpathy's llama2.c but I renamed it when I added support for other architectures.
Probable a coincidence :)
On a related note, IMO it would be pretty cool if we could get an LLM implementation that provides and RCI interface like all the old computational codes used to.
If you could exhaustively list all the interesting algorithms (hard but feasible) you could potentially prove a lower bound for each one's complexity by writing a shorter than n implementation (hard, probably infeasiblel and show positively that GP's prop isn't true. On the other hand showing that it was true would require either some very clever proof which can't apply to all programs, but somehow only these interesting ones (very likely impossible) or enumerate all C^n programs where C is the number of possible lines (something like 64^80) and show that none of them implements at least one of the interesting algorithms (absurdly impossible).
The key point here is that we are looking at algorithms already discovered in human history rather than enumerating all possible interesting algorithms. Of course there is an interesting algorithm that is very large, but humans don't discover them in practice. If you look up a list of greatest algorithms in history, they will be rather small in length. Many of them can be sketched in a whiteboard
I think that what is happening here is that our minds just can't hold billions of concepts at once. So if you have an algorithm with billions of things, it was most likely produced by a machine. Handcrafted things, on the other hand, are smaller in comparison
Another thing is that our minds like conceptual simplicity and view simplicity as a kind of beauty. So if we have a great algorithm but it is too large, we look for ways to express them in succinct ways (the right abstractions can help with that, and also help with understanding the algorithm better). We end up succeeding because the algorithms themselves had low Kolmogorov complexity (and thus, if they are too large they probably can be further compressed)
Even standard-compliant JPEG decoder would be hard to squeeze without some serious codegolfing. Discarding some barely used features gets you close to that limit, though [1].
Smallest popular TCP/IP stack [2] is ~20kLoC.
[1] https://github.com/richgel999/picojpeg
[2] https://savannah.nongnu.org/projects/lwip/
A single concept might be implementing just ARP or a discrete cosine transform. If you wanted to do a full TCP stack or JPEG decoder, that would make a lot more sense after building their internal components one by one.
Who knows, there might be a warp drive in there somewhere!
https://github.com/huggingface/candle
During inference you feed an input into the NN and it passes through it in "forwards" direction (i.e. from input to output), being modified according to the "weights" that were learnt during training, to derive the output.
During training, each training sample is first fed forwards through the NN, the same way as for inference, but then the output of the model (which at the beginning of training will be random/wrong) is compared to the correct/desired output for that training sample, and a corresponding error value will then be fed backwards (from output to input) through the NN according to the "backpropagation" mechanism to update the weights.
Training is a lot more involved than inference since it involves this backpropagation step.
[0] https://www.perceptivebits.com/building-gpt-from-scratch-in-...
Vision Model https://huggingface.co/spaces/radames/Candle-Moondream-2 Blip Image Captioning https://huggingface.co/spaces/radames/Candle-BLIP-Image-Capt... Microsoft Phi 2 https://huggingface.co/spaces/radames/Candle-phi1-phi2-wasm-...