He bases this decision on all of the experience he has amassed, as a 61 year old man in the tech industry. An actual 17 year old, with 17 years of experience, would not think like this, nor should they.
>Whoa. I’m 19 and I trained a 100M language model from scratch. Did a v2 now with a new SFT experiment to see if I can get better results on same size.
Yes. And they almost certainly have a better understanding of their own situation that him. This is not a dig at Paul Graham, the closer anyone is in age, the better they understand what they have to deal with. I'm roughly in the middle between Paul G and the 17 year old, and even though I'm really quite fascinated with zoomer culture and probably come more in touch with it than most (due to relatives in the age range etc.) I realize I have very little idea what it's like to grow up in the world they grow up in.
Naturally 17 year olds don't think long-term like this which is why PG's advice is so useful. It gives them a pathway to follow that they likely wouldn't have reasoned otherwise.
I told my much younger brother when he was 12 what programming was and it'd be a great career. He looked into it and within months was writing CLI games. Eventually releasing his own unity 3d game on steam as a teen.
Eventually he got into CS and did really well because none of it was scary and new. He parlayed that into role at Meta out of university.
My point being, 17 year olds have time to learn new skills and guidance can go a long way.
With the hindsight of experience, the remnants of my 18-year old energy go “woah, that’s cool!” at plenty of engineering feats… and my decades-older second brain goes “well d’oh, I could’ve just learned a trade to work on that!”
I think the last one was seeing a skilled electronics repairman do surgery on a CT machine controller.
I mean first that is already what plenty of 17yo are actually doing, because that is what they do at school or in parascholar activities. There are already countless of such tutorials where you can do that in an afternoon.
The pointless part though is precisely why Amazon and others are hunting for rare books, all the low hanging fruits have been picked already so just training a bigger model will simply mean burning more energy and money. Sure training a small one for the basic principle is a great pedagogical thing, training another one, medium, then maybe a large one, is also good in term of learning the process and architecture, but one should not expect it to be useful out of that context.
Pure players are precisely doing everything they can to corner the market by making their own scale unreachable by others. Smaller players with access to lesser infrastructure are thus betting on different market, e.g. embedded systems.
17yos should definitely build their (L)LMs from scratch and whatever bigger model they can train for free, or for cheap, but they should not expect that to bring them any riches.
There's this dilemma where in theory there's a ton of demand for engineers that can do real LLM machine-learning, but in practice there are very few available positions and entrepreneurship opportunities.
The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM company, and still then it's a struggle. Those few that do train, they spend most of their budget on compute and have relatively small teams.
Getting experience in this field requires having access to very expensive hardware to begin with. And the skills will be quite hard to convert into any real value for someone, leading to a decent income, unless you have a ton of funding from patient investors, or you have decent contacts in Bay Area networks to get hired at the right place.
With all due respect, paulg is in somewhat of a bubble, this is not congruent with the global situation.
That's not true, because everyone, everyone, everyone seems to want to do training. Which results in a 50 person company training, say, a voice model that then fails, because it's just not good enough.
In reality the problem is that it gets blasted out of the water by a much worse architecture trained on 10000x the infrastructure. And while I'm sure the freshly brought in ML student came up with a 10%, even 30% better architecture, it just doesn't matter. (and never mind that even OpenAI hasn't really solved a voice model yet. Try it. It can probably match 2026-quality call centers, but it's no substitute for an actually empowered human)
... and yet, if you look at what hyperscalers are getting paid for ... comfortably more than half the income is training. Which makes no sense on so many levels.
Everyone says they want to do training, because it's sexy and an easy way to justify raising mad funding rounds. Some manage, most don't.
I don't know where you are located, but in EU, in China, and yes even in Silicon Valley, the vast majority of companies do not do any real AI engineering. There's nothing wrong with it, it's just not a smart path for most purposes. You can do amazing things without training, and if you try to train, you cannot get anything amazing unless you burn millions.
Very few people can afford to play the long game and cross that dessert. And, sure, you will not get far without good engineering, but good engineering is definitely not sufficient and is not the primary bottleneck.
It took a long time to cross that desert, and no sane company would want to get stuck in a desert, unless it's specifically an R+D "desert crossing" company.
The big question is whether companies hold enough proprietary data to do useful things that for e.g. Anthropic, etc. can't easily replicate.
For some very niche cases I think this is probably the case but for the vast majority, the company's data isn't as useful as they think it is or anywhere near the size needed.
I think companies of all sizes will want their own models, or at least customised ones, for their own specific use cases or competition and security issues.
1. Both training and optimisation will get significantly cheaper and easier quickly.
2. Politics will probably get even more insane before a potential reprieve on the 20th of Jan 2029.
3. The big AI firms will become part of the surveillance capitalism network, if they're not already.
So I think for self-protection a lot of companies will be looking near to medium term AI independence.
The cost of training a model from scratch is going to be cost prohibitive for the vast majority of companies (even if renting the hardware needed for the 1-2 month training time). It's an interesting learning exercise, and some of the things learned can be applied to other parts of the process. There's also the issue of needing a huge amount of data needed to get decent weights.
Fine-tuning a model or LoRA based on the companies data set is more feasible but you're likely going to need several runs as you test/try out different base models, parameters, etc. This is why there are a lot of fine-tuned models on huggingface based on base or instruction-trained models from the larger AI companies that have released open weight models (Microsoft, Google, IBM, Mistral, DeepSeek, Qwen, etc.).
Training is limited on memory first (storing training data and weights) and computation second. Realistically you need to own or rent 2-8 H100/B100 devices or Google's TPUs.
The majority of workflows for a company providing AI capabilities are likely best solved by tailoring a system prompt for the chosen model, evaluating the prompt and model with tools like promptfoo, and then running it on a compute cloud provider (including AWS Bedrock). If the company is big/financially well off enough they could look at buying the hardware needed to run it on their own servers.
For other uses like agentic software development you'd need to spin up a suitable model on a compute cloud provider (or local hardware if the model is small enough) and then tell your IDE/editor to use that model. You would need some way of benchmarking and evaluating the models to see if they are capable of doing the tasks you need. -- There have been some tests done by people on YouTube that suggests that Qwen 3.8 27B is a decent model, but your needs may vary.
Most companies that build physical goods don't care for one second about their IT department other than how much money they can save per month, starting by outsourcing whole of it, thus they have little use for internal LLMs.
> +1, I have a friend, math PhD that's been working on ML research 5+ years in London yet he has not been able to find any position.
Maybe people simply don't want math PhDs but something else? Since 1-2 years ago I started doing consulting/freelancing in the ML space, but more on the infrastructure, deployments and similar stuff, as a general purpose developer, and I have a waiting list of clients interested in more work, some of them even trying to recruit me to work for them full-time as well. I'm based in continental Europe, fwiw.
This only proves the original point which is that there is not much demand for actual machine learning expertise because that is only carried out in a small number of places and what demands there is is for the more basic software carpentry like infrastructure and operations rather than the actual technology and Engineering side of things
I mean it makes sense right? Anthropic for instance has like, a couple hundred staff in London with plans to expand to somewhere just shy of a thousand. There are far far more ML/Maths/CS/Stats PhD's than there are openings. Especially in London there is no shortage of suitable candidates given Cambridge/Oxford/Imperial/UCL are surrounding it. 2% of the UK population has a PhD alone...
Except knowing how LLMs work don’t actually provide much understanding for using them. People don’t use LLMs the way we’ve built on most other tons or platforms. It’s more learning Unity hoping to be a better gamer.
Yes. It's like looking at the (Apollo) moon rocket launch and then suggesting teenagers should learn to build rockets in their garages for the coming space age.
It is viable as a toy project, but there are vanishingly few career opportunities.
But he’s not saying that it’s a career opportunity.
It seems to me like he’s saying that doing this thing would be 1) fun and 2) a great way to become employable in the future. I don’t believe he’s saying that this project would be some kind of job training exercise.
I "guess" you could put all your effort into moving up to the next level in computer science, or you could put the same progressive effort into reaching new levels in video games on the same hardware.
Alternatively you could even put all your effort into social activities and leave the technology to other people entirely.
It might even be possible to find a balance between things that are widely "understood" "socially", and those that are not ;)
There are more career opportunities building rockets than designing rockets. Lots of welders, machinists, electrical engineers etc build rockets, and those skills transfer.
The same is not so different for AI. A few people design novel AI, but there are a lot of people training AI (especially if you include fine tuning) and implementing AI, even as a hobby.
That seems like a perfectly good use of time for a kid in the 70’s! You’d learn a lot of engineering skills and demonstrate a tenaciousness that most people don’t have. I don’t think the goal is to predict what will be the important technology in 10 years. The goal is to challenge oneself with hard tasks and learn interesting stuff. A lot of the “AI” people today were compilers people yesterday
When I was 17 and still witnessing Apollo Moon launches, I wanted to build an AI that would handily outperform LLMs as we know them today.
But that was way back in the early 1970's and all I had to work with was a mainframe.
Well the mainframe itself wasn't bad, the real show-stopper was that I didn't own the computer outright, no strings attached, no debt, etc.
>I'd probably try to make an LLM that I could use on some specific problem.
I thought so too back then, still do so I guess this is one of those things that could stand the test of time. I always wanted to start with something a lot simpler than a Moon mission myself. At 17 I already had a significant breakthrough in the chem labs and it was from alternatives to a single processing step plus everything that descended from that, rather than trying to tackle a much more complex detailed multi-step synthesis. I was only 17 but I was not trying to be a slouch, I don't think pg was either at that age but his advice is not for just anybody. I couldn't have done it if I hadn't made major progress since being 16, and it really emphasized at the time how much maturity can make a difference. My imagination ran wild as I extrapolated :)
In a reply from LeCun to pg:
>>I'll figure out a set of methods and architectures beyond LLMs that can quickly learn to perform physical tasks as efficiently as humans and animals.
That last item is also what I would if I were 30, 40, 50, or 66 years old
I see no reason to stop at 66 either ;)
But I figured that people owning more computer power than I could ever afford were going to be doing something like this as soon as they could, without having to wait for something like an LLM to arrive before getting peoples' attention.
It did seem like things were going to take longer than you expect, so it's pretty good to have a lifetime of concentrating on the specialized natural science domain expertise, focused now for 50 full years on how it would combine if AI ever got good enough.
Both the natural science and the AI need to be a major cut above, I still see dramatic room for improvement in my own work. If I'm going to have to rely on "other peoples' AI" then that natural science component is going to have to pull a lot of weight to keep up with the kind of computers that only rich-as-hell high-rollers have access to.
I just finished fine tuning Gemma e2b for local code completion on my local machine.
This comment just reinforces what the post actually means. We need people that are LLM natives, computing solves itself with time and with scale adjustments
1. I very much still write code myself without an LLM when I need top quality
2. That's why I have an agentic agent as well installed, Qwen 27B, outrageously good, better than sonnet 2 years ago. And it's mine, I can give it confidential info to work with since I own the whole chain. See where I'm going with this?
I think his point is, if LLMs are the future (like computing is the future in the 90s), you should be an LLM-expert (equivalent of becoming a software developer).
I can see the point. It's unlikely that a 2.4T LLM will be integrated into, say, a pesticide drone. You'll still need some kind of LLMs to achieve maneuvers that "normal" programmings can't achieve.
But what if everything basically turn into that? Essentially, instead of build me a web app to solve X and do Y, build me an LLM to serve X and do Y. (unless the current LLMs are able to do it end-to-end but then they can hardly write coherent software/personal opinion).
i think ur wrong here … and if anything .. the people upvoting this are the ones living in a bubble.
over the past 6 months, i’ve been training models by taking academic papers in my domain (cybersecurity) whose authors haven’t released the implementation code, the training code, or both … and then trying to reconstruct them using Claude/Codex and then successfully training them ..
through this process, i’ve picked up ML frameworks (lightning ai) figured out how to prepare and stream super large audio datasets, and more importantly, successfully trained a model that i’m now using in my startup .. and the alpha is huge .. and i’m not even an engineer or developer in the traditional sense.
.. each training run currently uses an H100 and takes 3 to 4 days on DO … roughly $400 to $500 per run ..
More importantly going through this process has made me realise how insane this is going to become … people are going to be able to train models for an enormous range of specialised domains without necessarily being traditional ML engineers or working at one of the handful of companies doing serious training as u describe ..
as long as ur creative, have deep domain knowledge, and can figure out how to use the tools … the barrier to doing genuinely interesting ML work is collapsing
I think a lot of this is based on preconceptions. A lot of apps were made with Electron, because it was common wisdom that native is 'too hard'.
Now with LLMs, people write native apps in Rust, and I'd like to think some of them found that there isn't such a huge jump in difficulty they assumed there would be.
Oh yes I agree, LLMs are not that complex in principle, most engineers could build a toy version completely from scratch without too much difficulty.
But that’s the tip of the iceberg. If you have any ambitions of doing this professionally, it quickly becomes clear that all it’s all about knowing how to deal with problems that are only present at massive scale, when an LLM is actually L and becomes AI.
The mundane details about how to build a tiny autocomplete model and the maths behind it you can learn in a couple weeks easily. It’s not black magic, there are much harder areas of computers science.
It never was native "too hard" it was always "too expensive".
That's the same case finding companies that will actually pay for hand made LLM instead of using something from big providers will be hard because most companies won't be able to afford it.
Yeah if you find a company that will do that stuff directly, good for you, but you will have to be very lucky and you will have to compete with other people who followed PG advice.
So I would rather learn all there is about properly using LLMs and integrating them with existing systems, that will most likely by useful for 90% of companies out there.
Building business niche harnesses is in my opinion much better direction. Knowing what will work best in specific cases is it FTS or vector search, optimisation of usage, getting best results while using cheaper models, knowing how to use tools to run models on the servers, and all the tooling around that like various MCP or just tooling that will be provided to models.
That is what I am currently busy with and I already have customers for that knowledge.
FWIW, at least 20 Y Combinator startups have published ML research recently at ICLR, NeurIPS, ICML, and so on.
I think a lot of people assume that only the big AI labs can do cutting edge research, but there's a strong argument you can do it as part of little tech as well.
I agree with mostly all of this, but personally I wrote a toy LLM almost 5 years ago and while it never saw much use outside of boring my wife with a shitty command line demo with glee it did help me understand how they worked and how to apply them, played a lot with JAX and pytorch, ended up building a ghetto version of MCP and an LLM-Pool to proxy requests to my baby local models and so I didn't struggle to see the evolution of openrouter and MCP agentic workflows. The same way i'm really glad when I was younger I built a bad webserver by myself, a really painful SQLx type database, etc etc etc - none of these things led me to developing for Nginx or Oracle nor will knowing JAX get me a job at an AI research lab, but I do have a lot of depth in understanding how the technology works so that the flavors on top of them are easy to digest and make more use of immediately, and I think the same can be said for engineers coming into the field - if it's a spooky LLM box you aren't going to be squeezing the same amount of juice as the guy that knows how they work inside and out so having at least the understanding of a _babys first LLM_ is going to get you miles ahead of people who don't.
For anyone who wants to dork around there is https://github.com/rasbt/LLMs-from-scratch which is something amazing that I think anyone who wants to engineer things around LLMs should at least blast through and read.
And so I think the idea is more to understand tomorrow ... from first principles.
In the late 80's, as a teenager, I learned x86 assembly and C because that was the only way to squeeze out enough juice from my shitty CGA (and later VGA) card to programm the games/graphics that interested me.
I haven't written assembly in years.
But whatever I did in my career: it helped me and gave me an edge over my peers to have a foundation that is very close to the metal.
Current AI can automate significant amounts of grunt work in programming and math. It's good at running web searches and writing summaries. There are a few other niches where it is currently successful. But other than that, many corporate AI projects are spectacular failures.
So just given what we have in hand, assuming no further breakthroughs, then we're maybe looking at AI being somewhat bigger than the Internet. Which would make it a revolutionary technology, sure.
But to get from "a revolutionary technology" to "the substrate the future runs on", then you need to assume more breakthroughs: long-context operation over weeks or months, displacing human workers 100% instead of 75%, and the ability to directly economically compete with actual humans. And people are investing literal trillions of dollars to make that future come true, without really thinking through what truly competitive-with-human AI would actually mean. We might be looking at massive job loss, centralization of power, fully automated "companies" with no humans dominating markets, and other dystopian scenarios.
And in those worlds, it's unclear that being good at CUDA and matrix math will be all that helpful, careerwise. The AIs are already pretty good at that stuff. Data scientists get paid OK when they actually get hired, but it's not everything college students were promised in the 2010s, either.
We can't yet build a fully-general competitor for the human mind. But we're getting closer. And if we ever do build one, the consequences will be really weird in any number of ways. So I worry about visions of the future that assume AI keeps improving significantly, but that also assume it still somehow remains a "normal" technology that doesn't, for example, render most humans fundamentally uncompetitive.
> But to get from "a revolutionary technology" to "the substrate the future runs on", then you need to assume more breakthroughs: long-context operation over weeks or months, displacing human workers 100% instead of 75%, and the ability to directly economically compete with actual humans ... without really thinking through what truly competitive-with-human AI would actually mean. We might be looking at massive job loss, centralization of power, fully automated "companies" with no humans dominating markets, and other dystopian scenarios.
At 75% replacement of a worker we would already have huge job losses as each individual would be doing what several before did.
The only alleviation would be the creation of new equivalently paid jobs, which is no better than a hypothesis right now.
We don't know this. So many people are simply claiming this confidently, and a lot of them are betting their careers on it, but nobody has a crystal ball. Whenever someone tells you confidently, and without any doubt or qualifications, that something "is the future," be skeptical.
I remember when the Segway was definitely going to change urban planning worldwide.
"Necessity is the mother of invention" - limited hardware has always forced people to find cleverer ways of doing more with less. Current models are clearly nowhere near the efficiency limit (the brain does vastly more with far less power).
> Current models are clearly nowhere near the efficiency limit (the brain does vastly more with far less power).
I think this is disingenuous. One could say that drones are nowhere the efficiency limit either: a bee can fly for hours on the energy contained in just a few milligrams of honey, while our best battery-powered drones can't stay airborne for more than 30 minutes. But comparing energy efficiency of electric/mechanical devices to their biological counterparts is not an apples-to-apples comparison. There's a world of difference between the energy storage and delivery mechanisms.
And as many have pointed out already in the siblings, it's not just about the compute but the access to petabytes of training data.
A lot of startup companies are not training frontier models but help solve and optimize pain points of LLMs: cyber security, token usage, harnesses etc. These jobs don't require a PHD in machine learning but it does help if you understand LLMs at a deeper level.
The other thing to add as well is that the research teams who do the actual research work are relatively small and very specifically qualified which naturally keeps the barrier to entry high.
In that regard, it's not too different from mobile telephony. Mobile phones drove the electronics industry 20 years ago, but there is limited demand for people who really know how to build a phone (ie. build the hardware and write all the signal processing from scratch), as there aren't that many companies that do phones at the lowest level. A few of the engineers got rich (eg. Viterbi), but most 'just' made a good living. Most people who got rich off phones didn't do it by knowing how phones work.
Incidentally, the skills for the lowest levels of LLMs aren't that far removed from those needed for mobile telephony, in that both are based on maths, computation and information theory.
This is super interesting because I moved from mobile telephony into ML and data science, and information theory and working with data in statistically correct way was what helped me! This was 10 years ago though.
Good points, but I think we can expect the AI space to be more tumultuous.
What most people want from mobile technology is for it to work, not too expensively, and for it to get out of their way.
What most people want out of AI is for no leader to emerge and wield supremacy against the rest of us. People are afraid of it in ways they weren't afraid of mobile, so they're more willing to work together against whoever is in the lead.
Its more like an arms race and less like a utility. The disadvantage I face when my competition has better mobile coverage and bandwidth is minor. The disadvantage I face when my competition has better intelligence on tap is much more significant.
>* What most people want out of AI is for no leader to emerge and wield supremacy against the rest of us. People are afraid of it in ways they weren't afraid of mobile, so they're more willing to work together against whoever is in the lead.*
No.
That’s what people like us on HN want. The people out in “Greater Userland” just want the black box to answer their questions. They could care less who is behind it. They don’t yet attach their black box to Amazon or Microsoft etc. And most won’t care enough to be inconvenienced even when they do make the connection. (As your competition argument implies.)
Heck, a lot haven’t even made the connection between the black box that gives them answers and data centers. They think, “ ChatGPT good” and at the same time think “data centers bad”.
I was there at the start of the smartphone boom. I built a demo Android device that was capable of telephony/data, 3D rendering, etc. all before Google open-sourced the OS, for a SoC vendor that wasn't in Google's inner circle. Yet the industry was not interested in my junior profile during the subprime crisis.
I wonder if the modern incarnation of Android flagships would have taken off at all if not for Apple. When Android first showed up, the dominant "smartphones" were running Symbian and they had a very different view of the world when it comes to UX and hardware.
But it was generally seen as a gimmick instead of desired before Apple made it look good. Even when the iPhone came out, one of the jokes was how the grid of icons looks like how a Windows user's desktop would look like when they didn't understand the filesystem.
We don't really have the demand for as many telecom companies as actually exist in the world. There's a reason we just have one Whatsapp and one Instagram, not three or four almost-but-not-quite clones in every single country that mostly differ in branding. The reason for the current situation has mostly to do with regulation and traditional, enterprise, "obviously every country needs a separate local branch, because that's what mcDonalds does" thinking. Technology has very little to do with it.
This is why the telecom world now consist of equipment manufacturers, who do most of the hard tech stuff, and actual telecom companies, who operate the equipment, rig towers in their local country, and maybe write some glue code to integrate a core from vendor A, a billing system from vendor B and a CRM / corporate invoicing system from government-approved local vendor C.
Banking also works similarly, though modern Neobanks / Fintechs and bank consolidation are slowly dissolving the concept of national bank branches.
> With all due respect, paulg is in somewhat of a bubble, this is not congruent with the global situation.
Paul, I think, is talking about achieving outsized outcomes in relatively shorter timeframes (as the timing is just right for this tech) for high agency folks who can also afford the ordeal in wanting to maximize for impact & ambition. Of course, there's real risk one may get no where, but even in failure, given you were building the LLM yourself, you might end up with other adjacent, high reward opportunities.
> The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM company, and still then it's a struggle. Those few that do train, they spend most of their budget on compute and have relatively small teams.
And the job postings are often ridiculous. I recently was an AMD job advert in Germany for an ML Kernel Engineer, not Senior mind you. The requirements went something like
> Masters Degree required with strong preference for a PhD with peer reviewed articles in {journals_list}
> 10+ years of experience in C/C++
> GPU programming experience required
> 10 more ridiculous lines
No idea how a teenager self teaching himself LLMs is supposed to even get a shot...
I have always been pro fundamentals. It caused me trouble early in my career with bosses that didn’t understand why I would spend time trying to understand how something worked at a low level if I was a high level user. But then knowing the fundamentals gave me an edge as a designer and developer by understanding capabilities and limitations of the tools I was using. For example understanding how indexes work internally in a relational database. So I see the value in this type of work, not to land a job as a LLM researcher, but as an informed user of the tool.
You can train and run small models on an old gpu. That’s what I’m doing now at, well, much older than 17. Does it produce a useful model? No. Not even remotely.
However, I do learn stuff about models that takes it from “magic” to “useful tool I understand the limitations of.”
Do I do it for that reason? No not really, I’ve never had luck learning something because it would be good for my career. I do it because at my core I’m a bored teenager who wants to make the computer do cool shit.
Yes and no. I believe the point he is making is simply that there is no substitute for fundamentals and first-principles thinking.
We had scores of students study how microprocessors work and compilers work over decades, yet we have 3 or 4 major processor companies and a handful of programming languages. Yet, what they learned was probably crucial in their development as engineers.
We are also so early right now that even 2-3 years from now who knows how many LLMs and model firms survive (esp. given the "snake eating its tail" venture/investor funding situation)
This is kind of different though isn't it? Doing an assembly or compiler class has pretty clear benefits in this regard.
But LLMs are tools. Does a great engineer need to know how vscode works? Might be helpful to understand how extensions work, LSPs, and project configurations.
Usually when working with any tools, you need to understand how to get the most out of your tool for your needs and that's about it. Core fundamentals about how software and hardware works in general seems like it would be MUCH more useful than LLM core knowledge.
What about other machine learning related skills? Does this wave of LLM mean less need for that kind of work?
I would think not, but when I started to look into OCR options recently - assuming that obviously a dedicated tool would do a better job than an LLM - I was wrong (apparently).
Not that expect to make it big as a LLM researcher but building something from scratch gives a much deeper understanding than what you can get from simply using something.
Much in the same way as implementing and designing your own programming language makes you a much better programmer.
There are high school students competing in contests that cover parts of the (Math) theory behind AI. A lot of high school research programs are integrating AI with other things and complex mathematical models…
To me, this is bizarre as Calculus is barely taught in high schools (and likely poorly).
Don’t get me wrong, these kids certainly aren’t the usual lot.
Yet, I really wonder if they know the fundamentals. Do they even understand derivatives or just memorized the rule for polynomials? Can they even explain what a transistor is?
Feels like they are trying sprint before even learning to walk.
I think his point is to do this to understand deeply what they can do, what they can’t do, and what they can almost do. And then find the highest value ‘almost’ use case and push there. Which doesn’t necessarily mean improve the llm, could be applying it in just the right way for the use case. Of course, the bitter lesson makes this hard and risky. But no more risky than investing your time in learning anything else these days.
I bet this will get less true over time though as the rate of change slows down, allowing specialized models/training for specific use cases that aren't TAM heavy enough for the big labs to go after them. It's just now any general model is the best thing to use for everything and you're wasting money to build something on what will certainly be obsolete by the time you can get it to market
But you can train a small LLM with a gaming graphics card -- I managed one on a GTX 1660. I don't think pg is suggesting that you try to chase the frontier. It's more like building your own OS in the 80s, or web server in the 90s -- sure, you'll never match the commercial offerings or the big OS projects, but building something from scratch within the limits of the hardware you can afford is amazing educationally.
> The reality is that an incredibly small minority of companies in the world do any real training or optimisation.
At the scale you are probably imagining, this is true - but take the hype out of the OP and what you have is just someone saying the field of data science exists and is growing.
I don't think pg is giving advice on what will lead most directly to a job, but rather what is the best learning for a 17yo.
A 17yo who trains their own LLM will have a much richer understanding of what AI is, how it works, what its potential capabilities and pitfalls are, versus someone who spends the same time doing something else.
when you think about all of the advancements since Attention / GPT a lot of it has been somewhat more obvious than in other fields, as is typical with the massive flood of innovation that follows a big breakthrough.
Paul likely assumes there will be a sequence of additional papers with the same impact as Attention is all you need, which will spawn a lot of opportunity for a larger group of experts who are conversant enough to advance the field even if they do not themselves create such a major innovation. Not only is this deeply exciting, it is also highly meritocratic as there is still scarcity of the kind of intellect and creativity necessary to swim there.
Machine intelligence might soon surpass it, though, and Deepseek is 100% Chinese mainland educated. Paul's description of building an LLM from scratch is meant as a vague starting point for being an innovator of the highest value aspect of modern AI innovation, not as a specific prescription.
This is a wildly incorrect and myopic view on the world.
Finetuning model is cheap and incredibly useful for deployment. You don't need to pre-train a frontier llm from scratch to make useful models.
There is tons of domains where you and fine-tune llms and deploy them for value in companies and for your own entrepreneurship ambitions. I have made this a big part of my career for the last few years and now I'm working on finetuning models for starting my own companies.
I find the fine tune approach more interesting than straight to RAG and MCP.
End of the day they're all customized data stores and protocols to interact with them. May as well stick to a uniform toolkit with fine-tunes.
Not that other tools aren't useful. But reaching straight for a bunch of infrastructure reliant services is like jumping in with k8s when you're still at a stage where basic mocks in code are sufficient.
I won't roll my own encryption or UI lib but want to stay focused on the incompleteness of the project I have to ship not all the buttons and knobs of some dependency or framework. Same old manage context switch problem.
Both of you are right. There is demand for tailored (fine-tuned) models; almost every enterprise would theoretically benefit from them.
But there are also a lot of prerequisites, namely does the enterprise have its sh*t together on a technical level. Does it have the processes and data pipelines available to train and benefit from these models? Probably not!
Learning should not be done only as a direct path to getting paid.
Learn to create pattern matching and intuition to solve future problems.
When you are 17 it is a good time to understand how the world works so you can build on top of it in the future. If we assume most tech is going to have an LLM as part the stack, a solid basis in how LLMs work is likely to help you in future endeavors the same way a solid basis in how the web works helps you today.
Maybe a 17 year old should learn both. As a small anecdote when I was 17 I learned a lot about load balancers, failover, and building self-healing systems running small hosting company that had to be fault tolerant when I was attending high school. This wasn't at state of the art levels (e.g. I wasn't configuring gigabit routers or global CDNs -- but it was useful pattern matching for future problems)
I currently don't touch any of that tech, but I have working knowledge that still serves me today.
It is not a skill that you will use in your day to day life, but I think it is part of the fundamentals now. Sure, LLMs are in a bubble, just like the web during the dotcom bubble, but web didn't disappear, and I don't expect LLMs to, even after the bubble bursts.
I didn't write a LLM from scratch but it is on my "wishlist" so to speak. From what it seems, a GPT-1 class LLM can be done from scratch in a few days and tens of dollars of cloud compute or a high-end gaming GPU.
It is an exercise not unlike building a compiler, a school classic. You are unlikely to ever work on a compiler, but at least, now, you know your tools a little better. It is not about becoming an expert, that takes years, it is about knowing what you are doing.
If you intend to make software engineering your career, you will want more than surface knowledge. And that part is entirely on you, or on your school if you are a student. Companies will pay for you to learn the fundamentals, they want short term returns, because you may leave at any time. But you as a software engineer may have 40+ years left, so it is worth thinking long term. Claude code may be obsolete in the next few years, but linear algebra is not going anywhere.
Of course, because it's not the LLM that's special but the training data. Nowadays, your favourite AI service to generate code for an LLM whenever you ask for it.
Y'all are missing the point: It's probably less than 100 hours to learn the foundations of one of humanity's most-current breakthroughs. It's a disservice to any young hacker to not learn it. Here's your curriculum. Watch these:
This is a really stupid take. You can learn to build an LLM, there are great resources to do so (there are books about building them from scratch), you can use older model GPUs or rent them by the hour. The value of understanding them is really high for anyone building any application that uses an LLM at any point.
It similar to understanding how a very basic CPU works. Just because I'm not going to work at intel or nvidia or whatever optimizing the hell out of a chip, it doesn't mean I just throw my hands up and think "magic" - the basic architecture isn't that difficult, and the value of knowing it is astronomical for anyone writing software.
A single 3090 will train qwen 0.8B just fine. While it’s not a very capable model any training technique you would want to master can be used to make real progress. And all the skills you need to learn how to do this can be learned watching Andrej Karpathy’s zero to hero series (shame he quit educational content and went to anthropic)
I do not think it is a proper thing to do for 17 y.o., unless they are exceptionally mathematically gifted, as proper understanding of how LLMs are trained requires a good grasp of calculus, understanding modern OS and SDE tools for proper implementation of pipeline etc.
I'd rather simply write another mnist implementation and check if I really like all that AI stuff at first place. Even then, before going into mature-on-the-way-to-dying tech (LLMs) I'd rather focus on fundamentals - good ols linear models, regressions, stat etc.
> I do not think it is a proper thing to do for 17 y.o
If I'd get a buck every time someone said something like this to me when I was in the 13-18 range, I wouldn't have a ton of money, but it's so very annoying when people tell you this.
Regardless if they're "gifted" or not, regardless if you believe in myths like that or not, let children explore what they want to explore, even if you don't understand what it is or why they want to explore that, just let people explore, regardless of age.
It was such a terrible experience being a young kid growing up, with so many adults spending hours trying to convince me to stop sitting in front of the computer so much doing whatever; "why are you even trying to learn that stuff, you have to go to school to understand anything of this" and so much other similar trash.
Sorry, not your fault and I'm borderline trauma-dumping now, but really sad to see this sort of gatekeeping on HN of all places, age is irrelevant to learning ANYTHING, in my humble opinion at least.
Kids, find anything interesting? Jump into it, ignore what adults tell you, and do whatever you feel like, you'll find your place eventually.
I just voiced my opinion. I just think buiding an LLM from the scratch for 17 y.o. is pointless exercise, advising a teenager to do so is borderline irresponsible, and frankly PG is simply virtue signalling here, as LLMs are still trendy, esp. in his circles.
There still will be varyy small number of outliers among youngsters who'd be able to extract tremensous value from such an excercise, but for most that'd be _IMO_ waste of of time, with illusion of understanding w/o actually having any.
Same! I just happened to disagree with your opinion, and frankly, I'd say trying to gatekeep what people learn is closer to "borderline irresponsible" compared to asking people to build/learn/do X.
> youngsters who'd be able to extract tremensous value from such an excercise
But they're youngsters, who are about "extracting value"? Life is about fun, not extraction, not value, not avoiding waste of time but literally enjoy what you do, nothing is more important (IMO).
Then who knows, doing fun stuff sometimes lead to useful stuff, like in my life. But if you only think about "extracting most value for time spent" or similar "optimization strategies", then you'd never discover this part of life.
> Life is about fun, not extraction, not value, not avoiding waste of time but literally enjoy what you do, nothing is more important (IMO).
This is, pardon, demagoguery. There is always "future fun" and "present fun" which a normal person would assign different nonzero weights (https://en.wikipedia.org/wiki/Discounted_utility). Besides, building a LLM _truly_ from the scratch, just using the famous 2017 paper and numpy manuals is not fun at all, esp. for a high schooler.
How would you filter out garbage from your training data, for example? If you are trying to use someone elses corpus, would it be "from the scratch" then?
I quoted the wrong thing. I meant to quote that Calculus is needed for building LLMs when all the equations have been figured out already of what is needed.
Why would you tell people that the correct order is to build foundational knowledge before exploring a subject? For some (many?) people, a 'proper' understanding develops _after_ the exploration.
> Why would you tell people that the correct order
Because I can?. JK. Because that was my experience, of someone who is 2.5 older than 17?
> For some (many?) people, a 'proper' understanding develops _after_ the exploration.
I am afraid you have a too confrontational attitude here, but I'll answer anyway: because I do not believe you can simply "explore" such complex topics like building an LLMs. You'd simply be unable to build LLM drom scratch, unless you'd call cargo-cult chaining magic numpy incantations you've taken from Karpathy's tutorials "exploring".
If I were in "exploratory" state of mins, I'd rather go from entirely different side - I'd try playing with LoRA-ing existing small LLMs, such as venerable 2 y.o. Mistral Nemo, to get "feeling" for what training is and how hyperameters influence the process.
Then it is not "building llm from scratch" in my book. Just mindlees following instructions. Could be educational yes, but only trivially useful, if you have no bloody idea what you are doing.
> Fooling around and getting results is what made it fun. Understanding came later.
> No it's not useful, no you don't know what you're doing, and no it is not 'proper'. But you're learning.
It is sold by PG as something special though.
> Maybe just accept that not everyone starts from fundamental theory, and there are lots of people who start learning by fooling around.
Even then LLMs are strange thing to advice to play with, when there are so much more interesting and theoretically accesible for a 17 y.o. so I wonder why would you'd particularly single it out.
> I do not think it is a proper thing to do for 17 y.o., unless they are exceptionally mathematically gifted
I attempted many projects at a young age that I was absolutely not equipped for. The result of the attempts more often than not left me equipped, every time it left me better off. This is terrible advice.
That'd would be a terrible advice if there weren't a plenty of other things "you are not equipped for", but far less daunting both theoretically and practically. Such as, say, convolutional neural networks, or some older ML tech. Or even something totally unrelated to ML.
Transformers are difficult to understand even to people with strong ML background, let alone a teenager.
You are assuming the 17yo in question as an untrained underdeveloped savage. If I were 17 in 2026, I would certainly have exploited all the availabilities from 2010 on - including YouTube, OpenCourseware, the Web simply (Sebastian Raschka etc.) and LLMs.
That 17yo would have already built many uncommon bases, and would build further.
When I was not 17 at the times of GPT2, I decided to not bother with learning how to build LLMs because it’s too expensive for an individual. This escalated quickly.
I am kind of amazed how negative the comments are here, especially on HN.
Learning to hack something together in high school using the latest technology (vacuum tubes, radios, microprocessors, web/javascript) has been a common theme in the tech world for generations. With LLMs and online tutorials, this isn't even a difficult suggestion. Do people think learning new tech is somehow wasted effort?
> With LLMs and online tutorials, this isn't even a difficult suggestion.
Don't many of the commercial ones prevent you from using them to build LLMs?
I would say the reason for the negativity is not because it's a bad idea for a project, or that doing projects in general is a bad idea (it's not!), it's because it's a very specific thing that is not for everyone. The best thing about computing is the low barriers to entry. You can basically work on anything that takes your fancy. So those who are interested in ML will be drawn to learn about LLMs. They don't need anyone to tell them to do it. Telling everyone to do it reminds me of the "just learn to code" stuff of a decade ago. No, please don't, please find something you enjoy.
Completely agreed. The point is the knowledge, the learning and the journey. If a kid has a passion for building or toying with LLMs, then of course, by all means, please start tearing them apart or even build and train your own model. You'll learn a ton, even if you won't necessarily end up using it here and now. The learning experience will compound and of course that will be useful.
The above is, after all, the whole genesis of the word 'hacker'. We should celebrate that.
How exactly does one go about "tinkering" with an LLM? Any architectural change you introduce needs fine tuning. That needs data and compute
I tried to modify the embedding output of bert to make it generate box embeddings instead of point ones. At the time I had access to university provided A100 gpus but even with all that a training run took half a day. Models these days I don't think I can train it in any reasonable time with that much compute.
(As a TML person, I'm obviously biased, but I couldn't resist because of "tinkering").
TBF it's hard to imagine a real architecture change that wouldn't require a ton of compute, but you could certainly fine tune and play with different recipes, loss functions, etc. And Claude can carry you a lot of the way through doing this.
One fun task is to invent a tool and then train a small model to use it. You could export that small model and run it locally for free forever to do your thing. I think this is what a lot of Software Engineering will look like later.
There are a lot of other high level abstractions here to look at. Prime Intellect has one.
The other thing to play with is self-hosting small models, but IMO most of the interesting stuff is actually related to multi-gpu or multi-node inference so there's not necessarily a ton to learn here.
This is one question that is not just for kids but for all "gpu poor" adults as well. One concept that I found useful is "toy models": small compute models that replicate a certain, not very well explained phenomenon that the large models have. Of course, you can't have a toy model for everything, some behaviors are only visible at scale. But, there is a bunch of things like in-context learning that you can play with without burning a ton of compute.
The best analogy is strip mining (big labs) vs cave exploration (solitary/small teams). I think this is how science progresses at the boundaries by smart/curious/hardworking individuals because depth is a requisite for finding the right questions and then the answer. It is not for everyone and it does not always work. But you learn a ton even if it doesn't pan out to be a big breakthrough.
This guidebook covers pretraining, post training (SFT, RL) and a couple other topics. And others authors have also written books that fit on single node reasonable hardware.
If you want to start with a pretrained base I built Gemma 270m and released it last year. This fits on a raspberry pi.
The fundamentals of AI don't require industrial amounts of large scale. Think of it like this, when I was learning how a plane worked when I was a kid I didn't build a 747 at home, I started with scale sized model planes. Same idea here.
> Do people think learning new tech is somehow wasted effort?
No. But funnily enough that is a promise by some of the AI cretins and their boosters. Oh yeah best case scenario you learn how to build LLMs for us. We’ll employ you. And then ultimately that just becomes training data for the LLMs to do it themselves.
HN is a tech news site sponsored by rich people and populated by rich people. Hanging out with people you claim you're tired of listening to raises interesting questions.
The suggestion here seems more inline with not tinkering and hacking but first principles learning typical of science and math curriculum. What made hacking radios and JS cool was the short feedback loops to see the progress you made. Unfortunately for LLMs starting from scratch doesnt give a good feedback loop on improvement, although starting from pre-trained model to quantising, fine-tuning might be more aligned with the hacking culture
No, but there's a WHOLE ENTIRE WORLD of tech that isn't LLMs. LLMs aren't the future or anything. Sure they'll keep existing. Right now though all the opportunities exist for people looking literally anywhere else
Let's be very, very generous and say that an LLM is intelligent. Does intelligence eliminated the need for tools? If you have an executive assistant and your executive assistant has ten executive assistants, does that mean that Microsoft Excel is obsolete?
Why the everliving fuck has everyone stopped building tools?
This is essentially most discussion here. I close the HN tab so often after seeing the people that just came to disagree with someone on the internet. There are too few discussions here that come from a place of "That's a great perspective and I can see it being useful when applied in X, just trying to solicit a discussion here, I wonder how this could work or break down in condition Y?"
I have also noticed a general drop in fawning over and deification of 'made a billion, must be a visionary' figures, and a receding tolerance for every sundry kind of hot take and proclamation. Mustn't be related.
That would be true in the 80s, maybe 90s. The world changed. Tech's role in society changed. Its greedy. People in power in tech have made it clear they are here for the money and anything that supports it works. What you see (even here on hn) reflects that common knowledge that any new tech will only end up screwing people. This isn't my opinion but a summary of what I conclude people think of tech.
I learned HTML when I was 17 in about 1995 and it's certainly taken me on a pretty fun career path. Less technical than LLMs for sure, but 'figure out where the industry is going and move what you're learning to there' is solid advice.
While knowledge is always great, I would encourage people not to seek advice from successful people like this (survivorship bias).
Moreover I am not sure it is even good advice? Would you advise a 17 y.o. to learn how transistors work or how to code (i.e. is LLM training the right level in the stack)? LLM training, a discipline where relevant work is already out of reach for 99.999% of budgets really as essential as this post implies?
> Would you advise a 17 y.o. to learn how transistors work or how to code
how many of us out here are doing work directly in what we got a degree in? I majored in economics and now I'm a CTO.
I would absolutely advise a 17 yo to learn how to code, understand how transitors work and how to code an llm. even if he never works on llms, you basically end up with a kid with applied knowlege of statistics, math, physics hardware, logic and a whole lot of practice in critical thinking.
I think it is. He isn't saying to learn how to train a LLM so that you can go on to train LLMs. He's saying to learn it so that you gain a deep understanding of how LLMs work. Ordinary startups can still benefit from things like training or fine tuning highly specialised smaller models, knowing how to select and configure an appropriate model for the task at hand, knowing what software to use and why, understanding what's going on behind the scenes instead of treating everything like a black box, having a higher level of intuition about LLMs generally, etc.
Most computer science courses do in fact teach things which are lower level than coding, such as how transistors work.
As a 17 y.o (way back in the last century) I didn’t need to be advised to learn about transistors. I just had a thirst for the knowledge.
I would encourage everyone to learn something about transistors. They are one of mankind’s most useful discoveries.
Hey, I'm 19 CS undergrad but i don't really know what to do but I really wanna build something that can shape the world. Can you tell me more about why you would advice someone to learn about transistors? what are the possible career paths?
Not the original commenter, but in my view as a crusty old software engineer, the reason to learn how transistors is because they're interesting and the key to how all computation is implemented. Once you learn how transistors work, you can learn how logic gates work. And then how CPUs and memory work. Bit by bit, computers stop being magic boxes and start being something built from technologies you understand.
Now, for most software careers (including my own), that knowledge isn't directly practical. BUT, the most important things for a successful, fulfilling career is curiosity and a willingness to dive into subjects _without_ necessarily knowing how (or even if) that knowledge will yield practical applications. That drive is what's going to lead to insights and breakthroughs over your career that wouldn't happen if you just let abstractions be abstractions.
I just don't think he realizes how saturated it got over the years. Or maybe he knows at a conscious level, but not subconsciously. Like, in 2000 (his era), it would have been really smart to study the source of Linux or Apache. Would have paid dividends over decades. Cuz that knowledge was so rare.
The number of people hacking on LLMs now dwarfs the number of people hacking on web servers 30 years ago, by several orders of magnitude.
And if you turn back the clock even more, I mean just even having access to a computer, let alone owning one, would have put you at a massive advantage.
I don't know what to call it. The pioneers should be respected obviously, but at the same time you need to understand that for them, the game wasn't nearly as played out as it is now.
I just don't think you can afford to be dicking around with LLMs like you could afford to dick around with random Linux distros 20 years ago. Too many people willing to do it for free these days.
IMO these kind of advices never matters. Any individual still needs to make tens or hundreds of little decisions (every day) themselves, and that's what really makes all the difference.
Right - like advising people to learn nuclear power back in 1992. Perhaps a good idea, but super specific already. My gut is ML/AI is even more complex in 2026… I can’t even remember all the abbreviations and the new ones emerging. And every sub component, such as attention or embeddings, are actually a discipline of its own already.
I would encourage young people to be born rich. It's the best time in 100 years to be advantaged. Why waste your potential by having your labor stolen?
As someone who teaches AI at a university, understanding the basics of how the LLM works is really invaluable to understanding where and how they'll be applied.
You're right that it's probably a little too in the weeds, but it's also a nice clear and fun objective that teaches you the basics. Like building a TODO list in JavaScript to learn webdev or a Gameboy emulator in C++ to learn how a CPU works.
Y'all are missing the point: It's probably less than 80 hours to learn the foundations of one of humanity's most-current breakthroughs. It's a disservice to any young hacker to not learn it...
Andrej Karpathy has a great Youtube series on how to build LLMs from scratch. Perfect for somebody who just learned a lot of high school math. Would start there and get busy with some handson python coding.
I'll use this post as a shameless opportunity to tell more people about a little side project, I made:
languagemodelbuilder.com teaches you (in a few hours to days) how to build an LLM from scratch. It's entirely free, without accounts, and without data collection.
I wish this was available for young folk in Romania: As it happens: macOS: 2.70% of desktop operating-system usage in Romania; OS X: 2.55%; Combined Apple desktop share: approximately 5.25%; Windows: 90.65%; Linux: 4.02% or so AI tells me.
There are GitHub repos with the code from Sebastian Raschka's books, which build an LLM from scratch in PyTorch (some examples run fine even on CPU-only setups) using a GPT-2-like architecture, as well as his new book on reasoning models that came out recently.
[Off-topic] Your name sounds familiar...
I don't think individuals have the resources to build an interesting llm. The l stands for large. You need a dataset too. Llms are only interesting because theyre large
And it's basically a weekend project to put transformers together in a ML library and train it.
The follow up comment,train it to play a game also doesn't make sense? Llms Sony really play games and there are better ml approaches to do that?
Incredible counterexample, but oddly relatable.
I'd probably have achieved techbro 'post-economic' status earlier if I focused on Android dev instead of the shiny (and new at that time) Xamarin for Windows phones.
God I almost invested in Xamarin after Windows Phone got aborted, I did spend a lil time on UWP, but thank god Flutter came out not so long after that. After all these years I learned to stay away from Microsoft tech stack.
I also did, and had written code for/on other mobile devices before, and did write a large number of even very ambitious software for other mobile devices later.
I do not see any past constructive experience as a waste of time.
Writing, supervising and training LLMs are now the purview of... even larger LLMs. Optimising CUDA kernels; hand-writing SIMD assembly to speed up data loading; tinkering with your particular brand of DRAM to see if there's anything to gain from optimising for its memory topology and NUMA --- these are now the job of AI.
There is very little reason for humans to get all too engrossed in this type of work now, today, with the hope of being good enough at it to command a high salary in 3-5 years. AI can already do it incredibly well, and they can do it persistently and doggedly 24 hours a day.
2 years before everyone was doing custom training. What happened to all those today when frontier models itself become more powerful than custom trained ones?
The times are different. When I was 17, we had to buy records; a 17yo today can listen to the whole of the available produced music, plus interviews and all other uncommon and related material, for free from the comfort of "here and now".
Possibilities exist now that did not exist before. Those who do not exploit this are fools.
The amount of people who missed the point here is absurd. He's advocating for learning about how LLMs work. For the sake of learning. Because no one's going to invent the next thing without at least some understanding of the current thing.
As an aside, for someone interested and who's an absolute beginner, can someone please recommend good resources on how to build LLMs from scratch? Thank you in advance.
I second 1. I'm a newbie in neural networks and I think it's an excellent book! One of my barriers in ML is the resources, I find them overcomplicated or too simplistic without a mid term. It's not the case of this book, everything is well-explained. Neural Networks aren't fun for me, but this book makes it very interesting.
I think this is the best structured class out there that teaches how to scale LLMs . Hope the 17 year old knows linear algebra. Building an intuition for the shape of the matrices is important. A lot of understanding the 'building from scratch' means understanding choices like why RoPE instead of the original frequency based positional encoding. Start with Karpathy and then go to CS 336
If I were 17 again I'd prepare to go for volunteering overseas after high school for 1-2 years (plenty of free options in the EU where you might only need to cover the plane ticket). See the world, help others and then think about what you want to do.
520 comments
[ 0.30 ms ] story [ 181 ms ] thread>Whoa. I’m 19 and I trained a 100M language model from scratch. Did a v2 now with a new SFT experiment to see if I can get better results on same size.
Either way, this isn't really advice for 17 year olds. Pg is thinking out loud about the pathways for founders.
I told my much younger brother when he was 12 what programming was and it'd be a great career. He looked into it and within months was writing CLI games. Eventually releasing his own unity 3d game on steam as a teen.
Eventually he got into CS and did really well because none of it was scary and new. He parlayed that into role at Meta out of university.
My point being, 17 year olds have time to learn new skills and guidance can go a long way.
(Edit: And learn how honest business works)
I think the last one was seeing a skilled electronics repairman do surgery on a CT machine controller.
I mean first that is already what plenty of 17yo are actually doing, because that is what they do at school or in parascholar activities. There are already countless of such tutorials where you can do that in an afternoon.
The pointless part though is precisely why Amazon and others are hunting for rare books, all the low hanging fruits have been picked already so just training a bigger model will simply mean burning more energy and money. Sure training a small one for the basic principle is a great pedagogical thing, training another one, medium, then maybe a large one, is also good in term of learning the process and architecture, but one should not expect it to be useful out of that context.
Pure players are precisely doing everything they can to corner the market by making their own scale unreachable by others. Smaller players with access to lesser infrastructure are thus betting on different market, e.g. embedded systems.
17yos should definitely build their (L)LMs from scratch and whatever bigger model they can train for free, or for cheap, but they should not expect that to bring them any riches.
You should want to train a LLM from scratch as an intellectual curiosity itch that needs to be scratched.
The idea of learning one hot skill that has a pot of gold waiting at the end of it was a brief moment in time that came and went.
When I was 17, we would have said obviously support vector machines are the future. Neural networks overfit and don't work.
The reality is that an incredibly small minority of companies in the world do any real training or optimisation. It's unnecessary and inefficient for most purposes unless you are fully dedicated to being an LLM company, and still then it's a struggle. Those few that do train, they spend most of their budget on compute and have relatively small teams.
Getting experience in this field requires having access to very expensive hardware to begin with. And the skills will be quite hard to convert into any real value for someone, leading to a decent income, unless you have a ton of funding from patient investors, or you have decent contacts in Bay Area networks to get hired at the right place.
With all due respect, paulg is in somewhat of a bubble, this is not congruent with the global situation.
In reality the problem is that it gets blasted out of the water by a much worse architecture trained on 10000x the infrastructure. And while I'm sure the freshly brought in ML student came up with a 10%, even 30% better architecture, it just doesn't matter. (and never mind that even OpenAI hasn't really solved a voice model yet. Try it. It can probably match 2026-quality call centers, but it's no substitute for an actually empowered human)
... and yet, if you look at what hyperscalers are getting paid for ... comfortably more than half the income is training. Which makes no sense on so many levels.
e.g. https://valueaddvc.com/blog/inference-chips-vs-training-chip... (I get it, not great first source, but st
I don't know where you are located, but in EU, in China, and yes even in Silicon Valley, the vast majority of companies do not do any real AI engineering. There's nothing wrong with it, it's just not a smart path for most purposes. You can do amazing things without training, and if you try to train, you cannot get anything amazing unless you burn millions.
Very few people can afford to play the long game and cross that dessert. And, sure, you will not get far without good engineering, but good engineering is definitely not sufficient and is not the primary bottleneck.
For some very niche cases I think this is probably the case but for the vast majority, the company's data isn't as useful as they think it is or anywhere near the size needed.
1. Both training and optimisation will get significantly cheaper and easier quickly.
2. Politics will probably get even more insane before a potential reprieve on the 20th of Jan 2029.
3. The big AI firms will become part of the surveillance capitalism network, if they're not already.
So I think for self-protection a lot of companies will be looking near to medium term AI independence.
Fine-tuning a model or LoRA based on the companies data set is more feasible but you're likely going to need several runs as you test/try out different base models, parameters, etc. This is why there are a lot of fine-tuned models on huggingface based on base or instruction-trained models from the larger AI companies that have released open weight models (Microsoft, Google, IBM, Mistral, DeepSeek, Qwen, etc.).
Training is limited on memory first (storing training data and weights) and computation second. Realistically you need to own or rent 2-8 H100/B100 devices or Google's TPUs.
The majority of workflows for a company providing AI capabilities are likely best solved by tailoring a system prompt for the chosen model, evaluating the prompt and model with tools like promptfoo, and then running it on a compute cloud provider (including AWS Bedrock). If the company is big/financially well off enough they could look at buying the hardware needed to run it on their own servers.
For other uses like agentic software development you'd need to spin up a suitable model on a compute cloud provider (or local hardware if the model is small enough) and then tell your IDE/editor to use that model. You would need some way of benchmarking and evaluating the models to see if they are capable of doing the tasks you need. -- There have been some tests done by people on YouTube that suggests that Qwen 3.8 27B is a decent model, but your needs may vary.
There are plenty of areas were we need people to do this for insurances, banks etc.
AI/ML exists on many levels.
The only jobs that he found he was highly over qualified and they paid very little.
In any case, it doesn't look like there's this crazy rush to hire all ML talent, even the one that understand the math and technology deeply.
Maybe people simply don't want math PhDs but something else? Since 1-2 years ago I started doing consulting/freelancing in the ML space, but more on the infrastructure, deployments and similar stuff, as a general purpose developer, and I have a waiting list of clients interested in more work, some of them even trying to recruit me to work for them full-time as well. I'm based in continental Europe, fwiw.
Is it possible to see some of your old works? Personal research?
Citation needed. I'm guessing you're not counting children.
https://www.hesa.ac.uk/news/29-08-2024/higher-education-stud...
You’re missing the point. Understanding how Unity works fundamentally makes you a better Unity dev.
Writing your own game engine makes you realize that the Unity engine is not really that well written....
It is viable as a toy project, but there are vanishingly few career opportunities.
It seems to me like he’s saying that doing this thing would be 1) fun and 2) a great way to become employable in the future. I don’t believe he’s saying that this project would be some kind of job training exercise.
I "guess" you could put all your effort into moving up to the next level in computer science, or you could put the same progressive effort into reaching new levels in video games on the same hardware.
Alternatively you could even put all your effort into social activities and leave the technology to other people entirely.
It might even be possible to find a balance between things that are widely "understood" "socially", and those that are not ;)
The same is not so different for AI. A few people design novel AI, but there are a lot of people training AI (especially if you include fine tuning) and implementing AI, even as a hobby.
But that was way back in the early 1970's and all I had to work with was a mainframe.
Well the mainframe itself wasn't bad, the real show-stopper was that I didn't own the computer outright, no strings attached, no debt, etc.
>I'd probably try to make an LLM that I could use on some specific problem.
I thought so too back then, still do so I guess this is one of those things that could stand the test of time. I always wanted to start with something a lot simpler than a Moon mission myself. At 17 I already had a significant breakthrough in the chem labs and it was from alternatives to a single processing step plus everything that descended from that, rather than trying to tackle a much more complex detailed multi-step synthesis. I was only 17 but I was not trying to be a slouch, I don't think pg was either at that age but his advice is not for just anybody. I couldn't have done it if I hadn't made major progress since being 16, and it really emphasized at the time how much maturity can make a difference. My imagination ran wild as I extrapolated :)
In a reply from LeCun to pg:
>>I'll figure out a set of methods and architectures beyond LLMs that can quickly learn to perform physical tasks as efficiently as humans and animals. That last item is also what I would if I were 30, 40, 50, or 66 years old
I see no reason to stop at 66 either ;)
But I figured that people owning more computer power than I could ever afford were going to be doing something like this as soon as they could, without having to wait for something like an LLM to arrive before getting peoples' attention.
It did seem like things were going to take longer than you expect, so it's pretty good to have a lifetime of concentrating on the specialized natural science domain expertise, focused now for 50 full years on how it would combine if AI ever got good enough.
Both the natural science and the AI need to be a major cut above, I still see dramatic room for improvement in my own work. If I'm going to have to rely on "other peoples' AI" then that natural science component is going to have to pull a lot of weight to keep up with the kind of computers that only rich-as-hell high-rollers have access to.
I just finished fine tuning Gemma e2b for local code completion on my local machine.
This comment just reinforces what the post actually means. We need people that are LLM natives, computing solves itself with time and with scale adjustments
2. That's why I have an agentic agent as well installed, Qwen 27B, outrageously good, better than sonnet 2 years ago. And it's mine, I can give it confidential info to work with since I own the whole chain. See where I'm going with this?
I can see the point. It's unlikely that a 2.4T LLM will be integrated into, say, a pesticide drone. You'll still need some kind of LLMs to achieve maneuvers that "normal" programmings can't achieve.
But what if everything basically turn into that? Essentially, instead of build me a web app to solve X and do Y, build me an LLM to serve X and do Y. (unless the current LLMs are able to do it end-to-end but then they can hardly write coherent software/personal opinion).
over the past 6 months, i’ve been training models by taking academic papers in my domain (cybersecurity) whose authors haven’t released the implementation code, the training code, or both … and then trying to reconstruct them using Claude/Codex and then successfully training them ..
through this process, i’ve picked up ML frameworks (lightning ai) figured out how to prepare and stream super large audio datasets, and more importantly, successfully trained a model that i’m now using in my startup .. and the alpha is huge .. and i’m not even an engineer or developer in the traditional sense.
.. each training run currently uses an H100 and takes 3 to 4 days on DO … roughly $400 to $500 per run ..
More importantly going through this process has made me realise how insane this is going to become … people are going to be able to train models for an enormous range of specialised domains without necessarily being traditional ML engineers or working at one of the handful of companies doing serious training as u describe ..
as long as ur creative, have deep domain knowledge, and can figure out how to use the tools … the barrier to doing genuinely interesting ML work is collapsing
Now with LLMs, people write native apps in Rust, and I'd like to think some of them found that there isn't such a huge jump in difficulty they assumed there would be.
But that’s the tip of the iceberg. If you have any ambitions of doing this professionally, it quickly becomes clear that all it’s all about knowing how to deal with problems that are only present at massive scale, when an LLM is actually L and becomes AI.
The mundane details about how to build a tiny autocomplete model and the maths behind it you can learn in a couple weeks easily. It’s not black magic, there are much harder areas of computers science.
That's the same case finding companies that will actually pay for hand made LLM instead of using something from big providers will be hard because most companies won't be able to afford it.
Yeah if you find a company that will do that stuff directly, good for you, but you will have to be very lucky and you will have to compete with other people who followed PG advice.
So I would rather learn all there is about properly using LLMs and integrating them with existing systems, that will most likely by useful for 90% of companies out there.
Building business niche harnesses is in my opinion much better direction. Knowing what will work best in specific cases is it FTS or vector search, optimisation of usage, getting best results while using cheaper models, knowing how to use tools to run models on the servers, and all the tooling around that like various MCP or just tooling that will be provided to models.
That is what I am currently busy with and I already have customers for that knowledge.
I think a lot of people assume that only the big AI labs can do cutting edge research, but there's a strong argument you can do it as part of little tech as well.
For anyone who wants to dork around there is https://github.com/rasbt/LLMs-from-scratch which is something amazing that I think anyone who wants to engineer things around LLMs should at least blast through and read.
And so I think the idea is more to understand tomorrow ... from first principles.
In the late 80's, as a teenager, I learned x86 assembly and C because that was the only way to squeeze out enough juice from my shitty CGA (and later VGA) card to programm the games/graphics that interested me.
I haven't written assembly in years.
But whatever I did in my career: it helped me and gave me an edge over my peers to have a foundation that is very close to the metal.
Current AI can automate significant amounts of grunt work in programming and math. It's good at running web searches and writing summaries. There are a few other niches where it is currently successful. But other than that, many corporate AI projects are spectacular failures.
So just given what we have in hand, assuming no further breakthroughs, then we're maybe looking at AI being somewhat bigger than the Internet. Which would make it a revolutionary technology, sure.
But to get from "a revolutionary technology" to "the substrate the future runs on", then you need to assume more breakthroughs: long-context operation over weeks or months, displacing human workers 100% instead of 75%, and the ability to directly economically compete with actual humans. And people are investing literal trillions of dollars to make that future come true, without really thinking through what truly competitive-with-human AI would actually mean. We might be looking at massive job loss, centralization of power, fully automated "companies" with no humans dominating markets, and other dystopian scenarios.
And in those worlds, it's unclear that being good at CUDA and matrix math will be all that helpful, careerwise. The AIs are already pretty good at that stuff. Data scientists get paid OK when they actually get hired, but it's not everything college students were promised in the 2010s, either.
We can't yet build a fully-general competitor for the human mind. But we're getting closer. And if we ever do build one, the consequences will be really weird in any number of ways. So I worry about visions of the future that assume AI keeps improving significantly, but that also assume it still somehow remains a "normal" technology that doesn't, for example, render most humans fundamentally uncompetitive.
At 75% replacement of a worker we would already have huge job losses as each individual would be doing what several before did.
The only alleviation would be the creation of new equivalently paid jobs, which is no better than a hypothesis right now.
Citation needed
We don't know this. So many people are simply claiming this confidently, and a lot of them are betting their careers on it, but nobody has a crystal ball. Whenever someone tells you confidently, and without any doubt or qualifications, that something "is the future," be skeptical.
I remember when the Segway was definitely going to change urban planning worldwide.
AI is a great solution in niche areas but generally doesn't make much money. All the large companies are in the negative.
The steam engine was less of a bubble and was much more revolutionary and had a greater impact.
I think this is disingenuous. One could say that drones are nowhere the efficiency limit either: a bee can fly for hours on the energy contained in just a few milligrams of honey, while our best battery-powered drones can't stay airborne for more than 30 minutes. But comparing energy efficiency of electric/mechanical devices to their biological counterparts is not an apples-to-apples comparison. There's a world of difference between the energy storage and delivery mechanisms.
And as many have pointed out already in the siblings, it's not just about the compute but the access to petabytes of training data.
Incidentally, the skills for the lowest levels of LLMs aren't that far removed from those needed for mobile telephony, in that both are based on maths, computation and information theory.
What most people want from mobile technology is for it to work, not too expensively, and for it to get out of their way.
What most people want out of AI is for no leader to emerge and wield supremacy against the rest of us. People are afraid of it in ways they weren't afraid of mobile, so they're more willing to work together against whoever is in the lead.
Its more like an arms race and less like a utility. The disadvantage I face when my competition has better mobile coverage and bandwidth is minor. The disadvantage I face when my competition has better intelligence on tap is much more significant.
No.
That’s what people like us on HN want. The people out in “Greater Userland” just want the black box to answer their questions. They could care less who is behind it. They don’t yet attach their black box to Amazon or Microsoft etc. And most won’t care enough to be inconvenienced even when they do make the connection. (As your competition argument implies.)
Heck, a lot haven’t even made the connection between the black box that gives them answers and data centers. They think, “ ChatGPT good” and at the same time think “data centers bad”.
But it was generally seen as a gimmick instead of desired before Apple made it look good. Even when the iPhone came out, one of the jokes was how the grid of icons looks like how a Windows user's desktop would look like when they didn't understand the filesystem.
We don't really have the demand for as many telecom companies as actually exist in the world. There's a reason we just have one Whatsapp and one Instagram, not three or four almost-but-not-quite clones in every single country that mostly differ in branding. The reason for the current situation has mostly to do with regulation and traditional, enterprise, "obviously every country needs a separate local branch, because that's what mcDonalds does" thinking. Technology has very little to do with it.
This is why the telecom world now consist of equipment manufacturers, who do most of the hard tech stuff, and actual telecom companies, who operate the equipment, rig towers in their local country, and maybe write some glue code to integrate a core from vendor A, a billing system from vendor B and a CRM / corporate invoicing system from government-approved local vendor C.
Banking also works similarly, though modern Neobanks / Fintechs and bank consolidation are slowly dissolving the concept of national bank branches.
Paul, I think, is talking about achieving outsized outcomes in relatively shorter timeframes (as the timing is just right for this tech) for high agency folks who can also afford the ordeal in wanting to maximize for impact & ambition. Of course, there's real risk one may get no where, but even in failure, given you were building the LLM yourself, you might end up with other adjacent, high reward opportunities.
And the job postings are often ridiculous. I recently was an AMD job advert in Germany for an ML Kernel Engineer, not Senior mind you. The requirements went something like
> Masters Degree required with strong preference for a PhD with peer reviewed articles in {journals_list} > 10+ years of experience in C/C++ > GPU programming experience required > 10 more ridiculous lines
No idea how a teenager self teaching himself LLMs is supposed to even get a shot...
1. I can make turn a stone and water into a delicious soup
"17 year olds, learn to build an LLM from scratch"
2. This soup would be more delicious if we add a few carrots. Does anyone have carrots
"Increase your chance of success by getting a Masters degree"
3. How about potatoes?
"And get a PHD"
4. What about some salt?
"And publish some peer reviewed articles in {journals_list}
5. We should also add beef
"Now work in the industry for 10 years"
6. See, this soup is delicious, and I made it all with a stone
"See, you're rich, and it's all because you learned LLMs as a 17 year old"
However, I do learn stuff about models that takes it from “magic” to “useful tool I understand the limitations of.”
Do I do it for that reason? No not really, I’ve never had luck learning something because it would be good for my career. I do it because at my core I’m a bored teenager who wants to make the computer do cool shit.
We had scores of students study how microprocessors work and compilers work over decades, yet we have 3 or 4 major processor companies and a handful of programming languages. Yet, what they learned was probably crucial in their development as engineers.
We are also so early right now that even 2-3 years from now who knows how many LLMs and model firms survive (esp. given the "snake eating its tail" venture/investor funding situation)
But LLMs are tools. Does a great engineer need to know how vscode works? Might be helpful to understand how extensions work, LSPs, and project configurations.
Usually when working with any tools, you need to understand how to get the most out of your tool for your needs and that's about it. Core fundamentals about how software and hardware works in general seems like it would be MUCH more useful than LLM core knowledge.
I would think not, but when I started to look into OCR options recently - assuming that obviously a dedicated tool would do a better job than an LLM - I was wrong (apparently).
Not that expect to make it big as a LLM researcher but building something from scratch gives a much deeper understanding than what you can get from simply using something.
Much in the same way as implementing and designing your own programming language makes you a much better programmer.
There are high school students competing in contests that cover parts of the (Math) theory behind AI. A lot of high school research programs are integrating AI with other things and complex mathematical models…
To me, this is bizarre as Calculus is barely taught in high schools (and likely poorly).
Don’t get me wrong, these kids certainly aren’t the usual lot.
Yet, I really wonder if they know the fundamentals. Do they even understand derivatives or just memorized the rule for polynomials? Can they even explain what a transistor is?
Feels like they are trying sprint before even learning to walk.
At the scale you are probably imagining, this is true - but take the hype out of the OP and what you have is just someone saying the field of data science exists and is growing.
A 17yo who trains their own LLM will have a much richer understanding of what AI is, how it works, what its potential capabilities and pitfalls are, versus someone who spends the same time doing something else.
Paul likely assumes there will be a sequence of additional papers with the same impact as Attention is all you need, which will spawn a lot of opportunity for a larger group of experts who are conversant enough to advance the field even if they do not themselves create such a major innovation. Not only is this deeply exciting, it is also highly meritocratic as there is still scarcity of the kind of intellect and creativity necessary to swim there.
Machine intelligence might soon surpass it, though, and Deepseek is 100% Chinese mainland educated. Paul's description of building an LLM from scratch is meant as a vague starting point for being an innovator of the highest value aspect of modern AI innovation, not as a specific prescription.
Finetuning model is cheap and incredibly useful for deployment. You don't need to pre-train a frontier llm from scratch to make useful models.
There is tons of domains where you and fine-tune llms and deploy them for value in companies and for your own entrepreneurship ambitions. I have made this a big part of my career for the last few years and now I'm working on finetuning models for starting my own companies.
End of the day they're all customized data stores and protocols to interact with them. May as well stick to a uniform toolkit with fine-tunes.
Not that other tools aren't useful. But reaching straight for a bunch of infrastructure reliant services is like jumping in with k8s when you're still at a stage where basic mocks in code are sufficient.
I won't roll my own encryption or UI lib but want to stay focused on the incompleteness of the project I have to ship not all the buttons and knobs of some dependency or framework. Same old manage context switch problem.
But there are also a lot of prerequisites, namely does the enterprise have its sh*t together on a technical level. Does it have the processes and data pipelines available to train and benefit from these models? Probably not!
Learning should not be done only as a direct path to getting paid.
Learn to create pattern matching and intuition to solve future problems.
When you are 17 it is a good time to understand how the world works so you can build on top of it in the future. If we assume most tech is going to have an LLM as part the stack, a solid basis in how LLMs work is likely to help you in future endeavors the same way a solid basis in how the web works helps you today.
Maybe a 17 year old should learn both. As a small anecdote when I was 17 I learned a lot about load balancers, failover, and building self-healing systems running small hosting company that had to be fault tolerant when I was attending high school. This wasn't at state of the art levels (e.g. I wasn't configuring gigabit routers or global CDNs -- but it was useful pattern matching for future problems)
I currently don't touch any of that tech, but I have working knowledge that still serves me today.
Think long term.
I didn't write a LLM from scratch but it is on my "wishlist" so to speak. From what it seems, a GPT-1 class LLM can be done from scratch in a few days and tens of dollars of cloud compute or a high-end gaming GPU.
It is an exercise not unlike building a compiler, a school classic. You are unlikely to ever work on a compiler, but at least, now, you know your tools a little better. It is not about becoming an expert, that takes years, it is about knowing what you are doing.
If you intend to make software engineering your career, you will want more than surface knowledge. And that part is entirely on you, or on your school if you are a student. Companies will pay for you to learn the fundamentals, they want short term returns, because you may leave at any time. But you as a software engineer may have 40+ years left, so it is worth thinking long term. Claude code may be obsolete in the next few years, but linear algebra is not going anywhere.
I feel like the "ALWAYS HAS BEEN" meme is apropos here.
-- 3Blue 1Brown's Neural Network Series: https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_6700...
-- Karpathy on LLMs: https://www.youtube.com/watch?v=7xTGNNLPyMI
-- Stanford CS336: https://www.youtube.com/watch?v=JuoVZkPBiKk
Then do this hands-on:
-- Karpathy's zero-to-hero: https://karpathy.ai/zero-to-hero.html
It similar to understanding how a very basic CPU works. Just because I'm not going to work at intel or nvidia or whatever optimizing the hell out of a chip, it doesn't mean I just throw my hands up and think "magic" - the basic architecture isn't that difficult, and the value of knowing it is astronomical for anyone writing software.
Wait no it’s not, that was always happening.
What’s crazy is that people still believe in it.
I'd rather simply write another mnist implementation and check if I really like all that AI stuff at first place. Even then, before going into mature-on-the-way-to-dying tech (LLMs) I'd rather focus on fundamentals - good ols linear models, regressions, stat etc.
If I'd get a buck every time someone said something like this to me when I was in the 13-18 range, I wouldn't have a ton of money, but it's so very annoying when people tell you this.
Regardless if they're "gifted" or not, regardless if you believe in myths like that or not, let children explore what they want to explore, even if you don't understand what it is or why they want to explore that, just let people explore, regardless of age.
It was such a terrible experience being a young kid growing up, with so many adults spending hours trying to convince me to stop sitting in front of the computer so much doing whatever; "why are you even trying to learn that stuff, you have to go to school to understand anything of this" and so much other similar trash.
Sorry, not your fault and I'm borderline trauma-dumping now, but really sad to see this sort of gatekeeping on HN of all places, age is irrelevant to learning ANYTHING, in my humble opinion at least.
Kids, find anything interesting? Jump into it, ignore what adults tell you, and do whatever you feel like, you'll find your place eventually.
There still will be varyy small number of outliers among youngsters who'd be able to extract tremensous value from such an excercise, but for most that'd be _IMO_ waste of of time, with illusion of understanding w/o actually having any.
Same! I just happened to disagree with your opinion, and frankly, I'd say trying to gatekeep what people learn is closer to "borderline irresponsible" compared to asking people to build/learn/do X.
> youngsters who'd be able to extract tremensous value from such an excercise
But they're youngsters, who are about "extracting value"? Life is about fun, not extraction, not value, not avoiding waste of time but literally enjoy what you do, nothing is more important (IMO).
Then who knows, doing fun stuff sometimes lead to useful stuff, like in my life. But if you only think about "extracting most value for time spent" or similar "optimization strategies", then you'd never discover this part of life.
This is, pardon, demagoguery. There is always "future fun" and "present fun" which a normal person would assign different nonzero weights (https://en.wikipedia.org/wiki/Discounted_utility). Besides, building a LLM _truly_ from the scratch, just using the famous 2017 paper and numpy manuals is not fun at all, esp. for a high schooler.
Can you provide an example?
Because I can?. JK. Because that was my experience, of someone who is 2.5 older than 17?
> For some (many?) people, a 'proper' understanding develops _after_ the exploration.
I am afraid you have a too confrontational attitude here, but I'll answer anyway: because I do not believe you can simply "explore" such complex topics like building an LLMs. You'd simply be unable to build LLM drom scratch, unless you'd call cargo-cult chaining magic numpy incantations you've taken from Karpathy's tutorials "exploring".
If I were in "exploratory" state of mins, I'd rather go from entirely different side - I'd try playing with LoRA-ing existing small LLMs, such as venerable 2 y.o. Mistral Nemo, to get "feeling" for what training is and how hyperameters influence the process.
That's what I would call exploring.
I too started exploring programming as a teen by cargo culting. Fooling around and getting results is what made it fun. Understanding came later.
Then it is not "building llm from scratch" in my book. Just mindlees following instructions. Could be educational yes, but only trivially useful, if you have no bloody idea what you are doing.
> Fooling around and getting results is what made it fun. Understanding came later.
It is not how LLMs are "built from scratch".
But you're learning.
Maybe just accept that not everyone starts from fundamental theory, and there are lots of people who start learning by fooling around.
It is sold by PG as something special though.
> Maybe just accept that not everyone starts from fundamental theory, and there are lots of people who start learning by fooling around.
Even then LLMs are strange thing to advice to play with, when there are so much more interesting and theoretically accesible for a 17 y.o. so I wonder why would you'd particularly single it out.
I attempted many projects at a young age that I was absolutely not equipped for. The result of the attempts more often than not left me equipped, every time it left me better off. This is terrible advice.
Transformers are difficult to understand even to people with strong ML background, let alone a teenager.
That 17yo would have already built many uncommon bases, and would build further.
That is, a 17yo with proper mentality.
Learning to hack something together in high school using the latest technology (vacuum tubes, radios, microprocessors, web/javascript) has been a common theme in the tech world for generations. With LLMs and online tutorials, this isn't even a difficult suggestion. Do people think learning new tech is somehow wasted effort?
Don't many of the commercial ones prevent you from using them to build LLMs?
I would say the reason for the negativity is not because it's a bad idea for a project, or that doing projects in general is a bad idea (it's not!), it's because it's a very specific thing that is not for everyone. The best thing about computing is the low barriers to entry. You can basically work on anything that takes your fancy. So those who are interested in ML will be drawn to learn about LLMs. They don't need anyone to tell them to do it. Telling everyone to do it reminds me of the "just learn to code" stuff of a decade ago. No, please don't, please find something you enjoy.
The above is, after all, the whole genesis of the word 'hacker'. We should celebrate that.
I tried to modify the embedding output of bert to make it generate box embeddings instead of point ones. At the time I had access to university provided A100 gpus but even with all that a training run took half a day. Models these days I don't think I can train it in any reasonable time with that much compute.
(As a TML person, I'm obviously biased, but I couldn't resist because of "tinkering").
TBF it's hard to imagine a real architecture change that wouldn't require a ton of compute, but you could certainly fine tune and play with different recipes, loss functions, etc. And Claude can carry you a lot of the way through doing this.
One fun task is to invent a tool and then train a small model to use it. You could export that small model and run it locally for free forever to do your thing. I think this is what a lot of Software Engineering will look like later.
There are a lot of other high level abstractions here to look at. Prime Intellect has one.
The other thing to play with is self-hosting small models, but IMO most of the interesting stuff is actually related to multi-gpu or multi-node inference so there's not necessarily a ton to learn here.
So, you build one from scratch.
The best analogy is strip mining (big labs) vs cave exploration (solitary/small teams). I think this is how science progresses at the boundaries by smart/curious/hardworking individuals because depth is a requisite for finding the right questions and then the answer. It is not for everyone and it does not always work. But you learn a ton even if it doesn't pan out to be a big breakthrough.
https://ravinkumar.com/GenAiGuidebook/book_intro.html
This guidebook covers pretraining, post training (SFT, RL) and a couple other topics. And others authors have also written books that fit on single node reasonable hardware.
If you want to start with a pretrained base I built Gemma 270m and released it last year. This fits on a raspberry pi.
https://developers.googleblog.com/en/introducing-gemma-3-270...
The fundamentals of AI don't require industrial amounts of large scale. Think of it like this, when I was learning how a plane worked when I was a kid I didn't build a 747 at home, I started with scale sized model planes. Same idea here.
No. But funnily enough that is a promise by some of the AI cretins and their boosters. Oh yeah best case scenario you learn how to build LLMs for us. We’ll employ you. And then ultimately that just becomes training data for the LLMs to do it themselves.
But why are people cynical? they ask.
Good. It took a long while for ICs to be paid more than 'management' in the US. It's still a hard-fought battle here.
// a flood of coders in the '10s-early '20s whose only creed was FYPM and "what's the minimum
I wonder what possibly could have happened in that timeframe?
https://www.theguardian.com/technology/2014/apr/24/apple-goo... https://en.wikipedia.org/wiki/High-Tech_Employee_Antitrust_L...
Why the everliving fuck has everyone stopped building tools?
https://github.com/raiyanyahya/how-to-train-your-gpt
Moreover I am not sure it is even good advice? Would you advise a 17 y.o. to learn how transistors work or how to code (i.e. is LLM training the right level in the stack)? LLM training, a discipline where relevant work is already out of reach for 99.999% of budgets really as essential as this post implies?
how many of us out here are doing work directly in what we got a degree in? I majored in economics and now I'm a CTO.
I would absolutely advise a 17 yo to learn how to code, understand how transitors work and how to code an llm. even if he never works on llms, you basically end up with a kid with applied knowlege of statistics, math, physics hardware, logic and a whole lot of practice in critical thinking.
I think it is. He isn't saying to learn how to train a LLM so that you can go on to train LLMs. He's saying to learn it so that you gain a deep understanding of how LLMs work. Ordinary startups can still benefit from things like training or fine tuning highly specialised smaller models, knowing how to select and configure an appropriate model for the task at hand, knowing what software to use and why, understanding what's going on behind the scenes instead of treating everything like a black box, having a higher level of intuition about LLMs generally, etc.
Most computer science courses do in fact teach things which are lower level than coding, such as how transistors work.
Now, for most software careers (including my own), that knowledge isn't directly practical. BUT, the most important things for a successful, fulfilling career is curiosity and a willingness to dive into subjects _without_ necessarily knowing how (or even if) that knowledge will yield practical applications. That drive is what's going to lead to insights and breakthroughs over your career that wouldn't happen if you just let abstractions be abstractions.
I’d find some high level tutorials online that maybe use one of the free circuit simulator tools.
falstad.com/circuit/
Nice animation on this site
If you are really curious there are many simple circuits you can build.
It’s quite a thrill to get an LED to blink with a self built circuit.
I’m also a crusty old sw eng. I did spend sometime being a young electronics engineer.
Good luck with the journey.
The number of people hacking on LLMs now dwarfs the number of people hacking on web servers 30 years ago, by several orders of magnitude.
And if you turn back the clock even more, I mean just even having access to a computer, let alone owning one, would have put you at a massive advantage.
I don't know what to call it. The pioneers should be respected obviously, but at the same time you need to understand that for them, the game wasn't nearly as played out as it is now.
I just don't think you can afford to be dicking around with LLMs like you could afford to dick around with random Linux distros 20 years ago. Too many people willing to do it for free these days.
Hell yeah. Transistors are pretty awesome.
You're right that it's probably a little too in the weeds, but it's also a nice clear and fun objective that teaches you the basics. Like building a TODO list in JavaScript to learn webdev or a Gameboy emulator in C++ to learn how a CPU works.
Since he has not given any reference point it is equally as good advice as „just learn everything slightly valuable“.
So the real question is: „What should I not learn in favor of learning this.“
Or in other words: His comment is not usable because it can mean anything or nothing.
Here's your curriculum. Watch these:
* 3Blue 1Brown's Neural Network Series: https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_6700... * Stanford CS336: https://www.youtube.com/watch?v=JuoVZkPBiKk * Karpathy on LLMs: https://www.youtube.com/watch?v=7xTGNNLPyMI
Then do this hands-on:
* Karpathy's zero-to-hero: https://karpathy.ai/zero-to-hero.html
I have my opinion on this but I'd like to hear the HN opinion, I will just say one thing:
If you are starting with little knowledge, like a 17 year old would, letting an LLM explain it to you is a terrible idea.
languagemodelbuilder.com teaches you (in a few hours to days) how to build an LLM from scratch. It's entirely free, without accounts, and without data collection.
Thanks a ton for building this.
And it's basically a weekend project to put transformers together in a ML library and train it.
The follow up comment,train it to play a game also doesn't make sense? Llms Sony really play games and there are better ml approaches to do that?
I do not see any past constructive experience as a waste of time.
There is very little reason for humans to get all too engrossed in this type of work now, today, with the hope of being good enough at it to command a high salary in 3-5 years. AI can already do it incredibly well, and they can do it persistently and doggedly 24 hours a day.
Possibilities exist now that did not exist before. Those who do not exploit this are fools.
2. LLM from 0 to Hero, and nanoGPT by Andrej Karpathy
(Fully free, of course.)