“Many of the jobs we do today would have looked like trifling wastes of time to people a few hundred years ago, but nobody is looking back at the past, wishing they were a lamplighter. If a lamplighter could see the world today, he would think the prosperity all around him was unimaginable. And if we could fast-forward a hundred years from today, the prosperity all around us would feel just as unimaginable.” Loving the techno optimism.
I got a kick out of that. Spoken like a man who doesnt have to concern himself with the price of housing or any other necessity as we watch this strange short lived phenomena called the middle class disappear and we're back to the haves and have-nots.
It's a lot more comforting to work on AI (or any tech for that matter) if you believe or force yourself to believe it will be used for good. OpenAI's for-profit and gatekeeping approach is unlikely to be the path to the prosperity Sam Altman envisions.
> Although it will happen incrementally, astounding triumphs – fixing the climate, establishing a space colony, and the discovery of all of physics – will eventually become commonplace.
Paging Dr. Bullshit, we've got an optimist on the line who'd like to have a word with you.
I think that's part of the carefully-crafted hype messaging. Close enough to get excited about, but far enough away that by the time we get there people will have forgotten we were supposed to have it by then.
If anything, I would say that that's a very optimistic take. The hype train is strong, but that's largely what it is once you look at the details. What we have right now is impressive, but no one has shown anything close to a possible path from where we are right now to AGI. The things we can do right now are fancy, but they're fancy in the same way good autocomplete is fancy. To me, it feels like a local maxima, but it's very unclear whether the specific set of approaches we're exploring right now can lead to something more.
I'm not convinced, and neither is Sam Altman himself [0]. Also, if that projection holds, and that's a big if, the purported breakthrough would cost 10^6 times as much as GPT-4 took to train. That's over 100 million dollars [1] times a million. That adds up to over 100 trillion dollars, in the ballpark of four times the GDP of the whole of United States.
The thing is that it looks like, or perhaps I should say it's "understood" at this point, that transformer's abilities scale pretty much linearly with compute (there is also some evidence they scale exponentially with parameter count, but just some evidence).
Right now there is insane amounts of money being thrown at AI because progress is matching projections. There doesn't seem to be a leveling off or diminishing returns taking place. And that's just compute, we could probably freeze compute and still make insane progress just because optimizations have so much momentum right now too.
Yeah, that's my number one question, too. Sure, he happened to be appointed the manager of the team who cracked intuitive algorithms through deep learning, but what does he know about superintelligence? IMO that's a completely separate question, and "foundation models continue to improve" is absolutely not related to whether or not an intelligence explosion is guaranteed or not. I'd trust someone like Yudkowsky way more on this, or really anyone who has engaged with academic literature on the subjects of intentionality, receptive vs. spontaneous reasoning, or really any academic literature of any kind...
Does anyone know if he's published thoughts on any serious lit? So far I've just seen him play the "I know stuff you don't because I get to see behind the scenes" card over and over, which seems a little dubious at this point. I was convinced they would announce AGI in December 2023, so I'm far from a hater! It just seems clear that they're/he's guessing at this point, rather than reporting or reasoning.
Really he assumes two huge breakthroughs, both of which I find plausible but far from guaranteed:
With nearly-limitless intelligence and abundant energy
This is painting a picture of a utopia. I'm not sure about that.
The entire AI trend - long term is based on the idea that AI will profoundly change the world. This has sparked a global race for developing better AI systems and the more dangerous winner takes all outcome.
It is therefore not surprising that billions of dollars are being spent to develop more powerful AI systems as well as to restructure operations around them.
All the existing systems we have must fundamentally change for the better if we want a good future.
The positive aspects / utopia promises have much more visibility to the public than the negative effects / dystopian world.
ARE WE TO pretend that Human greed, selfishness, desires to dominate and control, animalistic behaviour, use of technologies for war and other destructive purposes don't exist?
We are living in times of war and chaos and uncertainty. Increasingly advanced technology is being used on the battlefield in more covert and strategic ways.
History is repeating itself again in many ways. Have we failed to learn? The consequences might be harsher with more advanced technology.
I have read and thought deeply about several anti AI doomer takes from prominent researchers and scientists but I haven't seen any which aren't based on assumptions or foolproof. For something that profoundly changes the world, it's bad to base your hopes on assumptions.
I see people dunking on llms which might not be AI's final form. Then they extrapolate that and say there is nothing to worry about. It is a matter of when not if.
The thought of being useless or worse being treated as nothing more than pests is worrying. Job losses are minor in comparison.
The only hope I have is that we are all in this together. I hope peace and goodwill prevails. I hope necessary actions are taken before it's too late.
A more pragmatic perspective indicates that there are more pressing problems that need to be addressed if we want to avoid a doomer scenario.
The sentiment to me is “we need unlimited compute and data” both of which are clearly limited. There is definitely more technology to invent and understand in order for us to do more with less
The future will be abundant, because deep learning works. To achieve that, we need to be calm, but cautious. And, we need to fund infra (chips and power) so that AGI isn't limited to the ultra-wealthy.
My take:
* Foom/doom isn't helpful. But, calm cautiousness is. If you're acting from a place of fear and emotional dysregulation, you'll make ineffective choices. If you get calm and regulated first, and then take actions, they'll be more effective. (This is my issue with AGI-risk people, they often seem triggered/fear/alarm-driven rather than calm but cautious)
* Piece is kind of a manifesto for raising money for AI infra
* Sam's done a podcast before about meditation where he talked about similar themes of "prudence without fear" and the dangers of "deep fear and panic and anxiety" and instead the importance of staying "calm and centered during hard and stressful moments" - responding, not reacting (strong +1)
* It's no accident that o1 is very good at math physics and programming. It'll keep getting much better here. Presumably this is the path for AGI to lead to abundance and cheaper energy by "solving physics"
Well put! I would disagree on two fundamental points, tho:
1. If you honestly think that millions/billions of people are at serious risk of avoidable harm that everyone else is ignoring, "calm down" can be a hard dictum to follow. Sam Altman has won, it's easy for him psychologically to say "well, lets just stick to the status quo and do our best every day and it'll probably work out". Made-in-house bias is strongest when "in-house" is your own mind, after all.
2. Your scare quotes makes it seem like you might agree, but: physics is the study of the physical world, thinking it can be 'solved' is like thinking mathematics, psychology, or anthropology can be 'solved'. It's fundamentally anti-science and very, very dangerous to be talking like that. Truth isn't absolutely relative, but science also isn't a collection of facts written in stone that we need to finish unearthing; it's a collection of intellectual tools.
Few short learning performance scales with model size. Afaik they don't see a plateau yet and the race is on the ingest more data and come up with better tuning techniques.
"AI is going to get better with scale" to me says almost nothing at all. It includes anything from we 100x the scale and get 1% improvement to 2x the scale and get "AGI".
To paraphrase Goggins, "Who's gonna carry the cabbage?"
While it's true there are a lot of jobs obsoleted by technological progress, the vision of personal AI teams creating a new age of prosperity only makes sense for knowledge workers. Sure, a field worker picking cabbage could also have an AI team to coordinate medical care. But in this brilliant future, are the lowest members of society suddenly well-paid?
The steam engine and subsequent Industrial Revolution created a lot of jobs and economic productivity, sure, but a huge amount of those jobs were dirty, dangerous factory jobs, and the lion's share of the productivity was ultimately captured by robber barons for quite some time. The increase in standard of living could only be seen in aggregate on pages of statistics from the mahogany-paneled offices of Standard Oil, while the lives of the individuals beneath those papers more often resembled Sinclair's Jungle.
Altman's suggestion that avoiding AI capture by the rich merely requires more compute is laughable. We have enormous amounts of compute currently, and its productivity is already captured by a small number of people compared to the vast throngs that power civilization in total. Why would AI make this any different? The average person does not understand how AI works and does not have the resources to utilize it. Any further advancements in AI, including "personalized AI teams," will not be equally shared, they will be packaged into subscription services and sold, only to enrich those who already control the vast majority of the world's wealth.
The thing is: robotics is knowledge work. Supposing a scenario in which AI makes advancing fields of engineering and science much more rapid, it will be leveraged to build and cheapen robotic labor. There would be a gap period where AI is smart but unable to perform labor without humans, which could be ugly, and then we reach effective post-scarcity and post-humans-being-useful. Where we go from there could be heaven or hell depending on who's in charge.
> If we want to put AI into the hands of as many people as possible, we need to drive down the cost of compute and make it abundant (which requires lots of energy and chips). If we don’t build enough infrastructure, AI will be a very limited resource that wars get fought over and that becomes mostly a tool for rich people.
i think this is the prevailing wisdom but theres an angle that openai doesnt value and therefore isnt mentioned. There's far more compute sitting idle in everyone's offices and homes and pockets than there are in the $100bn openai cluster. it just isnt useful for training because physics. but its useful for inference. local LLMs ship this-next year in Chrome (gemini nano) and Apple (apple intelligence) that will truly be available for everyone instead of going thru OpenAI's infra. they'll be worse than GPT4, but only for a couple more years.
Especially when you separate the ethereal "hard problems" from every day queries local LLMs can answer equally as well as SOTA models, the value proposition for these expensive models plummets. If it can't solve real hard, long horizon problems the 10% lift on a given benchmark is not a material value prop to the end user to choose a local free version over the API costs or the monthly subscription.
I like Sam's philosophy on this and I generally agree with him. However, I do not like how all the wealthy AI people are hand-waving the massive labor market shift in the coming years.
> As one example, we expect that this technology can cause a significant change in labor markets (good and bad) in the coming years, but most jobs will change more slowly than most people think, and I have no fear that we’ll run out of things to do (even if they don’t look like “real jobs” to us today). People have an innate desire to create and to be useful to each other, and AI will allow us to amplify our own abilities like never before. As a society, we will be back in an expanding world, and we can again focus on playing positive-sum games.
It's very easy as an extremely rich person to just say, "don't worry, in the end it'll be better for all of us." Maybe that's true on a societal scale, but these are people's entire worlds being destroyed.
Imagine you went to college for a medical specialty for 8-10 years, you come out as an expert, and 2 years later that entire field is handled by AI and salaries start to tank. Imagine you have been a graphic designer for 20 years supporting your 3 children and bam a diffusion model can do your job for a fraction of the cost. Imagine you've been a stenographer working in courtrooms to support your ill parents and suddenly ASR can do your job better than you can. This is just simple stuff we can connect the dots on now. There will be orders of magnitude more shifts that we can't even imagine right now.
To someone like Sam, everything will be fine. He can handle the massive societal shift because he has options. Every a moderately wealthy person will be OK.
But the entire middle class is going to start really freaking the fuck out soon as more and more jobs disappear. You're already seeing anti-AI sentiment all over the web. Even in expert circles, you can see skepticism. People saying things like, "how do I opt out of Apple Intelligence?" People don't WANT more grammar correction or AI emojis in their lives, they just want to survive and thrive and own a house.
How are we going to handle this? Sam's words of "if we could fast-forward a hundred years from today, the prosperity all around us would feel just as unimaginable" doesn't mean shit to a family of 4 who went through layoffs in the year 2025 because AI took their job while Microsoft's stock grows 50%.
For this reason I read Andrew Yang’s “The War on Normal People”. Besides UBI and “social credits”, I don’t see him offer that many other solutions to this problem. UBI also still needs to be proven as far as I’m aware.
When o1 was released, I ran an internal eval and saw it plainly outperforming our highly educated colleagues. I had goosebumps, and haven’t been able to sleep well for days. This will dramatically impact society in 2-5 years.
Do you know of any relevant material related to this?
Welcome to the anxiety party, it sucks in here. As someone who's been working on AI theory full time for ~1 year, I desperately wish we could go back to the days of my faraway youth (5 years ago) before intuition was cracked on accident by spellcheck algorithms. I agree with him that it holds the key to massive prosperity, but selfishly, it's gonna upend my life and the lives of everyone I love. Already has for me, as I grapple with how to (ethically) pay rent while spending all day lighting the Warning Beacons of Gondor...
The only real answer, IMHO, is to vote for political systems that put control of society (and AI) in the hands of the public. Call it socialism, call it Georgism, call it anarcho-free-market-space-communism, call it whatever you want; there's no way that "a tiny number of people have immense inherited power" (capitalism) and "people fundamentally understand themselves as members of a tribe put in opposition to all other tribes by default" (nationalism) mesh well with an intelligence explosion.
Here's to hoping the haters are right, and we all turn out to be wrong! I'll be thrilled if Sam Altman is just a rich company leader in 10 years, and intuitive algorithms are still confined to direct usage (chatbots).
o1 was what got you stirred up? It honestly feels like an incremental change at best to me. I had similar feelings about gpt-3.5, but since then my fears have normalized into a sort of dull, typical (for me) cynicism (so no sleepless nights).
We're going to need to link the two. Those wealthy enough to not care, we're going to have to organize and make them care. Ideally we can find a way to do it nonviolently.
I wish I could upvote this more than once. I feel like every conversation about AI changing society comes from rich founders telling Joe banker to "Just start a company. AI will make it easy."
The reality is, this transition is going to be painful for the average person.
Wow, great catch. Something tells me he rolled this himself. Clearly he's trying to coin a term for personal legacy reasons, and I say godspeed. Holocene is a little vague, the information age is too entrenched to get anyone's attention, a/the singular age / the singularity are way too deeply associated with the doomer community, and the cybernetic age (a term coined for academia!) is too associated with playful science fiction.
I'm personally rooting for cognitive being the word of the next few decades, but that's just a shout from the sidelines. Only time will tell what humanity latches on to, but I wouldn't be surprised if this blog post/subdomain was referenced in a Wikipedia page's Etymology section in 10-15 years...
Although this blog post & discussion has my anxiety at an 8, something's oddly comforting about the thought of Sam Altman fiddling with tailwind classes to get his profound aesthetic just-right. Something undeniably relatable and human. Hate the man all you want (I do!), but he's clearly acting in some sort of good faith.
You are right. I overlooked the simplicity of the headline. Thanks for calling attention to that.
> Something tells me he rolled this himself.
This would be cool.
> I'm personally rooting for cognitive being the word of the next few decades
I like that one too.
> he's clearly acting in some sort of good faith.
He's always been one to think and write for himself. Huge respect for him. Even though it needles me every moment that they still call themselves "Open"AI, I have so much respect for the guy, especially because PG basically told the world he was the next Michael Jordan of startups, and he actually went and fulfilled that. Not many people have it in them to live up to hype like that (Lebron being the only other one I can think of OOTOMH)
> Deep learning works, and we will solve the remaining problems. We can say a lot of things about what may happen next, but the main one is that AI is going to get better with scale
I'm not an AI skeptic at all, I use llms all the time, and find them very useful. But stuff like this makes me very skeptical of the people who are making and selling AI.
It seems like there was a really sweet spot wrt the capabilities AI was able to "unlock" with scale over the last couple years, but my high level sense is that each meaningful jumps of baseline raw "intelligence" required an exponential increases in scale, in terms of training data and computation, and we've reached the ceiling of "easily available" increases, it's not as easy to pour "as much as it takes" into GPT5 if it turns out you need more than A Microsoft.
This is the part that really gets me. This is a thing that you say to your team, and a thing you say to your investors, but this isn't a thing that you can actually believe with certainty is it?
you need some amount of irrational definite optimism + knowing things others dont to be a good founder. that kind of reality distortion field is why sam is sam and we are here debating phrases on an orange website.
Related, I tongue-in-cheek believe that something analogous to the actual SCP object for a "reality distortion field" may in fact exist. There is zero good explanation for "Teflon Don" or the North Carolina Lieutenant Governor getting away with all the stuff they do while Al Franken got politically crucified.
The least-magical answer for that is that some people have fundamentally different ways of approaching the world, and certain things will be tolerated by certain sets of supporters.
Why not? People believe in all sorts of weird stuff, theirs just happens to be one you don't agree with. Some people believe there are gods up in the sky that will smite them, and go to war with people that believe in a different god that will smite them for different reasons. Some people believe we landed on the Moon, others do not. What matters is what can you convince others to do based on your rational.
With enough time it seems a reasonable assertion, but the key part is how much time. It feels like he thinks "any day now" where I think it'll be much longer. This all of course assumes that "the remaining problems" means to achieve human-like intelligence, which is perhaps the wrong problem to be solving in the first place. I'd rather have AI systems that don't have human flaws.
I'm not here to defend sama, but certain things cannot be proven until they arrive - they can only be extrapolated from existing observations and theoretical limits.
Imagine the Uranium Committee of early 40's, where Szilard and others were babbling about 10kg of some magical metal exploding briefly with the power of a sun, with the best evidence being some funky trail in an alcohol vapor chamber.
Maybe sama is right, maybe not, but the absence of evidence is not evidence of absence.
I'm sure you know that people in the AI community have been predicting big things ever since, I don't know, the 1970s? It's only 10 years away again. This time it's for real, right?
Alchemists predicted the transmutation of metals into gold for centuries, and on a sunny day in the 20th century, it arrived (a bit radioactive, but still).
Unless the human brain is made of some sacred substance, the worst-case scenario is that we will extrapolate current scanning methods into the future and run the scanned model in silica. I'm not recommending this "just for fun," but the laws of physics don't forbid it.
>Alchemists predicted the transmutation of metals into gold for centuries, and on a sunny day in the 20th century, it arrived (a bit radioactive, but still).
So is Sam Altman the modern day alchemist? Making predictions based on faulty methods and faulty understanding (per your gold example)?
What will happen is that we'll shift the economy around based on inflated tech promises and ruin people's lives. No big deal I guess.
If you are comparing AI to alchemy, a subject that after thousands of years still isn’t delivering on its promises (even with the assistance of modern technological magic), then surely you can see how that’s something of a self-own.
I have little doubt that even when we have superintelligent AI solving science and such problems way beyond humans it will still be dismissed as extra spicy autocomplete.
The AI predictions based on Moore's law type reasoning by Kurzweil, Moravec etc have been pretty accurate and not subject to the it's always 10 years ahead thing.
I agree. He's probably been conditioned by experience to speak with confidence until proven wrong ("strong opinions, weakly held"), but I don't like it either. Oh... the lost art of saying, "In my opinion."
Hah, atomic power is a great point of comparison: people in the "atomic age" expected atomic power to be everywhere. Electricity too cheap to measure, cars/planes/appliances all powered by small nuclear reactors... That's without going into the real nonsense like radium toothpaste.
And here we are today where nuclear energy is limited to nuclear weapons, a small set of military vehicles and <10% of the world's electricity production. Not nothing, sure, but nothing like past predictions either.
Last I checked the giant nuclear fusion reactor in the sky is driving an substantial increase in solar energy.
The toothpaste and similar products were pretty ill advised, vaseline and uranium glass are still collectable and are seeing a ressurrence of new interest: https://old.reddit.com/r/uraniumglass/
It's about stuff we don't know yet. From today's lens, the essay seems absurd. But I think it's hinging on continued discoveries that improve one or all of learning algorithms, compute efficiency, compute cost and applying algorithms to real world problems better.
5 years ago, I wouldn't have believed any of what exists today. I saw internal demos that showed 2nd or 3rd grade reading comprehension in 2017 and statements were made about how in the next decade, we will probably reach college level comprehension. We have come so far beyond that in less than half the time. Technology isn't about scaling incrementally and continuing on the same path using the same principles we know today. It's about disruption that felt impossible before - that feels like a constant to me now. Seeing everything I've seen in the last 20 years, it's going to continue to happen. We just can't see it yet.
Scaling Improvement has never been Linear though. Every next gen model so far has required at least an order of magnitude increase in compute, sometimes several more. So it's not a new revelation and these companies are aware of that. Microsoft for instance is building a 100B data center for a future next generation model releasing in 2028.
If models genuinely keep making similar leaps each generation then we're still a few generations before "More than a Microsoft".
So at what point do the linear increases in capability not justify the exponential compute and data requirements, or when do we run out of resources to throw at it?
I never said I thought increase in ability was linear either. We're encroaching on phenomena that's genuinely hard to describe/put a number on but GPT-3 is worlds apart of 2 and it feels like 4 is easily ten times better than the OG 3.
I can say Improvement lags behind compute somewhat but that’s really it.
That said, it's ultimately up to the people footing the bill isn't it ?
The question is: For a given problem in machine intelligence, what's the expected time-horizon for a 'good' solution?
Over the last, say, five years, a pile of 50+ year problems have been toppled by the deep learning + data + compute combo. This includes language modeling (divorced from reasoning), image generation, audio generation, audio separation, image segmentation, protein folding, and so on.
(Audio separation is particularly close to my heart; the 'cocktail party problem' has been a challenge in audio processing for 100+ years, and we now have great unsupervised separation algorithms (MixIT), which hardly anyone knows about. That's an indicator of how much great stuff is happening right now.)
So, when we look at some of our known 'big' problems in AI/ML, we ask, 'what's the horizon for figuring this out?' Let's look at reasoning...
We know how to do 'reasoning' with GOFAI, and we've got interesting grafts of LLMs+GOFAI for some specific problems (like the game of Diplomacy, or some of the math olympiad solvers).
"LLMs which can reason" is a problem which has only been open for a year or two tops, and which we're already seeing some interesting progress on. Either there's something special about the problem which will make it take another 50+ years to solve, or there's nothing special about it and people will cook up good and increasingly convenient solutions over the next five years or so. (Perhaps a middle ground is 'it works but takes so much compute that we have to wait for new materials science for chip making to catch up.')
Progress might be logarithmic in compute, but compute (transistors/sqinch and transistors/$) is growing exponentially with time.
Despite what skeptics have been saying for decades, Moore's Law is alive and well - and we haven't even figured out how to stack wafers in 3 dimensions yet!
Oh wow! Could you please share what processors are exponentially faster than those of 10 years ago? I'm not seeing any here: https://www.cpubenchmark.net
Macbook Airs have 20 billion+ transistors, compared to 50 million on the Pentium 4 in the early 2000s. Moore's law is about transistor density, not processor speed, which is gated by thermal limits.
As large as the absolute largest models are today, they are still microscopic compared to our brains. A 1.7T param model would only store an actual total of about 850 GB if fully saturated (4 bits of information per weight estimated for bf16 transformers), a lot less than a human brain with 150T synapses running in full analog precision. We need to scale the current gen of models at least another 10-100x to even reach the human level of complexity, something we'll be able to do in the next two decades.
And well then there's going beyond just text. Current multimodal models are basically patchwork bullshit, separately trained image/audio to text/embeddings encoders slapped onto an existing model and hoping it does something neat. Tokenization and samplers are likewise bullshit that's there to compensate for lack of training compute. Once we have enough to be able to brute force it properly with bytes in, bytes out, regardless of data format, the results should be way better.
Comparing a human brain in these terms makes it incredibly obvious how inefficient the human brain actually is. A 1.7T model can answer questions about practically anything. You say a human brain has 150T params. So what? It struggles to give masterful answers in even 1 domain, let alone dozens/hundreds. We need to stop comparing parameters and synapses as if they actually matter, because AFAIK, they really don't.
Well once again it turns out that what is hard for people is easy for computers, and vice versa. The things we go to college for 6 years for they can (relatively) master in a week of pretraining. We are optimized to smartly kill things, eat them, and reproduce, that's what machines will beat us at last lol. Right now a human expert is still obviously better in depth, but nowhere close in breadth. Probably not for much longer though, at least on the historical time scale.
And granted a lot of parts of the human brain are dedicated to specific jobs that are entirely nonexistent in a normal LLM (kinematics, vision, sound, touch, taste, smell, autonomic organ control) so the actual bit we should be comparing for just language and reasoning is way smaller. Still the brain is pretty efficient inference energy wise, it's like the ultimate mixture of experts, extreme amounts of sparse storage and most of it is not computed until needed. The router must be pretty good.
Boston Dynamics and Waymo might not have gotten human levels of competency with those two particular tasks, but we've already got robots that are better than drunk/tired/angry humans at it, and they're getting better at it.
In terms of energy use the human brain is way more efficient than LLMs. It's a completely different hardware model - the brain may have trillions of synapses but they only fire occasionally. I agree you have to compare more on the results than number of synapses etc.
It's something that gives me pause on the idea that we have to build many GW of power stations. It may be possible to get much more energy efficient AI via better algorithms.
> Comparing a human brain in these terms makes it incredibly obvious how inefficient the human brain actually is. A 1.7T model can answer questions about practically anything. You say a human brain has 150T params. So what? It struggles to give masterful answers in even 1 domain, let alone dozens/hundreds. We need to stop comparing parameters and synapses as if they actually matter, because AFAIK, they really don't.
Meanwhile the best class AI is trained on the output of human intelligence. When AI can learn by itself in the same way humans do (or even more efficient ways) that's when we can say human intelligence has been surpassed. Until then it's just tools for humans to use to augment their intelligence.
Analog systems are not known for being very precise- they're noisy, signals get corrupted easily- and that's why we prefer digital ones. As soon as we had the technology, we switched everything we could- audio and video recording, telephone calls, photography, to a digital medium. This makes me wonder if the seemingly extraordinary efficiency of artificial neural networks is simply due to the precision with which they can be trained.
With respect to brain activity, how do we know its really noise, and not just layers of meaning--or at least purpose--which we don't yet understand?
If straightforward binary signaling was so universally superior, I think the worldwide network of over a quintillion ruthlessly self-replicating nanobots would be using a much more heavily after the last billion years.
Electrical analog systems are since there's 1001 ways to create or conduct away electricity that adds noise, chemical ones might not suffer nearly as much too.
There's also an interesting bit I've observed with LLMs, quantization in range of 4-8 bits doesn't reduce performance as much as it mainly reduces consistency. If you generate an answer a bunch of times and take the average you'll end up with roughly the same result as an fp16/32 would do every time. In nature, being inconsistent is usually weighed negatively with death... so that's probably why it hasn't caught on even if it is more efficient. Or this is enough of a different abstraction that we can't draw parallels anyway.
> But stuff like this makes me very skeptical of the people who are making and selling AI.
What is there to be skeptical of? OpenAI made their current product using a 10G$ investment plus a few they are not disclosing, and now they will start to do it at scale.
Yes you are correct that jumps in intelligence were enabled by exponential increases in scale. That makes me more bullish on AI, not less. It suggests that we can continue exponentially scaling compute like we have done for the past few decades, and also get intelligence improvements from it.
> If we want to put AI into the hands of as many people as possible, we need to drive down the cost of compute and make it abundant (which requires lots of energy and chips). If we don’t build enough infrastructure, AI will be a very limited resource that wars get fought over and that becomes mostly a tool for rich people.
This seems to be the key of the piece to me. It's his manifesto for raising money for the infra side of things. And, it resonates: I don't want ASI to only be affordable to the ultra rich.
I don't see how the current state of AI tech is not for the ultra-rich. It's hundreds of billions of dollars worth of investment from who? Corporations and those who own the most shares of those corporations(the rich).
Billionaires could make so much change happen but instead they are building bunkers and riding giant dicks into space while simultanously touting that they are looking out for humanity.
> to put AI into the hands of as many people as possible, we need to drive down the cost of compute and make it abundant (which requires lots of energy and chips)
Not really. The problem is that learning requires scale, mostly of data. That scale places AI providers at the nexus of value, with OpenAI as the presumptive market organizer and leader. Reducing compute costs would just mean they can capture more of the value. Data costs orders of magnitudes more than compute because it requires curation, so even if individual developers could get compute, they can't get data, so size/access matters.
That's good for this community and could be good for the state of the art and the overall potential contributions of AI. And more paying customers could be good for OpenAI. But it won't put AI in front of non-paying customers/developers, unless their value is otherwise harvested.
> I don't want ASI to only be affordable to the ultra rich.
As a developer or consumer?
And you don't mean it's ok if AI is only affordable for the moderately rich, do you? I agree it's hard to state which developers/people/customers should be subsidized. Generally we subsidize education but not profit or war. Sometimes culture. Companies will subsidize complementary goods and input factors. Otherwise? Not much history of benevolent subsidy.
AI has as much potential to shape society as freeways and the automobile did in the US, but few understand how, and I've seen no plans on point.
With electric energy networks and transportation, the central government has a role in reducing hostaging by hold-out's. With education, states have an incentive to attract and build talent (albeit now reduced with trans-national outsourcing and remote work). But otherwise, it's private enterprise and resource-weighted customers.
Changing that is not really Sam Altman's job. His job is to deliver that value, sooner rather than later. Most would be uncomfortable with AI overlords expressing opinions on cultural values or economic distributions to be imposed.
* "Prudence without fear" (Sam referencing another quote)
* "if you create the descendants of humanity from a place of, deep fear and panic and anxiety, that seems to me you're likely to make some very bad choices or certainly not reflect the best of humanity."
* "the ability to sort of like, stay calm and centered during hard and stressful moments, and to make decisions that are where you're not too reactive"
Easy for him to say: AI is almost guaranteed to hand a massive W to capital and L to labor. He is holding a title to rule over hell in one hand and promising to lead us to heaven with the other.
Who's going to work at all these companies then? Unless every single profession suddenly only requires 1 person to do the entire thing with no management, coordination or hierarchies, a lot of people will be labor not capital.
Yes, so there will be all sorts of attempts, and the productive forces will compete with each other, while landowners and owners of harder to disrupt industries will have the power to choose among them, deciding who will be allowed to be successful and who will fail.
There isn't a single ordinary person at the table, no labour unions, no political parties or anything democratic. It's Microsoft, it's Google, it's some venture-capital owned French firm, some venture-capital owned German firm, etc. Maybe if Schmidhuber's stuff is good as he hopes it is maybe there'll be an Austrian firm, etc., but mostly, it'll be a capital intensive business controlled by people with capital.
> * "if you create the descendants of humanity from a place of, deep fear and panic and anxiety, that seems to me you're likely to make some very bad choices or certainly not reflect the best of humanity."
This line of reasoning doesn't hold for me, as you could apply it to any technology, including ones actually very likely to destroy human civilization.
Sometimes, not building a given thing at all is better than building it with even the best intentions.
I'm personally not sure on which side AI falls, but denying that such things exist at all seems intellectually dishonest.
"What is good?" or in other words "What are people for?" is a question that cannot be answered by intelligence no matter how great, because the complexity of the question is a function of the intelligence of the asking entity and it's always greater than the intelligence of the asking entity (human or transhuman cyborg or whatever.)
AI is a side-show.
Intelligence is ambient in living tissue, so we already have as much intelligence as is adaptive. We don't need more. As talking apes made out of soggy mud wrapped around calcium twigs living in the greasy layer between hard vacuum and a droplet of lava which in turn is orbiting a puddle of hydrogen in the hem of the skirt of a black hole our problems are just not that complicated.
Heck, we are surrounded by four-billion year-old self-improving nanotechnology that automatically provides almost all our physical needs. It's even solar-powered! The whole life-support system was fully automatic until we fucked it up in our ignorance. But we're no longer ignorant, eh?
The vast majority of our problems today are the result of our incredible resounding success. We have the solutions we need. Most of them were developed in the 1970's when the oil got expensive for a few minutes.
Must we boil the oceans just to have a talking computer tell us to get on with it? Can't we just do like the Wizard of Oz? Have a guy in a box with a voice changer and a fancy light show tell us to "love each other"? Mechanical Turk God? We can use holograms.
Existential horror of this whole conversation aside, this is a beautifully written comment. I'd read your novel, but you don't seem to have one, so I'll have to settle for following you on Mastodon.
> "What are people for?" is a question that cannot be answered by intelligence no matter how great
It absolutely can be answered, but only the the intender. Who is and who was and who is to come. Or, if you side with the "Nietzche is right" side of the conversation "who will be or who may have come to be today or who recently came to be again". The former is eucatastrophic, the latter is dystopic.
“This may turn out to be the most consequential fact about all of history so far. It is possible that we will have superintelligence in a few thousand days (!); it may take longer, but I’m confident we’ll get there.”
I am a believer that people like sam are not lying. Anyone using these models daily probably believes the same. The o1 model, if prompted correctly, can architect a code base in a way that my decade+ of premium software experience cannot. Prompted incorrectly, it looks incompetent. The abilities of the future are already here, you just need to know how to use the models.
I'm using these models daily, and I don't believe that they're a direct path to superintelligence (unless you'd consider something like the printing press to have been a direct path to, say, the integrated circuit or the Internet).
> Prompted incorrectly, it looks incompetent. The abilities of the future are already here, you just need to know how to use the models.
Something purportedly intelligent shouldn't need "correct usage", as it should arguably be able to infer and clarify all ambiguities itself, no?
And I'm surprised how many people think they can confidently tell whether they're looking at an s-curve or an exponential function based on very limited data points. I don't even doubt that superintelligence is a very real possibility! But it might or might not happen, and if it does, it might or might not be based on deep learning.
As a counterexample: The maximum speed of travel for the average person for millenia used to be as fast as they could run, then it was as fast as the fastest horse can run, and then within a century it has accelerated to almost the speed of sound – at which it has plateaued.
Looking purely at the decades of acceleration, you might have very well concluded from the data that we'd be making significant headway towards getting within double-digit percentages of the speed of light at this point.
Maybe don't think of it as curves or functions. Just go through an LLM and think about all the things that could be improved. It's a long list once you get into the details. By and large...sota models are:
a) trained on crappy data, including questionable RLHF feedback.
b) trained with questionable embedding layers.
c) trained with questionable loss functions
d) trained with questionable optimizers
e) trained at questionable precision (somewhat related to d)
f) are very big which stops fast iteration around all the above.
It's kinda like semiconductors. You don't have to think of it as a curve - just ask people who are really close to them and they'll have a laundry list of stupid stuff which is currently done and will likely be improved upon over time.
You can't write a nice article about it. But you can talk about it with a simple "things are going to get a loooot better". The problem with thinking about progress in functions is it suggests some underlying law exists, and there isn't one.
Even if someone could point to a function and say "10x better" by 2030 - what does that even mean in the context of an LLM for example?
Interesting observation. My experience with o1 has been much more mundane. Sometimes I get the response I wanted, sometime it hallucinates, often times it writes buggy code. I've been experiencing this since ChatGPT was first released.
I actually originally wrote off the o1 model. Another thing I have found it's good at is finding bugs in a ton of code. Give it ten coding files, and a stack trace, it can find the bug.
Not really. Nobody expected China to produce as much solar power as it does now. What makes you think we know how much renewable energy we will produce in 2040?
Its sad for me to watch such comments.. Really. People always just focus on one particular issue/problem and try to solve it ignoring anything else. Open your eyes and look a bit broader view. Energy is NOT the only problem. Waste is another very serious one, usually ommited because.. Lets dump it to some 3th world countries and issue is gone right? Nope...
One should always evaluate why they feel such emotions. Is it because you want to or have a tendency towards doom/gloom?
> Energy is NOT the only problem. Waste is another very serious one
That's called moving the goal post. If you want to talk about waste from renewables you're more than welcome to but don't call us myopic for understanding that you can limit the scope of the discussion so that we are focused on the topic at hand.
There's always a group that's a step ahead ready to complain about the next goalpost.
First it's that we can't make enough renewable energy, then it's too expensive, then it's that we produce some amount of waste to create the renewable energy, then it's that we'll have so much energy we can't store it all, then it's that renewable energy is not public or free. At every step there's some issue that people like you want to point to as if to say we should just sit where we are born idle and do nothing, change nothing because it's not perfect or there are consequences.
The growth in renewables can not meet their demands, fusion is a moonshot that probably will not happen, nuclear takes too long to build. Where will the energy come from if not fossil fuels?
That means a ton of new power generation will need to come online. Last year > 85% of new utility scale power facilities was renewable in the US, solar is finally cheaper than natural gas.
The most likely outcome is that the new power demand will drive this cost down even further.
"humanity discovered an algorithm that could really, truly learn any distribution of data (or really, the underlying “rules” that produce any distribution of data)..."
This statement is manifestly untrue. Neural networks are useful, many hidden layers is useful, all of these architectures are useful, but the idea that they can learn anything, is based less on empirical results and more on what Sam Altman needs to convince people of to get this capital investments.
We also personify synapses and axons in human brain tissue, though. My point is, while I agree with your first sentence to a degree, we shouldn’t judge the whole solely by its elementary parts. Clearly an LLM exhibits very different behavior from a conventional database.
Wait that's it ? I guess humans also exhibit similar behavior to a search algorithm in certain instances. Nothing about LLM inference seems particularly similar to search even with our limited understanding.
All you're saying here is Input goes in > Output comes out. Well no shit.
I think LLM inference is extremely similar to search.
One of the killer features of LLMs is summarization (which can be thought of as searching a noisy set of data for the most relevant information) and document QA, which is also a search function.
Even implementation wise, transformers encode information in their weights and include that relevant information in their response.
Image generation models work the same way as google image search. Key word soup in -> relevant image out.
They encode information seen in their training within their weights then filter down those weight over many layers and what’s leftover is relevant data.
Idk how you’d look at how these models work and what they do and not see search.
> The entire vocabulary around machine learning is and always has been really weird.
I would argue it took a staggeringly weird turn around 2022/23. Machine learning has been around for a long time and only recently since OpenAI and it's slavish desire to harness true AI (which thanks to their horseshit now has to be called AGI) and Sam Altman in particular's delusional ramblings upon the topic that he clearly barely understands beyond it's ability to get his company fantastical amounts of capital has it truly gone off the rails.
It has to be. Most people don't understand the basic math involved and hence you can't explain it in concrete terms (neither what it's doing or how it's doing it) so you have to sort of make up analogies. It's an impossible task.
100%. If they could learn anything, then shouldn't modern ML systems be able to solve the big mysteries in science -- since we have large datasets there describing the phenomena in various ways? E.g. dark energy, dark matter, matter-antimatter asymmetry, or even outstanding problems in pure mathematics.
The intention of this sama post is as you said, it's to build narrative so he can raise his trillion from the Arab world or other problematic sources.
Well, you could certainly train a big-ass model to mimic the distribution of all that physics data. That doesn't mean the model could, eg, formulate interesting new theories which explain why that distribution has its particular structure.
> the idea that they can learn anything, is based less on empirical results and more on what Sam Altman needs to convince people of to get this capital investments.
Techbros love to pretend that they created digital gods (and by extension are gods themselves). We should all be thankful, worship, and of course surrender unconditionally -- Sam's will be done, amen.
Almost certainly true, and all the people crapping all over his description should really take a step back and consider that. He isn't out on some island all by himself here.
Universal approximation doesn't mean we've got (or ever will) algorithms to learn good enough models for any problem and the resources to run them, just that those models conceptually exist.
If you read what he wrote closely, he doesn't claim what you just claimed. Read it word for word:
"humanity discovered an algorithm that could really, truly learn any distribution of data (or really, the underlying “rules” that produce any distribution of data). To a shocking degree of precision, the more compute and data available, the better it gets at helping people solve hard problems"
He's an optimistic guy, no doubt, but he isn't full of shit.
Six months ago, he probably could have gotten away with saying this and there would have likely have been enough people who were still impressed enough with the trajectory of LLMs to back him on it. But these days, most of us have encountered the all-too-common failure mode where the LLM shows its hand, that it doesn't truly understand anything, and that it's just _very very good_ at prediction. Each new generation gets even better at that prediction, but still hits its weird stumbling points, because its still the same algorithm, and that algorithm cannot do what he is ascribing to it.
These are the words of a man who has an incredible amount of money sunk into something and as such, is having a really hard time taking an honest accounting of it.
1. What failure mode do LLMs have that proves they don't understand anything at all ? And why can't i prove the same with humans (who have an abundance of failure modes)
2. You genuinely think that a system whose goal is to predict the data it's given and continues to improve is limited in what it can learn ? Of all the shortcomings of the Transformer architecture, its objective function is not one of them.
> What failure mode do LLMs have that proves they don't understand anything at all ?
Try to get it to write something in a programming language not commonly used on the internet, say Forth or Brainfuck, with only the specifications of said languages. Humans are able to grasp the law of reality through a model and use it to act upon the real world.
> You genuinely think that a system whose goal is to predict the data it's given and continues to improve is limited in what it can learn?
Not GP, but Image generators have ingested more images that I've seen in my life and still can't grasp basic things like perspective or anatomy. Things that people can learn from a book or two. And there are software that already have models for both.
>Try to get it to write something in a programming language not commonly used on the internet, say Forth or Brainfuck, with only the specifications of said languages. Humans are able to grasp the law of reality through a model and use it to act upon the real world.
My Experience with this has been SOTA LLMs generating sensible code at rates much greater than random chance even if it may not be as good as i'd like. I don't see how that is evidence LLMs don't understand anything at all especially since there are probably humans who would write less workable code.
>Not GP, but Image generators have ingested more images that I've seen in my life and still can't grasp basic things like perspective or anatomy.
The human brain didn't poof from thin air. It's the result of billions of years of evolution tuning it for real world navigation and vision amongst other things. You are not a blank slate. All Modern NNs are much more blank slate than the brain has been for at least millions of years.
You're moving the bars. In fact, these bars are so laughably low, I don't know that we're having the same conversation any more.
Nobody's saying it can't write "sensible code at rates much greater than random chance." We're not competing with an army of typing monkeys here. We're saying it actually doesn't "know" anything, and regularly demonstrates that quality, despite it seeming very much like something that knows things, most of the time. You're being tricked by a clever algorithm.
> All Modern NNs are much more blank slate than the brain has been for at least millions of years.
All well and good if we were talking about interesting research and had millions of years to let these algorithms prove themselves out, I suppose. But we're talking about industries that are being created out of whole cloth and/or destroyed, depending on where you stand, and the time frame is in single-digit years, if not less. And these things will still confidently make elementary mistakes and get lost in their own context.
Look, they're obviously not useless, but they're a tool with weaknesses and strengths. And people like pg who are acting like there ARE no weaknesses, or that a simple application of will and money will erase them, they are selling us a bill of goods.
Yeah and I'm saying this is a nonsense statement if you can't create a test (one that would also not disqualify humans) that demonstrates this. If you are saying what LLMs do is "fake understanding" then "fake understanding" should be testable unless you're just making stuff up.
>All well and good if we were talking about interesting research and had millions of years to let these algorithms prove themselves out, I suppose
Did you even read what the commenter I replied to was saying. This is irrelevant. We don't need to wait millions of years for anything.
Come on now. This description is basically the universal approximation theorem. He isn't just making stuff up. You can take issue with the theorem and have a debate around it, but he isn't just wildly off base making stuff up here.
It's a layman commentary on the Universal Approximation Theory so it is true.
The problem with the UAT is that it's never said anything about how trivial such an exercise would be. But he obviously believes we've stumbled on the architecture to get us there (for problems we care about anyway)
But… he’s saying something here that is academically true: that neural networks can approximate any possible function, to any arbitrary degree of precision you require (given infinite capacity / depth).
I will highlight one thing, which is that the theorem does not say anything about it being practical to learn this function, given available data or any specific optimization technique.
'the underlying “rules” that produce any distribution of data...' is clearly meant to convince the reader that it can produce something we would describe as a "rule", that is, a coherent and comprehensible regulating principle. This isn't just because he isn't being precise enough; he quite clearly wants the reader to understand this as neural networks being able to create a mental model of anything, in a manner similar to how a biological neural network would.
It doesn't, it can't, and it won't in our lifetimes.
446 comments
[ 2.5 ms ] story [ 306 ms ] threadSam Altman is the last guy we want helping lead that revolution.
Paging Dr. Bullshit, we've got an optimist on the line who'd like to have a word with you.
With the current hype wave it feels like we’re almost there but this piece makes me think we’re not.
[0]: From GPT-4 to AGI: Counting the OOMs https://situational-awareness.ai/from-gpt-4-to-agi/
[0] https://www.wired.com/story/openai-ceo-sam-altman-the-age-of...
[1] https://en.wikipedia.org/wiki/GPT-4#Training
Right now there is insane amounts of money being thrown at AI because progress is matching projections. There doesn't seem to be a leveling off or diminishing returns taking place. And that's just compute, we could probably freeze compute and still make insane progress just because optimizations have so much momentum right now too.
Does anyone know if he's published thoughts on any serious lit? So far I've just seen him play the "I know stuff you don't because I get to see behind the scenes" card over and over, which seems a little dubious at this point. I was convinced they would announce AGI in December 2023, so I'm far from a hater! It just seems clear that they're/he's guessing at this point, rather than reporting or reasoning.
Really he assumes two huge breakthroughs, both of which I find plausible but far from guaranteed:
The entire AI trend - long term is based on the idea that AI will profoundly change the world. This has sparked a global race for developing better AI systems and the more dangerous winner takes all outcome. It is therefore not surprising that billions of dollars are being spent to develop more powerful AI systems as well as to restructure operations around them.
All the existing systems we have must fundamentally change for the better if we want a good future.
The positive aspects / utopia promises have much more visibility to the public than the negative effects / dystopian world.
ARE WE TO pretend that Human greed, selfishness, desires to dominate and control, animalistic behaviour, use of technologies for war and other destructive purposes don't exist?
We are living in times of war and chaos and uncertainty. Increasingly advanced technology is being used on the battlefield in more covert and strategic ways.
History is repeating itself again in many ways. Have we failed to learn? The consequences might be harsher with more advanced technology.
I have read and thought deeply about several anti AI doomer takes from prominent researchers and scientists but I haven't seen any which aren't based on assumptions or foolproof. For something that profoundly changes the world, it's bad to base your hopes on assumptions.
I see people dunking on llms which might not be AI's final form. Then they extrapolate that and say there is nothing to worry about. It is a matter of when not if.
The thought of being useless or worse being treated as nothing more than pests is worrying. Job losses are minor in comparison.
The only hope I have is that we are all in this together. I hope peace and goodwill prevails. I hope necessary actions are taken before it's too late.
A more pragmatic perspective indicates that there are more pressing problems that need to be addressed if we want to avoid a doomer scenario.
My take:
* Foom/doom isn't helpful. But, calm cautiousness is. If you're acting from a place of fear and emotional dysregulation, you'll make ineffective choices. If you get calm and regulated first, and then take actions, they'll be more effective. (This is my issue with AGI-risk people, they often seem triggered/fear/alarm-driven rather than calm but cautious)
* Piece is kind of a manifesto for raising money for AI infra
* Sam's done a podcast before about meditation where he talked about similar themes of "prudence without fear" and the dangers of "deep fear and panic and anxiety" and instead the importance of staying "calm and centered during hard and stressful moments" - responding, not reacting (strong +1)
* It's no accident that o1 is very good at math physics and programming. It'll keep getting much better here. Presumably this is the path for AGI to lead to abundance and cheaper energy by "solving physics"
1. If you honestly think that millions/billions of people are at serious risk of avoidable harm that everyone else is ignoring, "calm down" can be a hard dictum to follow. Sam Altman has won, it's easy for him psychologically to say "well, lets just stick to the status quo and do our best every day and it'll probably work out". Made-in-house bias is strongest when "in-house" is your own mind, after all.
2. Your scare quotes makes it seem like you might agree, but: physics is the study of the physical world, thinking it can be 'solved' is like thinking mathematics, psychology, or anthropology can be 'solved'. It's fundamentally anti-science and very, very dangerous to be talking like that. Truth isn't absolutely relative, but science also isn't a collection of facts written in stone that we need to finish unearthing; it's a collection of intellectual tools.
Could anyone elaborate on this? Further down he talks about the necessity of bringing the cost of computing down. Is that really the bottleneck?
https://splab.sdu.edu.cn/GPT3.pdf
While it's true there are a lot of jobs obsoleted by technological progress, the vision of personal AI teams creating a new age of prosperity only makes sense for knowledge workers. Sure, a field worker picking cabbage could also have an AI team to coordinate medical care. But in this brilliant future, are the lowest members of society suddenly well-paid?
The steam engine and subsequent Industrial Revolution created a lot of jobs and economic productivity, sure, but a huge amount of those jobs were dirty, dangerous factory jobs, and the lion's share of the productivity was ultimately captured by robber barons for quite some time. The increase in standard of living could only be seen in aggregate on pages of statistics from the mahogany-paneled offices of Standard Oil, while the lives of the individuals beneath those papers more often resembled Sinclair's Jungle.
Altman's suggestion that avoiding AI capture by the rich merely requires more compute is laughable. We have enormous amounts of compute currently, and its productivity is already captured by a small number of people compared to the vast throngs that power civilization in total. Why would AI make this any different? The average person does not understand how AI works and does not have the resources to utilize it. Any further advancements in AI, including "personalized AI teams," will not be equally shared, they will be packaged into subscription services and sold, only to enrich those who already control the vast majority of the world's wealth.
We need to sell the idea of abundance for everyone so investors and employees will feel good about dedicating their livelihood to our organization!
i think this is the prevailing wisdom but theres an angle that openai doesnt value and therefore isnt mentioned. There's far more compute sitting idle in everyone's offices and homes and pockets than there are in the $100bn openai cluster. it just isnt useful for training because physics. but its useful for inference. local LLMs ship this-next year in Chrome (gemini nano) and Apple (apple intelligence) that will truly be available for everyone instead of going thru OpenAI's infra. they'll be worse than GPT4, but only for a couple more years.
> As one example, we expect that this technology can cause a significant change in labor markets (good and bad) in the coming years, but most jobs will change more slowly than most people think, and I have no fear that we’ll run out of things to do (even if they don’t look like “real jobs” to us today). People have an innate desire to create and to be useful to each other, and AI will allow us to amplify our own abilities like never before. As a society, we will be back in an expanding world, and we can again focus on playing positive-sum games.
It's very easy as an extremely rich person to just say, "don't worry, in the end it'll be better for all of us." Maybe that's true on a societal scale, but these are people's entire worlds being destroyed.
Imagine you went to college for a medical specialty for 8-10 years, you come out as an expert, and 2 years later that entire field is handled by AI and salaries start to tank. Imagine you have been a graphic designer for 20 years supporting your 3 children and bam a diffusion model can do your job for a fraction of the cost. Imagine you've been a stenographer working in courtrooms to support your ill parents and suddenly ASR can do your job better than you can. This is just simple stuff we can connect the dots on now. There will be orders of magnitude more shifts that we can't even imagine right now.
To someone like Sam, everything will be fine. He can handle the massive societal shift because he has options. Every a moderately wealthy person will be OK.
But the entire middle class is going to start really freaking the fuck out soon as more and more jobs disappear. You're already seeing anti-AI sentiment all over the web. Even in expert circles, you can see skepticism. People saying things like, "how do I opt out of Apple Intelligence?" People don't WANT more grammar correction or AI emojis in their lives, they just want to survive and thrive and own a house.
How are we going to handle this? Sam's words of "if we could fast-forward a hundred years from today, the prosperity all around us would feel just as unimaginable" doesn't mean shit to a family of 4 who went through layoffs in the year 2025 because AI took their job while Microsoft's stock grows 50%.
When o1 was released, I ran an internal eval and saw it plainly outperforming our highly educated colleagues. I had goosebumps, and haven’t been able to sleep well for days. This will dramatically impact society in 2-5 years.
Do you know of any relevant material related to this?
https://intelligence.org/files/IEM.pdf
Welcome to the anxiety party, it sucks in here. As someone who's been working on AI theory full time for ~1 year, I desperately wish we could go back to the days of my faraway youth (5 years ago) before intuition was cracked on accident by spellcheck algorithms. I agree with him that it holds the key to massive prosperity, but selfishly, it's gonna upend my life and the lives of everyone I love. Already has for me, as I grapple with how to (ethically) pay rent while spending all day lighting the Warning Beacons of Gondor...
The only real answer, IMHO, is to vote for political systems that put control of society (and AI) in the hands of the public. Call it socialism, call it Georgism, call it anarcho-free-market-space-communism, call it whatever you want; there's no way that "a tiny number of people have immense inherited power" (capitalism) and "people fundamentally understand themselves as members of a tribe put in opposition to all other tribes by default" (nationalism) mesh well with an intelligence explosion.
Here's to hoping the haters are right, and we all turn out to be wrong! I'll be thrilled if Sam Altman is just a rich company leader in 10 years, and intuitive algorithms are still confined to direct usage (chatbots).
The reality is, this transition is going to be painful for the average person.
with the billionaire AI moguls taking the role of the french kings
and the data centres taking the role of the palaces
Surprisingly complicated HTML source code for a simple blog post.
Here it is as:
Plain HTML: https://hub.scroll.pub/sama/index.html
Text: https://hub.scroll.pub/sama/index.txt
Scroll: https://hub.scroll.pub/sama/index.scroll
I'm personally rooting for cognitive being the word of the next few decades, but that's just a shout from the sidelines. Only time will tell what humanity latches on to, but I wouldn't be surprised if this blog post/subdomain was referenced in a Wikipedia page's Etymology section in 10-15 years...
Although this blog post & discussion has my anxiety at an 8, something's oddly comforting about the thought of Sam Altman fiddling with tailwind classes to get his profound aesthetic just-right. Something undeniably relatable and human. Hate the man all you want (I do!), but he's clearly acting in some sort of good faith.
You are right. I overlooked the simplicity of the headline. Thanks for calling attention to that.
> Something tells me he rolled this himself.
This would be cool.
> I'm personally rooting for cognitive being the word of the next few decades
I like that one too.
> he's clearly acting in some sort of good faith.
He's always been one to think and write for himself. Huge respect for him. Even though it needles me every moment that they still call themselves "Open"AI, I have so much respect for the guy, especially because PG basically told the world he was the next Michael Jordan of startups, and he actually went and fulfilled that. Not many people have it in them to live up to hype like that (Lebron being the only other one I can think of OOTOMH)
I'm not an AI skeptic at all, I use llms all the time, and find them very useful. But stuff like this makes me very skeptical of the people who are making and selling AI.
It seems like there was a really sweet spot wrt the capabilities AI was able to "unlock" with scale over the last couple years, but my high level sense is that each meaningful jumps of baseline raw "intelligence" required an exponential increases in scale, in terms of training data and computation, and we've reached the ceiling of "easily available" increases, it's not as easy to pour "as much as it takes" into GPT5 if it turns out you need more than A Microsoft.
This is the part that really gets me. This is a thing that you say to your team, and a thing you say to your investors, but this isn't a thing that you can actually believe with certainty is it?
At a minimum, we could ask for the evidence.
Imagine the Uranium Committee of early 40's, where Szilard and others were babbling about 10kg of some magical metal exploding briefly with the power of a sun, with the best evidence being some funky trail in an alcohol vapor chamber.
Maybe sama is right, maybe not, but the absence of evidence is not evidence of absence.
Unless the human brain is made of some sacred substance, the worst-case scenario is that we will extrapolate current scanning methods into the future and run the scanned model in silica. I'm not recommending this "just for fun," but the laws of physics don't forbid it.
So is Sam Altman the modern day alchemist? Making predictions based on faulty methods and faulty understanding (per your gold example)?
What will happen is that we'll shift the economy around based on inflated tech promises and ruin people's lives. No big deal I guess.
Alchemists were early scientists who later branched into fields like chemistry, mathematics, and physics (Newton explored alchemy).
Altman leads a team of experts in neural networks, programming, and HW design. While he might be mistaken, dismissing him outright is difficult.
When we are successfully turning base metals into gold, hit me up.
"Deep learning works, and we will solve the remaining problems."
"It is possible that we will have superintelligence in a few thousand days (!)"
Technically a few thousand days covers quite a range. 20 thousand is 55 years.
On the Kurzweil graph, extrapolating hardware progress from 1900 through 2000, superintelligence seems to be roughly 2035, depending on how you define things. https://www.researchgate.net/figure/Kurzweils-8-71-chart-of-...
And here we are today where nuclear energy is limited to nuclear weapons, a small set of military vehicles and <10% of the world's electricity production. Not nothing, sure, but nothing like past predictions either.
The toothpaste and similar products were pretty ill advised, vaseline and uranium glass are still collectable and are seeing a ressurrence of new interest: https://old.reddit.com/r/uraniumglass/
5 years ago, I wouldn't have believed any of what exists today. I saw internal demos that showed 2nd or 3rd grade reading comprehension in 2017 and statements were made about how in the next decade, we will probably reach college level comprehension. We have come so far beyond that in less than half the time. Technology isn't about scaling incrementally and continuing on the same path using the same principles we know today. It's about disruption that felt impossible before - that feels like a constant to me now. Seeing everything I've seen in the last 20 years, it's going to continue to happen. We just can't see it yet.
If models genuinely keep making similar leaps each generation then we're still a few generations before "More than a Microsoft".
That said, it's ultimately up to the people footing the bill isn't it ?
https://www.bloomberg.com/news/articles/2024-09-20/microsoft...
Over the last, say, five years, a pile of 50+ year problems have been toppled by the deep learning + data + compute combo. This includes language modeling (divorced from reasoning), image generation, audio generation, audio separation, image segmentation, protein folding, and so on.
(Audio separation is particularly close to my heart; the 'cocktail party problem' has been a challenge in audio processing for 100+ years, and we now have great unsupervised separation algorithms (MixIT), which hardly anyone knows about. That's an indicator of how much great stuff is happening right now.)
So, when we look at some of our known 'big' problems in AI/ML, we ask, 'what's the horizon for figuring this out?' Let's look at reasoning...
We know how to do 'reasoning' with GOFAI, and we've got interesting grafts of LLMs+GOFAI for some specific problems (like the game of Diplomacy, or some of the math olympiad solvers).
"LLMs which can reason" is a problem which has only been open for a year or two tops, and which we're already seeing some interesting progress on. Either there's something special about the problem which will make it take another 50+ years to solve, or there's nothing special about it and people will cook up good and increasingly convenient solutions over the next five years or so. (Perhaps a middle ground is 'it works but takes so much compute that we have to wait for new materials science for chip making to catch up.')
Despite what skeptics have been saying for decades, Moore's Law is alive and well - and we haven't even figured out how to stack wafers in 3 dimensions yet!
And well then there's going beyond just text. Current multimodal models are basically patchwork bullshit, separately trained image/audio to text/embeddings encoders slapped onto an existing model and hoping it does something neat. Tokenization and samplers are likewise bullshit that's there to compensate for lack of training compute. Once we have enough to be able to brute force it properly with bytes in, bytes out, regardless of data format, the results should be way better.
And granted a lot of parts of the human brain are dedicated to specific jobs that are entirely nonexistent in a normal LLM (kinematics, vision, sound, touch, taste, smell, autonomic organ control) so the actual bit we should be comparing for just language and reasoning is way smaller. Still the brain is pretty efficient inference energy wise, it's like the ultimate mixture of experts, extreme amounts of sparse storage and most of it is not computed until needed. The router must be pretty good.
Until you have AGI you can't say this since until then we don't know how much the different parts costs to replace with AI systems.
It's something that gives me pause on the idea that we have to build many GW of power stations. It may be possible to get much more energy efficient AI via better algorithms.
Meanwhile the best class AI is trained on the output of human intelligence. When AI can learn by itself in the same way humans do (or even more efficient ways) that's when we can say human intelligence has been surpassed. Until then it's just tools for humans to use to augment their intelligence.
Analog systems are not known for being very precise- they're noisy, signals get corrupted easily- and that's why we prefer digital ones. As soon as we had the technology, we switched everything we could- audio and video recording, telephone calls, photography, to a digital medium. This makes me wonder if the seemingly extraordinary efficiency of artificial neural networks is simply due to the precision with which they can be trained.
If straightforward binary signaling was so universally superior, I think the worldwide network of over a quintillion ruthlessly self-replicating nanobots would be using a much more heavily after the last billion years.
There's also an interesting bit I've observed with LLMs, quantization in range of 4-8 bits doesn't reduce performance as much as it mainly reduces consistency. If you generate an answer a bunch of times and take the average you'll end up with roughly the same result as an fp16/32 would do every time. In nature, being inconsistent is usually weighed negatively with death... so that's probably why it hasn't caught on even if it is more efficient. Or this is enough of a different abstraction that we can't draw parallels anyway.
What is there to be skeptical of? OpenAI made their current product using a 10G$ investment plus a few they are not disclosing, and now they will start to do it at scale.
Perfectly normal stuff.
By the way, what's the World's GDP again?
Like everything else being sold, the marketing is 95% BS.
LLMs are amazing and wonderful tools, but me thinks we’re near a plateau in capability. Now investors are pumping for ROI before that becomes evident.
After we reach the plateau in capabilities, the next phase is cutting production and operating costs to maximize margins.
I’m the meantime, expect the marketing to get increasingly cringe until the bubble bursts.
This seems to be the key of the piece to me. It's his manifesto for raising money for the infra side of things. And, it resonates: I don't want ASI to only be affordable to the ultra rich.
Billionaires could make so much change happen but instead they are building bunkers and riding giant dicks into space while simultanously touting that they are looking out for humanity.
Not really. The problem is that learning requires scale, mostly of data. That scale places AI providers at the nexus of value, with OpenAI as the presumptive market organizer and leader. Reducing compute costs would just mean they can capture more of the value. Data costs orders of magnitudes more than compute because it requires curation, so even if individual developers could get compute, they can't get data, so size/access matters.
That's good for this community and could be good for the state of the art and the overall potential contributions of AI. And more paying customers could be good for OpenAI. But it won't put AI in front of non-paying customers/developers, unless their value is otherwise harvested.
> I don't want ASI to only be affordable to the ultra rich.
As a developer or consumer?
And you don't mean it's ok if AI is only affordable for the moderately rich, do you? I agree it's hard to state which developers/people/customers should be subsidized. Generally we subsidize education but not profit or war. Sometimes culture. Companies will subsidize complementary goods and input factors. Otherwise? Not much history of benevolent subsidy.
AI has as much potential to shape society as freeways and the automobile did in the US, but few understand how, and I've seen no plans on point.
With electric energy networks and transportation, the central government has a role in reducing hostaging by hold-out's. With education, states have an incentive to attract and build talent (albeit now reduced with trans-national outsourcing and remote work). But otherwise, it's private enterprise and resource-weighted customers.
Changing that is not really Sam Altman's job. His job is to deliver that value, sooner rather than later. Most would be uncomfortable with AI overlords expressing opinions on cultural values or economic distributions to be imposed.
More of a "everything's fine, nothing to worry about".
While, there is already job disruption, and widespread misinformation.
It isn't in some future, it is already happening.
Reminds me of these quotes from Sam on this podcast episode (https://www.youtube.com/watch?v=KfuVSg-VJxE)
* "Prudence without fear" (Sam referencing another quote)
* "if you create the descendants of humanity from a place of, deep fear and panic and anxiety, that seems to me you're likely to make some very bad choices or certainly not reflect the best of humanity."
* "the ability to sort of like, stay calm and centered during hard and stressful moments, and to make decisions that are where you're not too reactive"
There isn't a single ordinary person at the table, no labour unions, no political parties or anything democratic. It's Microsoft, it's Google, it's some venture-capital owned French firm, some venture-capital owned German firm, etc. Maybe if Schmidhuber's stuff is good as he hopes it is maybe there'll be an Austrian firm, etc., but mostly, it'll be a capital intensive business controlled by people with capital.
This line of reasoning doesn't hold for me, as you could apply it to any technology, including ones actually very likely to destroy human civilization.
Sometimes, not building a given thing at all is better than building it with even the best intentions.
I'm personally not sure on which side AI falls, but denying that such things exist at all seems intellectually dishonest.
There's a good reason we get pessimists to design safety-critical systems!
AI is a side-show.
Intelligence is ambient in living tissue, so we already have as much intelligence as is adaptive. We don't need more. As talking apes made out of soggy mud wrapped around calcium twigs living in the greasy layer between hard vacuum and a droplet of lava which in turn is orbiting a puddle of hydrogen in the hem of the skirt of a black hole our problems are just not that complicated.
Heck, we are surrounded by four-billion year-old self-improving nanotechnology that automatically provides almost all our physical needs. It's even solar-powered! The whole life-support system was fully automatic until we fucked it up in our ignorance. But we're no longer ignorant, eh?
The vast majority of our problems today are the result of our incredible resounding success. We have the solutions we need. Most of them were developed in the 1970's when the oil got expensive for a few minutes.
Must we boil the oceans just to have a talking computer tell us to get on with it? Can't we just do like the Wizard of Oz? Have a guy in a box with a voice changer and a fancy light show tell us to "love each other"? Mechanical Turk God? We can use holograms.
I thought I would get reamed for farting in church but it seems to have gone over well.
See ya Space Cowboy
It absolutely can be answered, but only the the intender. Who is and who was and who is to come. Or, if you side with the "Nietzche is right" side of the conversation "who will be or who may have come to be today or who recently came to be again". The former is eucatastrophic, the latter is dystopic.
I am a believer that people like sam are not lying. Anyone using these models daily probably believes the same. The o1 model, if prompted correctly, can architect a code base in a way that my decade+ of premium software experience cannot. Prompted incorrectly, it looks incompetent. The abilities of the future are already here, you just need to know how to use the models.
> Prompted incorrectly, it looks incompetent. The abilities of the future are already here, you just need to know how to use the models.
Something purportedly intelligent shouldn't need "correct usage", as it should arguably be able to infer and clarify all ambiguities itself, no?
As a counterexample: The maximum speed of travel for the average person for millenia used to be as fast as they could run, then it was as fast as the fastest horse can run, and then within a century it has accelerated to almost the speed of sound – at which it has plateaued.
Looking purely at the decades of acceleration, you might have very well concluded from the data that we'd be making significant headway towards getting within double-digit percentages of the speed of light at this point.
a) trained on crappy data, including questionable RLHF feedback. b) trained with questionable embedding layers. c) trained with questionable loss functions d) trained with questionable optimizers e) trained at questionable precision (somewhat related to d) f) are very big which stops fast iteration around all the above.
It's kinda like semiconductors. You don't have to think of it as a curve - just ask people who are really close to them and they'll have a laundry list of stupid stuff which is currently done and will likely be improved upon over time.
But when talking about future growth potential, I don't think you can get around making assumptions about the shape of the growth function.
Even if someone could point to a function and say "10x better" by 2030 - what does that even mean in the context of an LLM for example?
One should always evaluate why they feel such emotions. Is it because you want to or have a tendency towards doom/gloom?
> Energy is NOT the only problem. Waste is another very serious one
That's called moving the goal post. If you want to talk about waste from renewables you're more than welcome to but don't call us myopic for understanding that you can limit the scope of the discussion so that we are focused on the topic at hand.
There's always a group that's a step ahead ready to complain about the next goalpost. First it's that we can't make enough renewable energy, then it's too expensive, then it's that we produce some amount of waste to create the renewable energy, then it's that we'll have so much energy we can't store it all, then it's that renewable energy is not public or free. At every step there's some issue that people like you want to point to as if to say we should just sit where we are born idle and do nothing, change nothing because it's not perfect or there are consequences.
That means a ton of new power generation will need to come online. Last year > 85% of new utility scale power facilities was renewable in the US, solar is finally cheaper than natural gas.
The most likely outcome is that the new power demand will drive this cost down even further.
This statement is manifestly untrue. Neural networks are useful, many hidden layers is useful, all of these architectures are useful, but the idea that they can learn anything, is based less on empirical results and more on what Sam Altman needs to convince people of to get this capital investments.
We don’t personify database interactions the same way we personify setting weights in a neural network.
Text query in -> relevant text out.
I don’t say that search algorithms “learn” or “think” outside of ML.
Wait that's it ? I guess humans also exhibit similar behavior to a search algorithm in certain instances. Nothing about LLM inference seems particularly similar to search even with our limited understanding.
All you're saying here is Input goes in > Output comes out. Well no shit.
One of the killer features of LLMs is summarization (which can be thought of as searching a noisy set of data for the most relevant information) and document QA, which is also a search function.
Even implementation wise, transformers encode information in their weights and include that relevant information in their response.
Image generation models work the same way as google image search. Key word soup in -> relevant image out.
They encode information seen in their training within their weights then filter down those weight over many layers and what’s leftover is relevant data.
Idk how you’d look at how these models work and what they do and not see search.
Nobody says the algo that chatgpt was made with learned information which it encoded into weights, even if that’s technically correct.
I would argue it took a staggeringly weird turn around 2022/23. Machine learning has been around for a long time and only recently since OpenAI and it's slavish desire to harness true AI (which thanks to their horseshit now has to be called AGI) and Sam Altman in particular's delusional ramblings upon the topic that he clearly barely understands beyond it's ability to get his company fantastical amounts of capital has it truly gone off the rails.
I cannot wait to watch this bubble pop.
“Neural networks” “learned” data by being “trained” since they were first described in the late 1900s.
The same language was used in Ian Goodfellow’s (excellent) 2012 text book “Deep Learning”
The intention of this sama post is as you said, it's to build narrative so he can raise his trillion from the Arab world or other problematic sources.
In pseudocode, this is Sam Altman:
while(alive) { RaiseCapital() }
Techbros love to pretend that they created digital gods (and by extension are gods themselves). We should all be thankful, worship, and of course surrender unconditionally -- Sam's will be done, amen.
"humanity discovered an algorithm that could really, truly learn any distribution of data (or really, the underlying “rules” that produce any distribution of data). To a shocking degree of precision, the more compute and data available, the better it gets at helping people solve hard problems"
He's an optimistic guy, no doubt, but he isn't full of shit.
Six months ago, he probably could have gotten away with saying this and there would have likely have been enough people who were still impressed enough with the trajectory of LLMs to back him on it. But these days, most of us have encountered the all-too-common failure mode where the LLM shows its hand, that it doesn't truly understand anything, and that it's just _very very good_ at prediction. Each new generation gets even better at that prediction, but still hits its weird stumbling points, because its still the same algorithm, and that algorithm cannot do what he is ascribing to it.
These are the words of a man who has an incredible amount of money sunk into something and as such, is having a really hard time taking an honest accounting of it.
2. You genuinely think that a system whose goal is to predict the data it's given and continues to improve is limited in what it can learn ? Of all the shortcomings of the Transformer architecture, its objective function is not one of them.
Try to get it to write something in a programming language not commonly used on the internet, say Forth or Brainfuck, with only the specifications of said languages. Humans are able to grasp the law of reality through a model and use it to act upon the real world.
> You genuinely think that a system whose goal is to predict the data it's given and continues to improve is limited in what it can learn?
Not GP, but Image generators have ingested more images that I've seen in my life and still can't grasp basic things like perspective or anatomy. Things that people can learn from a book or two. And there are software that already have models for both.
My Experience with this has been SOTA LLMs generating sensible code at rates much greater than random chance even if it may not be as good as i'd like. I don't see how that is evidence LLMs don't understand anything at all especially since there are probably humans who would write less workable code.
>Not GP, but Image generators have ingested more images that I've seen in my life and still can't grasp basic things like perspective or anatomy.
The human brain didn't poof from thin air. It's the result of billions of years of evolution tuning it for real world navigation and vision amongst other things. You are not a blank slate. All Modern NNs are much more blank slate than the brain has been for at least millions of years.
Nobody's saying it can't write "sensible code at rates much greater than random chance." We're not competing with an army of typing monkeys here. We're saying it actually doesn't "know" anything, and regularly demonstrates that quality, despite it seeming very much like something that knows things, most of the time. You're being tricked by a clever algorithm.
> All Modern NNs are much more blank slate than the brain has been for at least millions of years.
All well and good if we were talking about interesting research and had millions of years to let these algorithms prove themselves out, I suppose. But we're talking about industries that are being created out of whole cloth and/or destroyed, depending on where you stand, and the time frame is in single-digit years, if not less. And these things will still confidently make elementary mistakes and get lost in their own context.
Look, they're obviously not useless, but they're a tool with weaknesses and strengths. And people like pg who are acting like there ARE no weaknesses, or that a simple application of will and money will erase them, they are selling us a bill of goods.
Yeah and I'm saying this is a nonsense statement if you can't create a test (one that would also not disqualify humans) that demonstrates this. If you are saying what LLMs do is "fake understanding" then "fake understanding" should be testable unless you're just making stuff up.
>All well and good if we were talking about interesting research and had millions of years to let these algorithms prove themselves out, I suppose
Did you even read what the commenter I replied to was saying. This is irrelevant. We don't need to wait millions of years for anything.
The problem with the UAT is that it's never said anything about how trivial such an exercise would be. But he obviously believes we've stumbled on the architecture to get us there (for problems we care about anyway)
https://en.m.wikipedia.org/wiki/Universal_approximation_theo...
I will highlight one thing, which is that the theorem does not say anything about it being practical to learn this function, given available data or any specific optimization technique.
It doesn't, it can't, and it won't in our lifetimes.