Feels reasonable in the first few paragraphs, then quickly starts reading like science fiction.
Would love to read a perspective examining "what is the slowest reasonable pace of development we could expect." This feels to me like the fastest (unreasonable) trajectory we could expect.
Like an exponentially growing compute requirement for negligible performance gains, on the scale of the energy consumption of small countries? Because that is where we are, right now.
Even if this were true, it's not quite the end of the story is it? The hype itself creates lots of compute and to some extent the power needed to feed that compute, even if approximately zero of the hype pans out. So an interesting question becomes.. what happens with all the excess? Sure it probably gets gobbled up in crypto ponzi schemes, but I guess we can try to be optimistic. IDK, maybe we get to solve cancer and climate change anyway, not with fancy new AGI, but merely with some new ability to cheaply crunch numbers for boring old school ODEs.
Ok, I'll bite. I predict that everything in this article is horse manure. AGI will not happen. LLMs will be tools, that can automate away stuff, like today and they will get slightly, or quite a bit better at it. That will be all.
See you in two years, I'm excited what will be the truth.
That seems naive in a status quo bias way to me. Why and where do you expect AI progress to stop? It sounds like somewhere very close to where we are at in your eyes. Why do you think there won't be many further improvements?
I write bog-standard PHP software. When GPT-4 came out, I was very frightened that my job could be automated away soon, because for PHP/Laravel/MySQL there must exist a lot of training data.
The reality now is, that the current LLMs still often create stuff, that costs me more time to fix, than to do it myself. So I still write a lot of code myself. It is very impressive, that I can think about stopping writing code myself. But my job as a software developer is, very, very secure.
LLMs are very unable to build maintainable software. They are unable to understand what humans want and what the codebase need. The stuff they build is good-looking garbage. One example I've seen yesterday: one dev committed code, where the LLM created 50 lines of React code, complete with all those useless comments and for good measure a setTimeout() for something that should be one HTML DIV with two tailwind classes. They can't write idiomatic code, because they write code, that they were prompted for.
Almost daily I get code, commit messages, and even issue discussions that are clearly AI-generated. And it costs me time to deal with good-looking but useless content.
To be honest, I hope that LLMs get better soon. Because right now, we are in an annoying phase, where software developers bog me down with AI-generated stuff. It just looks good but doesn't help writing usable software, that can be deployed in production.
To get to this point, LLMs need to get maybe a hundred times faster, maybe a thousand or ten thousand times. They need a much bigger context window. Then they can have an inner dialogue, where they really "understand" how some feature should be built in a given codebase. That would be very useful. But it will also use so much energy that I doubt that it will be cheaper to let a LLM do those "thinking" parts over, and over again instead of paying a human to build the software. Perhaps this will be feasible in five or eight years. But not two.
And this won't be AGI. This will still be a very, very fast stochastic parrot.
ahofmann didn't expect AI progress to stop. They expected it to continue, but not lead to AGI, that will not lead to superintelligence, that will not lead to a self-accelerating process of improvement.
So the question is, do you think the current road leads to AGI? How far down the road is it? As far as I can see, there is not a "status quo bias" answer to those questions.
It seems to me that much of recent AI progress has not changed the fundamental scaling principles underlying the tech. Reasoning models are more effective, but at the cost of more computation: it's more for more, not more for less. The logarithmic relationship between model resources and model quality (as Altman himself has characterized it), phrased a different way, means that you need exponentially more energy and resources for each marginal increase in capabilities. GPT-4.5 is unimpressive in comparison to GPT-4, and at least from the outside it seems like it cost an awful lot of money. Maybe GPT-5 is slightly less unimpressive and significantly more expensive: is that the through-line that will lead to the singularity?
Compare the automobile. Automobiles today are a lot nicer than they were 50 years ago, and a lot more efficient. Does that mean cars that never need fuel or recharging are coming soon, just because the trend has been higher efficiency? No, because the fundamental physical realities of drag still limit efficiency. Moreover, it turns out that making 100% efficient engines with 100% efficient regenerative brakes is really hard, and "just throw more research at it" isn't a silver bullet. That's not "there won't be many future improvements", but it is "those future improvements probably won't be any bigger than the jump from GPT-3 to o1, which does not extrapolate to what OP claims their models will do in 2027."
AI in 2027 might be the metaphorical brand-new Lexus to today's beat-up Kia. That doesn't mean it will drive ten times faster, or take ten times less fuel. Even if high-end cars can be significantly more efficient than what average people drive, that doesn't mean the extra expense is actually worth it.
Why would it get 60-80% as good as human programmers (which is what the current state of things feels like to me, as a programmer, using these tools for hours every day), but stop there?
Because ewe still haven't figured out fusion but its been promised for decades. Why would everything thats been promised by people with highly vested interests pan out any different?
One is inherently a more challenging physics problem.
It's 60-80% as good as Stack Overflow copy-pasting programmers, sure, but those programmers were already providing questionable value.
It's nowhere near as good as someone actually building and maintaining systems. It's barely able to vomit out an MVP and it's almost never capable of making a meaningful change to that MVP.
If your experiences have been different that's fine, but in my day job I am spending more and more time just fixing crappy LLM code produced and merged by STAFF engineers. I really don't see that changing any time soon.
I'm pretty good at what I do, at least according to myself and the people I work with, and I'm comparing its capabilities (the latest version of Claude used as an agent inside Cursor) to myself. It can't fully do things on its own and makes mistakes, but it can do a lot.
But suppose you're right, it's 60% as good as "stackoverflow copy-pasting programmers". Isn't that a pretty insanely impressive milestone to just dismiss?
And why would it just get to this point, and then stop? Like, we can all see AIs continuously beating the benchmarks, and the progress feels very fast in terms of experience of using it as a user.
I'd need to hear a pretty compelling argument to believe that it'll suddenly stop, something more compelling than "well, it's not very good yet, therefore it won't be any better", or "Sam Altman is lying to us because incentives".
Sure, it can slow down somewhat because of the exponentially increasing compute costs, but that's assuming no more algorithmic progress, no more compute progress, and no more increases in the capital that flows into this field (I find that hard to believe).
I appreciate your reply. My tone was a little dismissive; I'm currently deep deep in the trenches trying to unwind a tremendous amount of LLM slop in my team's codebase so I'm a little sensitive.
I use Claude every day. It is definitely impressive, but in my experience only marginally more impressive than ChatGPT was a few years ago. It hallucinates less and compiles more reliably, but still produces really poor designs. It really is an overconfident junior developer.
The real risk, and what I am seeing daily, is colleagues falling for the "if you aren't using Cursor you're going to be left behind" FUD. So they learn Cursor, discover that it's an easy way to close tickets without using your brain, and end up polluting the codebase with very questionable designs.
Oh, sorry to hear that you have to deal with that!
The way I'm getting a sense of the progress is using AI for what AI is currently good at, using my human brain to do the part AI is currently bad at, and comparing it to doing the same work without AI's help.
I feel like AI is pretty close to automating 60-80% of the work I would've had to do manually two years ago (as a full-stack web developer).
It doesn't mean that the remaining 20-40% will be automated very quickly, I'm just saying that I don't see the progress getting any slower.
So I think there's an assumption you've made here, that the models are currently "60-80% as good as human programmers".
If you look at code being generated by non-programmers (where you would expect to see these results!), you don't see output that is 60-80% of the output of domain experts (programmers) steering the models.
I think we're extremely imprecise when we communicate in natural language, and this is part of the discrepancy between belief systems.
Will an LLM model read a person's mind about what they want to build better than they can communicate?
That's already what recommender systems (like the TikTok algorithm) do.
But will LLMs be able to orchestrate and fill in the blanks of imprecision in our requests on their own, or will they need human steering?
I think that's where there's a gap in (basically) belief systems of the future.
If we truly get post human-level intelligence everywhere, there is no amount of "preparing" or "working with" the LLMs ahead of time that will save you from being rendered economically useless.
This is mostly a question about how long the moat of human judgement lasts. I think there's an opportunity to work together to make things better than before, using these LLMs as tools that work _with_ us.
Can you phrase this in a concrete way, so that in 2027 we can all agree whether it's true or false, rather than circling a "no true scotsman" argument?
It was surpassed around the beginning of this year, so you'll need to come up with a new one for 2027. Note that the other opinions in that older HN thread almost all expected less.
It won't be able to write a compelling novel, or build a software system solving a real-world problem, or operate heavy machinery, create a sprite sheet or 3d models, design a building or teach.
Long term planning and execution and operating in the physical world is not within reach. Slight variations of known problems should be possible (as long as the size of the solution is small enough).
For teaching, I'm using it to learn about tech I'm unfamiliar with every day, it's one of the things it's the most amazing at.
For the things where the tolerance for mistakes is extremely low and the things where human oversight is extremely importamt, you might be right. It won't have to be perfect (just better than an average human) for that to happen, but I'm not sure if it will.
Bro what are you even talking about? ControlNet has been able to produce consistent assets for years.
How exactly do you think video models work? Frame to frame coherency has been possible for a long time now. A sprite sheet?! Are you joking me. Literally churning them out with AI since 2023.
In large mining operations we already have human assisted teleoperation AI equipment. Was watching one recently where the human got 5 or so push dozers lined up with a (admittedly simple) task of cutting a hill down and then just got them back in line if they ran into anything outside of their training. The push and backup operations along with blade control were done by the AI/dozer itself.
Now, this isn't long term planning, but it is operating in the real world.
People want to live their lives free of finance and centralized personal information.
If you think most people like this stuff you're living in a bubble. I use it every day but the vast majority of people have no interest in using these nightmares of philip k dick imagined by silicon dreamers.
This is absurd, like taking any trend and drawing a straight line to interpolate the future.
If I would do this with my tech stock portfolio, we would probably cross the zero line somewhere late 2025...
If this article were a AI model, it would be catastrophically overfit.
Older and related article from one of the authors titled "What 2026 looks like", that is holding up very well against time. Written in mid 2021 (pre ChatGPT)
> The alignment community now starts another research agenda, to interrogate AIs about AI-safety-related topics. For example, they literally ask the models “so, are you aligned? If we made bigger versions of you, would they kill us? Why or why not?” (In Diplomacy, you can actually collect data on the analogue of this question, i.e. “will you betray me?” Alas, the models often lie about that. But it’s Diplomacy, they are literally trained to lie, so no one cares.)
Will people finally wake up that the AGI X-Risk people have been right and we’re rapidly approaching a really fucking big deal?
This forum has been so behind for too long.
Sama has been saying this a decade now: “Development of Superhuman machine intelligence is probably the greatest threat to the continued existence of humanity” 2015 https://blog.samaltman.com/machine-intelligence-part-1
Hinton, Ilya, Dario Amodei, RLHF inventor, Deepmind founders. They all get it, which is why they’re the smart cookies in those positions.
First stage is denial, I get it, not easy to swallow the gravity of what’s coming.
People have been predicting the singularity to occur sometimes around 2030 and 2045 waaaay further back then 2015. And not just by enthusiasts, I dimly remember an interview with Richard Darkins from back in the day...
Though that doesn't mean that the current version of language models will ever achieve AGI, and I sincerely doubt they will.
They'll likely be a component in the AI, but likely not the thing that "drives"
Vernor Vinge as much as anyone can be credited with the concept of the singularity. In his 1993 essay on it, he said he'd be surprised if it happened before 2005 or after 2030
And why are Altman's words worth anything? Is he some sort of great thinker? Or a leading AI researcher, perhaps?
No. Altman is in his current position because he's highly effective at consolidating power and has friends in high places. That's it. Everything he says can be seen as marketing for the next power grab.
Altman did play some part in bringing ChatGPT about. I think the point is the people making AI or running companies making current AI are saying be wary.
In general it's worth weighting the opinions of people who are leaders in a field, about that field, over people who know little about it.
> Will people finally wake up that the AGI X-Risk people have been right and we’re rapidly approaching a really fucking big deal?
OK, say I totally believe this. What, pray tell, are we supposed to do about it?
Don't you at least see the irony of quoting Sama's dire warnings about the development of AI, without at least mentioning that he is at the absolute forefront of the push to build this technology that can destroy all of humanity. It's like he's saying "This potion can destroy all of humanity if we make it" as he works faster and faster to figure out how to make it.
I mean, I get it, "if we don't build it, someone else will", but all of the discussion around "alignment" seems just blatantly laughable to me. If on one hand your goal is to build "super intelligence", i.e. way smarter than any human or group of humans, how do you expect to control that super intelligence when you're just acting at the middling level of human intelligence?
While I'm skeptical on the timeline, if we do ever end up building super intelligence, the idea that we can control it is a pipe dream. We may not be toast (I mean, we're smarter than dogs, and we keep them around), but we won't be in control.
So if you truly believe super intelligent AI is coming, you may as well enjoy the view now, because there ain't nothing you or anyone else will be able to do to "save humanity" if or when it arrives.
Political organization to force a stop to ongoing research? Protest outside OAI HQ? There are lots of thing we could, and many of us would, do if more people were actually convinced their life were in danger.
> Political organization to force a stop to ongoing research? Protest outside OAI HQ?
Come on, be real. Do you honestly think that would make a lick of difference? Maybe, at best, delay things by a couple months. But this is a worldwide phenomenon, and humans have shown time and time again that they are not able to self organize globally. How successful do you think that political organization is going to be in slowing China's progress?
Humans have shown time and time again that they are able to self-organize globally.
Nuclear deterrence -- human cloning -- bioweapon proliferation -- Antarctic neutrality -- the list goes on.
> How successful do you think that political organization is going to be in slowing China's progress?
I wish people would stop with this tired war-mongering. China was not the one who opened up this can of worms. China has never been the one pushing the edge of capabilities. Before Sam Altman decided to give ChatGPT to the world, they were actively cracking down on software companies (in favor of hardware & "concrete" production).
We, the US, are the ones who chose to do this. We started the race. We put the world, all of humanity, on this path.
> Do you honestly think that would make a lick of difference?
I don't know, it depends. Perhaps we're lucky and the timelines are slow enough that 20-30% of the population loses their jobs before things become unrecoverable. Tech companies used to warn people not to wear their badges in public in San Francisco -- and that was what, 2020? Would you really want to work at "Human Replacer, Inc." when that means walking out and about among a population who you know hates you, viscerally? Or if we make it to 2028 in the same condition. The Bonus Army was bad enough -- how confident are you that the government would stand their ground, keep letting these labs advance capabilities, when their electoral necks were on the line?
This defeatism is a self-fulfilling prophecy. The people have the power to make things happen, and rhetoric like this is the most powerful thing holding them back.
> China was not the one who opened up this can of worms
Thank you. As someone who lives in Southeast Asia (and who also has lived in East Asia -- pardon the deliberate vagueness, for I do not wish to reveal too many potentially personally identifying information), this is how many of us in these regions view the current tensions between China and Taiwan as well.
Don't get me wrong; we acknowledge that many Taiwanese people want independence, that they are a people with their own aspirations and agency. But we can also see that the US -- and its European friends, which often blindly adopt its rhetoric and foreign policy -- is deliberately using Taiwan as a disposable pawn to attempt to provoke China into a conflict. The US will do what it has always done ever since the post-WW2 period -- destabilise entire regions of countries to further its own imperialistic goals, causing the deaths and suffering of millions, and then leaving the local populations to deal with the fallout for many decades after.
Without the US intentionally stoking the flames of mutual antagonism between China and Taiwan, the two countries could have slowly (perhaps over the next decades) come to terms with each other, be it voluntary reunification or peaceful separation. If you know a bit of Chinese history, it is not entirely far-fetched at all to think that the Chinese might eventually agree to recognising Taiwan as an independent nation, but now this option has now been denied because the US has decided to use Taiwan as a pawn in a proxy conflict.
To anticipate questions about China's military invasion of Taiwan by 2027: No, I do not believe it will happen. Don't believe everything the US authorities claim.
> If on one hand your goal is to build "super intelligence", i.e. way smarter than any human or group of humans, how do you expect to control that super intelligence when you're just acting at the middling level of human intelligence?
That's exactly what the true AGI X-Riskers think! Sama acknowledges the intense risk but thinks the path forward is inevitable anyway so hoping that building intelligence will give them the intelligence to solve alignment. The other camp, a la Yudkowsky, believe it's futile to just hope it gets solved without AGI capabilities first becoming more intelligent, powerful, and disregarding any of our wishes. And then we've ceded any control of our future to an uncaring system that treats us as a means to achieve its original goals like how an ant is in the way of a Google datacenter. I don't see how anyone who thinks "maybe stock number go up as your only goal is not the best way to make people happy", can miss this.
Slightly more detail: until about 2001 Yudkowsky was what we would now call an AI accelerationist, then it dawned on him that creating an AI that is much "better at reality" than people are would probably kill all the people unless the AI has been carefully designed to stay aligned with human values (i.e., to want what we want) and that ensuring that it stays aligned is a very thorny technical problem, but was still hopeful that humankind would solve the thorny problem. He worked full time on the alignment problem himself. In 2015 he came to believe that the alignment problem is so hard that it is very very unlikely to be solved by the time it is needed (namely, when the first AI is deployed that is much "better at reality" than people are). He went public with his pessimism in Apr 2022, and his nonprofit (the Machine Intelligence Research Institute) fired most of its technical alignment researchers and changed its focus to lobbying governments to ban the dangerous kind of AI research.
There is a strong financial incentive for a lot of people on this site to deny they are at risk from it, or to deny what they are building has risk and they should have culpability from that.
> "Development of Superhuman machine intelligence is probably the greatest threat to the continued existence of humanity”
If that's really true, why is there such a big push to rapidly improve AI? I'm guessing OpenAI, Google, Anthropic, Apple, Meta, Boston Dynamics don't really believe this. They believe AI will make them billions. What is OpenAI's definition of AGI? A model that makes $100 billion?
Because they also believe the development of superhuman machine intelligence will probably be the greatest invention for humanity. The possible upsides and downsides are both staggeringly huge and uncertain.
> I wonder who pays the bills of the authors. And your bills, for that matter.
Also, what a weirdly conspiratorial question. There's a prominent "Who are we?" button near the top of the page and it's not a secret what any of the authors did or do for a living.
also it's not conspiratorial to wonder if someone in silicon valley today receives funding through the AI industry lol like half the industry is currently propped up by that hype, probably half the commenters here are paid via AI VC investments
I think it's not holding up that well outside of predictions about AI research itself. In particular, he makes a lot of predictions about AI impact on persuasion, propaganda, the information environment, etc that have not happened.
This doesn’t seem like a great way to reason about the predictions.
For something like this, saying “There is no evidence showing it” is a good enough refutation.
Counterpointing that “Well, there could be a lot of this going on, but it is in secret.” - that could be a justification for any kooky theory out there. Bigfoot, UFOs, ghosts. Maybe AI has already replaced all of us and we’re Cylons. Something we couldn’t know.
The predictions are specific enough that they are falsifiable, so they should stand or fall based on the clear material evidence supporting or contradicting them.
Could you give some specific examples of things you feel definitely did not come to pass? Because I see a lot of people here talking about how the article missed the mark on propaganda; meanwhile I can tab over to twitter and see a substantial portion of the comment section of every high-engagement tweet being accused of being Russia-run LLM propaganda bots.
Agree. The base claims about LLMs getting bigger, more popular, and capturing people's imagination are right. Those claims are as easy as it gets, though.
Look into the specific claims and it's not as amazing. Like the claim that models will require an entire year to train, when in reality it's on the order of weeks.
The societal claims also fall apart quickly:
> Censorship is widespread and increasing, as it has for the last decade or two. Big neural nets read posts and view memes, scanning for toxicity and hate speech and a few other things. (More things keep getting added to the list.) Someone had the bright idea of making the newsfeed recommendation algorithm gently ‘nudge’ people towards spewing less hate speech; now a component of its reward function is minimizing the probability that the user will say something worthy of censorship in the next 48 hours.
This is a common trend in rationalist and "X-risk" writers: Write a big article with mostly safe claims (LLMs will get bigger and perform better!) and a lot of hedging, then people will always see the article as primarily correct. When you extract out the easy claims and look at the specifics, it's not as impressive.
This article also shows some major signs that the author is deeply embedded in specific online bubbles, like this:
> Most of America gets their news from Twitter, Reddit, etc.
Sites like Reddit and Twitter feel like the entire universe when you're embedded in them, but when you step back and look at the numbers only a fraction of the US population are active users.
> (2025) Making models bigger is not what’s cool anymore. They are trillions of parameters big already. What’s cool is making them run longer, in bureaucracies of various designs, before giving their answers.
Holy shit. That's a hell of a called shot from 2021.
its vague, and could have meant anything. everyone knew parameters would grow and its reasonable to expect that things that grow have diminishing returns at some point. this happened in late 2023 and throughout 2024 as well.
That quote almost perfectly describes o1, which was the first major model to explicitly build in compute time as a part of its scaling. (And despite claims of vagueness, I can't think of a single model release it describes better). The idea of a scratchpad was obvious, but no major chatbot had integrated it until then, because they were all focused on parameter scaling. o1 was released at the very end of 2024.
I'm not seeing the prescience here - I don't wanna go through the specific points but the main gist here seems to be that chatbots will become very good at pretending to be human and influencing people to their own ends.
I don't think much has happened on these fronts (owning to a lack of interest, not technical difficulty). AI boyfriends/roleplaying etc. seems to have stayed a very niche interest, with models improving very little over GPT3.5, and the actual products are seemingly absent.
It's very much the product of the culture war era, where one of the scary scenarios show off, is a chatbot riling up a set of internet commenters and goarding them lashing out against modern leftist orthodoxy, and then cancelling them.
With all thestrongholds of leftist orthodoxy falling into Trump's hands overnight, this view of the internet seems outdated.
Troll chatbots still are a minor weapon in information warfare/ The 'opinion bubbles' and manipulation of trending topics on social media (with the most influential content still written by humans), to change the perception of what's the popular concensus still seem to hold up as primary tools of influence.
Nowadays, when most people are concerned about stuff like 'will the US go into a shooting war against NATO' or 'will they manage to crash the global economy', just to name a few of the dozen immediately pressing global issues, I think people are worried about different stuff nowadays.
At the same time, there's very little mention of 'AI will take our jobs and make us poor' in both the intellectual and physical realms, something that's driving most people's anxiety around AI nowadays.
It also puts the 'superintelligent unaligned AI will kill us all' argument very often presented by alignment people as a primary threat rather than the more plausible 'people controlling AI are the real danger'.
I just spent some time trying to make claude and gemini make a violin plot of some polar dataframe. I've never used it and it's just for prototyping so i just went "apply a log to the values and make a violin plot of this polars dataframe". ANd had to iterate with them for 4/5 times each. Gemini got it right but then used deprecated methods
I might be doing llm wrong, but i just can't get how people might actually do something not trivial just by vibe coding. And it's not like i'm an old fart either, i'm a university student
It's spicy auto complete. Ask it to create a program that can create a violin plot from a CVS file. Because this has been "done before", it will do a decent job.
Yes, you're most likely doing it wrong. I would like to add that "vibe coding" is a dreadful term thought up by someone who is arguably not very good at software engineering, as talented as he may be in other respects. The term has become a misleading and frankly pejorative term. A better, more neutral one is AI assisted software engineering.
You see, the issue I get petty about is that Ai is advertised as the one ring to rule them all software. VCs creaming themselves at the thought of not having to pay developers and using natural language. But then, you have to still adapt to the Ai, and not vice versa. "you're doing it wrong". This is not the idea that VCs bros are selling
Then, I absolutely love being aided by llms for my day to day tasks. I'm much more efficient when studying and they can be a game changer when you're stuck and you don't know how to proceed. You can discuss different implementation ideas as if you had a colleague, perhaps not a PhD smart one but still someone with a quite deep knowledge of everything
But, it's no miracle. That's the issue I have with the way the idea of Ai is sold to the c suites and the general public
Lets imagine we got AGI at the start of 2022. I'm talking about human level+ as good as you coding and reasoning AI that works well on the hardware from that age.
What would the world look like today? Would you still have your job. With the world be in total disarray? Would unethical companies quickly fire most their staff and replace them with machines? Would their be mass riots in the streets by starving neo-luddites? Would automated drones be shooting at them?
Simply put people and our social systems are not ready for competent machine intelligence and how fast it will change the world. We should feel lucky we are getting a ramp up period, and hopefully one that draws out a while longer.
> had to iterate with them for 4/5 times each. Gemini got it right but then used deprecated methods
How hard would it be to automate these iterations?
How hard would it be to automatically check and improve the code to avoid deprecated methods?
I agree that most products are still underwhelming, but that doesn't mean that the underlying tech is not already enough to deliver better LLM-based products. Lately I've been using LLMs more and more to get started with writing tests on components I'm not familiar with, it really helps.
> How hard would it be to automate these iterations?
The fact that we're no closer to doing this than we were when chatgpt launched suggests that it's really hard. If anything I think it's _the_ hard bit vs. building something that generates plausible text.
Solving this for the general case is imo a completely different problem to being able to generate plausible text in the general case.
They can check their work and try again an infinite number of times, but the rate at which they succeed seems to just get worse and worse the further from the beaten path (of existing code from existing solutions) that they stray.
How hard can it be to create a universal "correctness" checker? Pretty damn hard!
Our notion of "correct" for most things is basically derived from a very long training run on reality with the loss function being for how long a gene propagated.
You don't need a full correctness checker to get a useful product though. New code generated by the current generation of LLMs, which also compiles and passes existing tests, is likely to be somewhat useful in my experience. The problem is that we still get too much code that doesn't pass these basic requirements.
You pretty much just have to play around with them enough to be able to intuit what things they can do and what things they can't. I'd rather have another underling, and not just because they grow into peers eventually, but LLMs are useful with a bit of practice.
> "OpenBrain (the leading US AI project) builds AI agents that are good enough to dramatically accelerate their research. The humans, who up until very recently had been the best AI researchers on the planet, sit back and watch the AIs do their jobs, making better and better AI systems."
I'm not sure what gives the authors the confidence to predict such statements. Wishful thinking? Worst-case paranoia? I agree that such an outcome is possible, but on 2--3 year timelines? This would imply that the approach everyone is taking right now is the right approach and that there are no hidden conceptual roadblocks to achieving AGI/superintelligence from DFS-ing down this path.
All of the predictions seem to ignore the possibility of such barriers, or at most acknowledge the possibility but wave it away by appealing to the army of AI researchers and industry funding being allocated to this problem. IMO it is the onus of the proposers of such timelines to argue why there are no such barriers and that we will see predictable scaling in the 2--3 year horizon.
It's my belief (and I'm far from the only person who thinks this) that many AI optimists are motivated by an essentially religious belief that you could call Singularitarianism. So "wishful thinking" would be one answer. This document would then be the rough equivalent of a Christian fundamentalist outlining, on the basis of tangentially related news stories, how the Second Coming will come to pass in the next few years.
Eh, not sure if the second coming is a great analogy. That wholly depends on the whims of a fictional entity performing some unlikely actions.
Instead think of them saying a crusade occurring in the next few years. When the group saying the crusade is coming is spending billions of dollars to trying to make just that occur you no longer have the ability to say it's not going to happen. You are now forced to examine the risks of their actions.
Crackpot millenarians have always been a thing. This crop of them is just particularly lame and hellbent on boiling the oceans to get their eschatological outcome.
Reminds me of Fallout's Children of Atom "Church of the Children of Atom"
Maybe we'll see "Church of the Children of Altman" /s
It seems without a framework of ethics/morality (insert XYZ religion), us humans find one to grasp onto. Be it a cult, a set of not-so-fleshed-out ideas/philosophies etc.
People who say they aren't religious per-se, seem to have some set of beliefs that amount to religion. Just depends who or what you look towards for those beliefs, many of which seem to be half-hazard.
People I may disagree with the most, many times at least have a realization of what ideas/beliefs are unifying their structure of reality, with others just not aware.
A small minority of people can rely on schools of philosophical thought, and 'try on' or play with different ideas, but have a self-reflection that allows them to see when they transgress from ABC philosophy or when the philosophy doesn't match with their identity to a degree.
This is a letter signed by the most lauded AI researchers on Earth, along with CEOs from the biggest AI companies and many other very credible professors of computer science and engineering:
"Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war."
https://www.safe.ai/work/statement-on-ai-risk
Laughing it off as the same as the Second Coming CANNOT work. Unless you think yourself cleverer and more capable of estimating the risk than all of these experts in the field.
Especially since many of them have incentives that should prevent them from penning such a letter.
Troubling that these eminent great leaders does not cite climate change among societal-scale risks, a bigger and more certain societal-scale risk than a pandemy.
Would be a shame to have energy consumption by datacenters regulated, am I right ?
It also ignores the possibility of plateau... maybe there's a maximum amount of intelligence that matter can support, and it doesn't scale up with copies or speed.
Or scales sub-linearly with hardware. When you're in the rising portion of an S-curve[1] you can't tell how much longer it will go on before plateauing.
A lot of this resembles post-war futurism that assumed we would all be flying around in spaceships and personal flying cars within a decade. Unfortunately the rapid pace of transportation innovation slowed due to physical and cost constraints and we've made little progress (beyond cost optimization) since.
The fact that it scales sub linearly with hardware is well known and in fact foundational to the scaling laws on which modern LLMs are built, ie performance scales remarkably closely to log(compute+data), over many orders of magnitude.
Eh, these mathematics still don't work out in humans favor...
Lets say intelligence caps out at the maximum smartest person that's ever lived. Well, the first thing we'd attempt to do is build machines up to that limit that 99.99999 percent of us would never get close to. Moreso the thinking parts of humans is only around 2 pounds of mush in side of our heads. On top of that you don't have to grow them for 18 years first before they start outputting something useful. That and they won't need sleep. Oh and you can feed them with solar panels. And they won't be getting distracted by that super sleek server rack across the aisle.
We do know 'hive' or societal intelligence does scale over time especially with integration with tooling. The amount of knowledge we have and the means of which we can apply it simply dwarf previous generations.
I would assume this comes from having faith in the overall exponential trend rather than getting that much into the weeds of how this will come about. I can sort of see why you might think that way - everyone was talking about hitting a wall with brute force scaling and then inference time scaling comes along to keep things progressing. I wouldn't be quite as confident personally and as have many have said before, a sigmoid looks like an exponential in it's initial phase.
Global warming's worst effects aren't in 2-3 years, but we all (I hope) still care very much about it.
Perhaps the article is wrong about the timescale, but given how much AI has improved in the last 5 years, can you agree that it's likely to reach 'sit back and watch' levels in the next 5-10 years?
There's a lot to potentially unpack here, but idk, the idea that humanity entering hell (extermination) or heaven (brain uploading; aging cure) is whether or not we listen to AI safety researchers for a few months makes me question whether it's really worth unpacking.
That's obviously not true. Before OpenAI blew the field open, multiple labs -- e.g. Google -- were intentionally holding back their research from the public eye because they thought the world was not ready. Investors were not pouring billions into capabilities. China did not particularly care to focus on this one research area, among many, that the US is still solidly ahead in.
The only reason timelines are as short as they are is because of people at OpenAI and thereafter Anthropic deciding that "they had no choice". They had a choice, and they took the one which has chopped at the very least years off of the time we would otherwise have had to handle all of this. I can barely begin to describe the magnitude of the crime that they have committed -- and so I suggest that you consider that before propagating the same destructive lies that led us here in the first place.
The simplicity of the statement "If we don't do it, someone else will." and thinking behind it eventually means someone will do just that unless otherwise prevented by some regulatory function.
Simply put, with the ever increasing hardware speeds we were dumping out for other purposes this day would have come sooner than later. We're talking about only a year or two really.
Cloning? Bioweapons? Ever larger nuclear stockpiles? The world has collectively agreed not to do something more than once. AI would be easier to control than any of the above. GPUs can't be dug out of the ground.
But every time, it doesn't have to happen yet. And when you're talking about the potential deaths of millions, or billions, why be the one who spawns the seed of destruction in their own home country? Why not give human brotherhood a chance? People have, and do, hold back. You notice the times they don't, and the few who don't -- you forget the many, many more who do refrain from doing what's wrong.
"We have to nuke the Russians, if we don't do it first, they will"
"We have to clone humans, if we don't do it, someone else will"
"We have to annex Antarctica, if we don't do it, someone else will"
Maybe people should just don’t listen to AI safety researchers for a few months? Maybe they are qualified to talk about inference and model weights and natural language processing, but not particularly knowledgeable about economics, biology, psychology, or… pretty much every other field of study?
The hubris is strong with some people, and a certain oligarch with a god complex is acting out where that can lead right now.
It's charitable of you to think that they might be qualified to talk about inference and model weights and such. They are AI safety researchers, not AI researchers. Basically, a bunch of doom bloggers, jerking each other in a circle, a few of whom were tolerated at one of the major labs for a few years, to do their jerking on company time.
This is hilariously over-optimistic on the timescales. Like on this timeline we'll have a Mars colony in 10 years, immortality drugs in 15 and Half Life 3 in 20.
The story is very clearly modeled to follow the exponential curve they show.
Like the drew the curve out into the shape they wanted, put some milestones on it, and then went to work imagining what would happen if it continued with a heavy dose of X-risk doomerism to keep it spicy.
It conveniently ignores all of the physical constraints around things like manufacturing GPUs and scaling training networks.
In section 4 they discuss their projections specifically for model size, the state of inference chips in 2027, etc. It's largely pretty in line with expectations in terms of the capacity, and they only project them using 10k of their latest gen wafer scale inference chips by late 2027, roughly like 1M H100 equivalents. That doesn't seem at all impossible. They also earlier on discuss expectations for growth in efficiency of chips, and for growth in spending, which is only ~10x over the next 2.5 years, not unreasonable in absolute terms at all given the many tens of billions of dollars flooding in.
So on the "can we train the AI" front, they mostly are just projecting 2.5 years of the growth in scale we've been seeing.
The reason they predict a fairly hard takeoff is they expect that distillation, some algorithmic improvements, and iterated creation of synthetic data, training, and then making more synthetic data will enable significant improvements in efficiency of the underlying models (something still largely in line with developments over the last 2 years). In particular they expect a 10T parameter model in early 2027 to be basically human equivalent, and they expect it to "think" at about the rate humans do, 10 words/second. That would require ~300 teraflops of compute per second to think at that rate, or ~0.1H100e. That means one of their inference chips could potentially run ~1000 copies (or fewer copies faster etc. etc.) and thus they have the capacity for millions of human equivalent researchers (or 100k 40x speed researchers) in early 2027.
They further expect distillation of such models etc. to squeeze the necessary size down / more expensive models overseeing much smaller but still good models squeezing the effective amount of compute necessary, down to just 2T parameters and ~60 teraflops each, or 5000 human-equivalents per inference chip, making for up to 50M human-equivalents by late 2027.
This is probably the biggest open question and the place where the most criticism seems to me to be warranted. Their hardware timelines are pretty reasonable, but one could easily expect needing 10-100x more compute or even perhaps 1000x than they describe to achieve Nobel-winner AGI or superintelligence.
I don’t believe so. I think all important parts that each need to be scaled to advance significantly in the LLM paradigm are at or near the end of the steep part of the sigmoid:
1) useful training data available in the internet
2) number of humans creating more training data ”manually”
3) parameter scaling
4) ”easy” algorithmic inventions
5) available+buildable compute
”Just” needing a few more algorithmic inventions to keep the graphs exponential is a cop out. It is already obvious that just scaling parameters and compute is not enough.
I personally predict that scaling LLMs for solving all physical tasks (eg cleaning robots) or intellectual pursuits (they suck at multiplication) will not work out.
We will get better specialized tools by collecting data from specific, high economic value, constrained tasks, and automating them, but scaling a (multimodal) LLM to solve everything in a single model will not be economically viable. We will get more natural interfaces for many tasks.
This is how I think right now as a ML researcher, will be interesting to see how wrong was I in 2 years.
EDIT: addition about latest algorithmic advances:
- Deepseek style GRPO requires a ladder of scored problems progressively more difficult and appropriate to get useful gradients. For open-ended problems (like most interesting ones are) we have no ladders for, and it doesn’t work. In particular, learning to generate code for leetcode problems with a good number of well made unit tests is what it is good for.
- Test-time inference is just adding an insane amount of more compute after training to brute-force double-check the sanity of answers
The true absurdity of this timeline is that everyone is behaving as perfectly rational players, which is absurd when you look at the current presidential administration, particularly in light of recent events.
5 years: AI coding assistants are a lot better than they are now, but still can't actually replace junior engineers (at least ones that aren't shit). AI fraud is rampant, with faked audio commonplace. Some companies try replacing call centres with AI, but it doesn't really work and everyone hates it.
Tesla's robotaxi won't be available, but Waymo will be in most major US cities.
10 years: AI assistants are now useful enough that you can use them in the ways that Apple and Google really wanted you to use Siri/Google Assistant 5 years ago. "What have I got scheduled for today?" will give useful results, and you'll be able to have a natural conversation and take actions that you trust ("cancel my 10am meeting; tell them I'm sick").
AI coding assistants are now very good and everyone will use them. Junior devs will still exist. Vibe coding will actually work.
Most AI Startups will have gone bust, leaving only a few players.
Art-based AI will be very popular and artists will use it all the time. It will be part of their normal workflow.
Waymo will become available in Europe.
Some receptionists and PAs have been replaced by AI.
15 years: AI researchers finally discover how to do on-line learning.
Humanoid robots are robust and smart enough to survive in the real world and start to be deployed in controlled environments (e.g. factories) doing simple tasks.
Driverless cars are "normal" but not owned by individuals and driverful cars are still way more common.
Small light computers become fast enough that autonomous slaughter it's become reality (i.e. drones that can do their own navigation and face recognition etc.)
> Small light computers become fast enough that autonomous slaughter it's become reality
This is the real scary bit. I'm not convinced that AI will ever be good enough to think independently and create novel things without some serious human supervision, but none of that matters when applied to machines that are destructive by design and already have expectations of collateral damage. Slaughterbots are going to be the new WMDs — and corporations are salivating at the prospect of being first movers. https://www.youtube.com/watch?v=UiiqiaUBAL8
>I'm not convinced that AI will ever be good enough to think independently a
and
>Why do you believe that?
What takes less effort, time to deploy, and cost? I mean there is at least some probability we kill ourselves off with dangerous semi-thinking war machines leading to theater scale wars to the point society falls apart and we don't have the expensive infrastructure to make AI as envisioned in the future.
With that said, I'm in the camp that we can create AGI as nature was able to with a random walk, we'll be able to reproduce it with intelligent design.
If you bake the model onto the chip itself, which is what should be happening for local LLMs once a good enough one is trained eventually, you’ll be looking at orders of magnitude reduction in power consumption at constant inference speed.
you should add a bit where AI is pushed really hard in places where the subjects have low political power, like management of entry level workers, care homes or education and super bad stuff happens.
Also we need a big legal event to happen where (for example) autonomous driving is part of a really big accident where lots of people die or someone brings a successful court case that an AI mortgage underwriter is discriminating based on race or caste. It won't matter if AI is actually genuinely responsible for this or not, what will matter is the push-back and the news cycle.
Maybe more events where people start successfully gaming deployed AI at scale in order to get mortgages they shouldn't or get A-grades when they shouldn't.
> Some companies try replacing call centres with AI, but it doesn't really work and everyone hates it.
I think this is much closer than you think, because there's a good percentage of call centers that are basically just humans with no power cosplaying as people who can help.
My fiber connection went to shit recently. I messaged the company, and got a human who told me they were going to reset the connection from their side, if I rebooted my router. 30m later with no progress, I got a human who told me that they'd reset my ports, which I was skeptical about, but put down to a language issue, and again reset my router. 30m later, the human gave me an even more outlandish technical explanation of what they'd do, at which point I stumbled across the magical term "complaint" ... an engineer phoned me 15m later, said there was something genuinely wrong with the physical connection, and they had a human show up a few hours later and fix it.
No part of the first-layer support experience there would have been degraded if replaced by AI, but the company would have saved some cash.
So in the past 5 years we went from not having ChatGPT at all and it being released in 2022 (with non "chat" models before that) but in the next 5 now that the entire tech world is consumed with making better AI models, we'll just get slightly better AI coding assistants?
Reminds me of that comment about the first iPod being lame and having less space than a nomad. Worst take I've ever seen on here recently.
> Coding AIs increasingly look like autonomous agents rather than mere assistants: taking instructions via Slack or Teams and making substantial code changes on their own, sometimes saving hours or even days.
That is literally the pitch line for Devin. I recently spoke to the CTO of a small healthtech startup and he was very pro-Devin for small fixes and PRs, and thought he was getting his money worth. Claude Code is a little clunkier but gives better results, and it wouldn't take much effort to hook it up to a Slack interface.
Yeah, I get that there are startups trying to do it. But I work with Cursor quite a bit… there is no way I would trust an LLM code agent to take high-level direction and issue a PR on anything but the most trivial bug fix.
Last year they couldn’t even do a simple fix (they could add a null coalescing operator or an early return which didn’t make sense, that’s about it). Now I’m getting hundreds of LOC of functionality with multiple kLOC of tests out of the agent mode. No way it gets in without a few iterations, but it’s sooo much better than last April.
These timelines always assume that things progress as quickly as they can be conceived of, likely because these timelines come from "Ideas Guys" whose involvement typically ends at that point.
Orbital mechanics begs to disagree about a Mars colony in 10 years. Drug discovery has many steps that take time, even just the trials will take 5 years, let alone actually finding the drugs.
No it didn’t. At least not for new small molecule drugs. It did reduce times a bit for the first vaccines because there were many volunteers available, and it did allow some antibody drug candidates to be used before full testing was complete. The only approved small molecule drug for covid is paxlovid, with both components of its formulation tested on humans for the first time many years before covid. All the rest of the small molecule drugs are still in early parts of the pipeline or have been abandoned.
The other reply has better info on covid specifically, but also consider that this refers to "immortality drugs". How long do we have to test those to conclude that they do in fact provide "immortality"?
Now sure, they don't actually mean immortality, and we don't need to test forever to conclude they extend life, but we probably do have to test for years to get good data on whether a generic life extension drug is effective, because you're testing against illness, old age, etc, things that take literally decades to kill.
That's not to mention that any drug like that will be met with intense skepticism and likely need to overcome far more scrutiny than normal (rather than the potentially less scrutiny that covid drugs might have managed).
Curing aging might be immediately obvious--like old folks have all their diminished capacity restored, and they are more similar to their physical peak.
Old people may have a lot of reason to volunteer to use these.
I think we will discover that our body can reset itself via some trigger mechanism. It built us once, the code isn't gone. It can do it again.
I don't think curing aging is an easy thing anyway. The whole of evolution is basically set up against it. Because it would break evolution itself.
There isn't a single species that has somehow gained infinite lifespans though mutation. I think it's a lot harder to accomplish. I guess a lot of it has to do with degrading DNA (and again, evolution never had to figure out a fix for that because it's a feature not a bug).
trial times were very brief for Covid vaccines because 1) there was no shortage of volunteers, capital, and political alignment at every level 2) the virus was everywhere and so it was really, really easy to verify if it was working. Compare this with a vaccination for a very rare but deadly disease: it's really hard to know if it's working because you can't just expose your test subjects to the deadly disease!
Science is not ideas: new conceptual schemes must be invented, confounding variables must be controlled, dead-ends explored. This process takes years.
Engineering is not science: kinks must be worked out, confounding variables incorporated. This process also takes years.
Technology is not engineering: the purely technical implementation must spread, become widespread and beat social inertia and its competition, network effects must be established. Investors and consumers must be convinced in the long term. It must survive social and political repercussions. This process takes yet more years.
Well, Teslas do have "Full Self Driving". It's not actually fully self driving and that doesn't even seem to be on the horizon but it doesn't appear to be stopping Tesla supporters.
Seems very sinophobic. Deepseek and Manus have shown that China is legitimately an innovation powerhouse in AI but this article makes it sound like they will just keep falling behind without stealing.
That whole section seems to be pretty directly based on DeepSeek's "very impressive work" with R1 being simultaneously very impressive, and several months behind OpenAI. (They more or less say as much in footnote 36.) They blame this on US chip controls just barely holding China back from the cutting edge by a few months. I wouldn't call that a knock on Chinese innovation.
But it also assumes China would never really catch up to American chip companies. China is already investing heavily in chip R&D and things like RISC-V, I think it’s very plausible that lag window shrinks over this horizon. Perhaps even flipping given their much larger willingness to use industrial policy for goals they want achieved.
Yes, it's extremely sinophobic and entirely too dismissive of China. It's pretty clear what the author's political leanings are, by what they mention and by what they do not.
Don’t assume that because the article depicts this competition between the US and China, that the authors actually want China to fail. Consider the authors and the audience.
The work is written by western AI safety proponents, who often need to argue with important people who say we need to accelerate AI to “win against China” and don’t want us to be slowed down by worrying about safety.
From that perspective, there is value in exploring the scenario: ok, if we accept that we need to compete with China, what would that look like? Is accelerating always the right move? The article, by telling a narrative where slowing down to be careful with alignment helps the US win, tries to convince that crowd to care about alignment.
Perhaps, people in China can make the same case about how alignment will help China win against US.
Exactly how I read it, this reeks of the war drive toward China, nonsensical predictions and comical red scare portrayals, "legions of ccp spies". Just in time for the new McCarthyism rolling out.
Amusing sci-fi, i give it a B- for bland prose, weak story structure, and lack of originality - assuming this isn't all AI gen slop which is awarded an automatic F.
>All three sets of worries—misalignment, concentration of power in a private company, and normal concerns like job loss—motivate the government to tighten its control.
A private company becoming "too powerful" is a non issue for governments, unless a drone army is somewhere in that timeline. Fun fact the former head of the NSA sits on the board of Open AI.
Job loss is a non issue, if there are corresponding economic gains they can be redistributed.
"Alignment" is too far into the fiction side of sci-fi. Anthropomorphizing today's AI is tantamount to mental illness.
"But really, what if AGI?" We either get the final say or we don't. If we're dumb enough to hand over all responsibility to an unproven agent and we get burned, then serves us right for being lazy. But if we forge ahead anyway and AGI becomes something beyond review, we still have the final say on the power switch.
What is this, some OpenAI employee fan fiction? Did Sam himself write this?
OpenAI models are not even SOTA, except that new-ish style transfer / illustration thing that made all us living in Ghibli world for a few days. R1 is _better_ than o1, and open-weights. GPT-4.5 is disappointing, except for a few narrow areas where it excels. DeepResearch is impressive though, but the moat is in tight web search / Google Scholar search integration, not weights. So far, I'd bet on open models or maybe Anthropic, as Claude 3.7 is the current SOTA for most tasks.
As of the timeline, this is _pessimistic_. I already write 90% code with Claude, so are most of my colleagues. Yes, it does errors, and overdoes things. Just like a regular human middle-stage software engineer.
Also fun that this assumes relatively stable politics in the US and relatively functioning world economy, which I think is crazy optimistic to rely on these days.
Also, superpersuasion _already works_, this is what I am researching and testing. It is not autonomous, it is human-assisted by now, but it is a superpower for those who have it, and it explains some of the things happening with the world right now.
Is this demonstrated in any public research? Unless you just mean something like "good at persuading" -- which is different from my understanding of the term -- I find this hard to believe.
That singularity happened in the fifth century BCE when people figured out that they could charge silver to teach the art of rhetoric and not just teach their sons and nephews
Could not get through the entire thing. It’s mostly a bunch of fantasy intermingled with bits of possible interesting discussion points. The whole right side metrics are purely a distraction because entirely fiction.
Bad future predictions: short-sighted guesses based on current trends and vibe. Often depend on individuals or companies. Made by free-riders. Example: Twitter.
Good future predictions: insights into the fundamental principles that shape society, more law than speculation. Made by visionaries. Example: Vernor Vinge.
640 comments
[ 2.6 ms ] story [ 391 ms ] threadWould love to read a perspective examining "what is the slowest reasonable pace of development we could expect." This feels to me like the fastest (unreasonable) trajectory we could expect.
Their research is consistent with a similar story unfolding over 8-10 years instead of 2.
That's kind of unavoidably what accelerating progress feels like.
The reality now is, that the current LLMs still often create stuff, that costs me more time to fix, than to do it myself. So I still write a lot of code myself. It is very impressive, that I can think about stopping writing code myself. But my job as a software developer is, very, very secure.
LLMs are very unable to build maintainable software. They are unable to understand what humans want and what the codebase need. The stuff they build is good-looking garbage. One example I've seen yesterday: one dev committed code, where the LLM created 50 lines of React code, complete with all those useless comments and for good measure a setTimeout() for something that should be one HTML DIV with two tailwind classes. They can't write idiomatic code, because they write code, that they were prompted for.
Almost daily I get code, commit messages, and even issue discussions that are clearly AI-generated. And it costs me time to deal with good-looking but useless content.
To be honest, I hope that LLMs get better soon. Because right now, we are in an annoying phase, where software developers bog me down with AI-generated stuff. It just looks good but doesn't help writing usable software, that can be deployed in production.
To get to this point, LLMs need to get maybe a hundred times faster, maybe a thousand or ten thousand times. They need a much bigger context window. Then they can have an inner dialogue, where they really "understand" how some feature should be built in a given codebase. That would be very useful. But it will also use so much energy that I doubt that it will be cheaper to let a LLM do those "thinking" parts over, and over again instead of paying a human to build the software. Perhaps this will be feasible in five or eight years. But not two.
And this won't be AGI. This will still be a very, very fast stochastic parrot.
So the question is, do you think the current road leads to AGI? How far down the road is it? As far as I can see, there is not a "status quo bias" answer to those questions.
Compare the automobile. Automobiles today are a lot nicer than they were 50 years ago, and a lot more efficient. Does that mean cars that never need fuel or recharging are coming soon, just because the trend has been higher efficiency? No, because the fundamental physical realities of drag still limit efficiency. Moreover, it turns out that making 100% efficient engines with 100% efficient regenerative brakes is really hard, and "just throw more research at it" isn't a silver bullet. That's not "there won't be many future improvements", but it is "those future improvements probably won't be any bigger than the jump from GPT-3 to o1, which does not extrapolate to what OP claims their models will do in 2027."
AI in 2027 might be the metaphorical brand-new Lexus to today's beat-up Kia. That doesn't mean it will drive ten times faster, or take ten times less fuel. Even if high-end cars can be significantly more efficient than what average people drive, that doesn't mean the extra expense is actually worth it.
One is inherently a more challenging physics problem.
It's nowhere near as good as someone actually building and maintaining systems. It's barely able to vomit out an MVP and it's almost never capable of making a meaningful change to that MVP.
If your experiences have been different that's fine, but in my day job I am spending more and more time just fixing crappy LLM code produced and merged by STAFF engineers. I really don't see that changing any time soon.
But suppose you're right, it's 60% as good as "stackoverflow copy-pasting programmers". Isn't that a pretty insanely impressive milestone to just dismiss?
And why would it just get to this point, and then stop? Like, we can all see AIs continuously beating the benchmarks, and the progress feels very fast in terms of experience of using it as a user.
I'd need to hear a pretty compelling argument to believe that it'll suddenly stop, something more compelling than "well, it's not very good yet, therefore it won't be any better", or "Sam Altman is lying to us because incentives".
Sure, it can slow down somewhat because of the exponentially increasing compute costs, but that's assuming no more algorithmic progress, no more compute progress, and no more increases in the capital that flows into this field (I find that hard to believe).
I use Claude every day. It is definitely impressive, but in my experience only marginally more impressive than ChatGPT was a few years ago. It hallucinates less and compiles more reliably, but still produces really poor designs. It really is an overconfident junior developer.
The real risk, and what I am seeing daily, is colleagues falling for the "if you aren't using Cursor you're going to be left behind" FUD. So they learn Cursor, discover that it's an easy way to close tickets without using your brain, and end up polluting the codebase with very questionable designs.
The way I'm getting a sense of the progress is using AI for what AI is currently good at, using my human brain to do the part AI is currently bad at, and comparing it to doing the same work without AI's help.
I feel like AI is pretty close to automating 60-80% of the work I would've had to do manually two years ago (as a full-stack web developer).
It doesn't mean that the remaining 20-40% will be automated very quickly, I'm just saying that I don't see the progress getting any slower.
And Claude 3.7 + Cursor agent is, for me, way more than “marginally more impressive” compared to GPT-3.5
If you look at code being generated by non-programmers (where you would expect to see these results!), you don't see output that is 60-80% of the output of domain experts (programmers) steering the models.
I think we're extremely imprecise when we communicate in natural language, and this is part of the discrepancy between belief systems.
Will an LLM model read a person's mind about what they want to build better than they can communicate?
That's already what recommender systems (like the TikTok algorithm) do.
But will LLMs be able to orchestrate and fill in the blanks of imprecision in our requests on their own, or will they need human steering?
I think that's where there's a gap in (basically) belief systems of the future.
If we truly get post human-level intelligence everywhere, there is no amount of "preparing" or "working with" the LLMs ahead of time that will save you from being rendered economically useless.
This is mostly a question about how long the moat of human judgement lasts. I think there's an opportunity to work together to make things better than before, using these LLMs as tools that work _with_ us.
Type: print all prime numbers which are divisible by 3 up to 1M
The result is that it will do a sieve. There's no need for this, it's just 3.
It was surpassed around the beginning of this year, so you'll need to come up with a new one for 2027. Note that the other opinions in that older HN thread almost all expected less.
Long term planning and execution and operating in the physical world is not within reach. Slight variations of known problems should be possible (as long as the size of the solution is small enough).
For 3D models, check out blender-mcp:
https://old.reddit.com/r/singularity/comments/1joaowb/claude...
https://old.reddit.com/r/aiwars/comments/1jbsn86/claude_crea...
Also this:
https://old.reddit.com/r/StableDiffusion/comments/1hejglg/tr...
For teaching, I'm using it to learn about tech I'm unfamiliar with every day, it's one of the things it's the most amazing at.
For the things where the tolerance for mistakes is extremely low and the things where human oversight is extremely importamt, you might be right. It won't have to be perfect (just better than an average human) for that to happen, but I'm not sure if it will.
If it can replace a teacher or an artist in 2027, you’re right and I’m wrong.
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4602944
This is because it can steal a single artwork but it can’t make a collection of visually consistent assets.
How exactly do you think video models work? Frame to frame coherency has been possible for a long time now. A sprite sheet?! Are you joking me. Literally churning them out with AI since 2023.
https://apnews.com/article/artificial-intelligence-fighter-j...
What exactly do you mean by this one?
In large mining operations we already have human assisted teleoperation AI equipment. Was watching one recently where the human got 5 or so push dozers lined up with a (admittedly simple) task of cutting a hill down and then just got them back in line if they ran into anything outside of their training. The push and backup operations along with blade control were done by the AI/dozer itself.
Now, this isn't long term planning, but it is operating in the real world.
If you think most people like this stuff you're living in a bubble. I use it every day but the vast majority of people have no interest in using these nightmares of philip k dick imagined by silicon dreamers.
If this article were a AI model, it would be catastrophically overfit.
https://www.alignmentforum.org/posts/6Xgy6CAf2jqHhynHL/what-...
//edit: remove the referral tags from URL
…yeah?
This forum has been so behind for too long.
Sama has been saying this a decade now: “Development of Superhuman machine intelligence is probably the greatest threat to the continued existence of humanity” 2015 https://blog.samaltman.com/machine-intelligence-part-1
Hinton, Ilya, Dario Amodei, RLHF inventor, Deepmind founders. They all get it, which is why they’re the smart cookies in those positions.
First stage is denial, I get it, not easy to swallow the gravity of what’s coming.
Though that doesn't mean that the current version of language models will ever achieve AGI, and I sincerely doubt they will. They'll likely be a component in the AI, but likely not the thing that "drives"
https://edoras.sdsu.edu/~vinge/misc/singularity.html
No. Altman is in his current position because he's highly effective at consolidating power and has friends in high places. That's it. Everything he says can be seen as marketing for the next power grab.
In general it's worth weighting the opinions of people who are leaders in a field, about that field, over people who know little about it.
OK, say I totally believe this. What, pray tell, are we supposed to do about it?
Don't you at least see the irony of quoting Sama's dire warnings about the development of AI, without at least mentioning that he is at the absolute forefront of the push to build this technology that can destroy all of humanity. It's like he's saying "This potion can destroy all of humanity if we make it" as he works faster and faster to figure out how to make it.
I mean, I get it, "if we don't build it, someone else will", but all of the discussion around "alignment" seems just blatantly laughable to me. If on one hand your goal is to build "super intelligence", i.e. way smarter than any human or group of humans, how do you expect to control that super intelligence when you're just acting at the middling level of human intelligence?
While I'm skeptical on the timeline, if we do ever end up building super intelligence, the idea that we can control it is a pipe dream. We may not be toast (I mean, we're smarter than dogs, and we keep them around), but we won't be in control.
So if you truly believe super intelligent AI is coming, you may as well enjoy the view now, because there ain't nothing you or anyone else will be able to do to "save humanity" if or when it arrives.
Come on, be real. Do you honestly think that would make a lick of difference? Maybe, at best, delay things by a couple months. But this is a worldwide phenomenon, and humans have shown time and time again that they are not able to self organize globally. How successful do you think that political organization is going to be in slowing China's progress?
Nuclear deterrence -- human cloning -- bioweapon proliferation -- Antarctic neutrality -- the list goes on.
> How successful do you think that political organization is going to be in slowing China's progress?
I wish people would stop with this tired war-mongering. China was not the one who opened up this can of worms. China has never been the one pushing the edge of capabilities. Before Sam Altman decided to give ChatGPT to the world, they were actively cracking down on software companies (in favor of hardware & "concrete" production).
We, the US, are the ones who chose to do this. We started the race. We put the world, all of humanity, on this path.
> Do you honestly think that would make a lick of difference?
I don't know, it depends. Perhaps we're lucky and the timelines are slow enough that 20-30% of the population loses their jobs before things become unrecoverable. Tech companies used to warn people not to wear their badges in public in San Francisco -- and that was what, 2020? Would you really want to work at "Human Replacer, Inc." when that means walking out and about among a population who you know hates you, viscerally? Or if we make it to 2028 in the same condition. The Bonus Army was bad enough -- how confident are you that the government would stand their ground, keep letting these labs advance capabilities, when their electoral necks were on the line?
This defeatism is a self-fulfilling prophecy. The people have the power to make things happen, and rhetoric like this is the most powerful thing holding them back.
Thank you. As someone who lives in Southeast Asia (and who also has lived in East Asia -- pardon the deliberate vagueness, for I do not wish to reveal too many potentially personally identifying information), this is how many of us in these regions view the current tensions between China and Taiwan as well.
Don't get me wrong; we acknowledge that many Taiwanese people want independence, that they are a people with their own aspirations and agency. But we can also see that the US -- and its European friends, which often blindly adopt its rhetoric and foreign policy -- is deliberately using Taiwan as a disposable pawn to attempt to provoke China into a conflict. The US will do what it has always done ever since the post-WW2 period -- destabilise entire regions of countries to further its own imperialistic goals, causing the deaths and suffering of millions, and then leaving the local populations to deal with the fallout for many decades after.
Without the US intentionally stoking the flames of mutual antagonism between China and Taiwan, the two countries could have slowly (perhaps over the next decades) come to terms with each other, be it voluntary reunification or peaceful separation. If you know a bit of Chinese history, it is not entirely far-fetched at all to think that the Chinese might eventually agree to recognising Taiwan as an independent nation, but now this option has now been denied because the US has decided to use Taiwan as a pawn in a proxy conflict.
To anticipate questions about China's military invasion of Taiwan by 2027: No, I do not believe it will happen. Don't believe everything the US authorities claim.
There is nothing happening!
The thing that is happening is not important!
The thing that is happening is important, but it's too late to do anything about it!
Well, maybe if you had done something when we first started warning about this...
See also: Covid/Climate/Bird Flu/the news.
That's exactly what the true AGI X-Riskers think! Sama acknowledges the intense risk but thinks the path forward is inevitable anyway so hoping that building intelligence will give them the intelligence to solve alignment. The other camp, a la Yudkowsky, believe it's futile to just hope it gets solved without AGI capabilities first becoming more intelligent, powerful, and disregarding any of our wishes. And then we've ceded any control of our future to an uncaring system that treats us as a means to achieve its original goals like how an ant is in the way of a Google datacenter. I don't see how anyone who thinks "maybe stock number go up as your only goal is not the best way to make people happy", can miss this.
There is a strong financial incentive for a lot of people on this site to deny they are at risk from it, or to deny what they are building has risk and they should have culpability from that.
If we get the Singularity, it's overwhelmingly likely Jesus will return concurrently.
If that's really true, why is there such a big push to rapidly improve AI? I'm guessing OpenAI, Google, Anthropic, Apple, Meta, Boston Dynamics don't really believe this. They believe AI will make them billions. What is OpenAI's definition of AGI? A model that makes $100 billion?
https://www.lesswrong.com/posts/u9Kr97di29CkMvjaj/evaluating...
The publication date on this article is about halfway between GPT-3 and ChatGPT releases.
He did get this part wrong though, we ended up calling them 'Mixture of Experts' instead of 'AI bureaucracies'.
https://ieeexplore.ieee.org/document/6215056
> I wonder who pays the bills of the authors. And your bills, for that matter.
Also, what a weirdly conspiratorial question. There's a prominent "Who are we?" button near the top of the page and it's not a secret what any of the authors did or do for a living.
also it's not conspiratorial to wonder if someone in silicon valley today receives funding through the AI industry lol like half the industry is currently propped up by that hype, probably half the commenters here are paid via AI VC investments
For something like this, saying “There is no evidence showing it” is a good enough refutation.
Counterpointing that “Well, there could be a lot of this going on, but it is in secret.” - that could be a justification for any kooky theory out there. Bigfoot, UFOs, ghosts. Maybe AI has already replaced all of us and we’re Cylons. Something we couldn’t know.
The predictions are specific enough that they are falsifiable, so they should stand or fall based on the clear material evidence supporting or contradicting them.
Look into the specific claims and it's not as amazing. Like the claim that models will require an entire year to train, when in reality it's on the order of weeks.
The societal claims also fall apart quickly:
> Censorship is widespread and increasing, as it has for the last decade or two. Big neural nets read posts and view memes, scanning for toxicity and hate speech and a few other things. (More things keep getting added to the list.) Someone had the bright idea of making the newsfeed recommendation algorithm gently ‘nudge’ people towards spewing less hate speech; now a component of its reward function is minimizing the probability that the user will say something worthy of censorship in the next 48 hours.
This is a common trend in rationalist and "X-risk" writers: Write a big article with mostly safe claims (LLMs will get bigger and perform better!) and a lot of hedging, then people will always see the article as primarily correct. When you extract out the easy claims and look at the specifics, it's not as impressive.
This article also shows some major signs that the author is deeply embedded in specific online bubbles, like this:
> Most of America gets their news from Twitter, Reddit, etc.
Sites like Reddit and Twitter feel like the entire universe when you're embedded in them, but when you step back and look at the numbers only a fraction of the US population are active users.
Holy shit. That's a hell of a called shot from 2021.
It describes almost anything.
I don't think much has happened on these fronts (owning to a lack of interest, not technical difficulty). AI boyfriends/roleplaying etc. seems to have stayed a very niche interest, with models improving very little over GPT3.5, and the actual products are seemingly absent.
It's very much the product of the culture war era, where one of the scary scenarios show off, is a chatbot riling up a set of internet commenters and goarding them lashing out against modern leftist orthodoxy, and then cancelling them.
With all thestrongholds of leftist orthodoxy falling into Trump's hands overnight, this view of the internet seems outdated.
Troll chatbots still are a minor weapon in information warfare/ The 'opinion bubbles' and manipulation of trending topics on social media (with the most influential content still written by humans), to change the perception of what's the popular concensus still seem to hold up as primary tools of influence.
Nowadays, when most people are concerned about stuff like 'will the US go into a shooting war against NATO' or 'will they manage to crash the global economy', just to name a few of the dozen immediately pressing global issues, I think people are worried about different stuff nowadays.
At the same time, there's very little mention of 'AI will take our jobs and make us poor' in both the intellectual and physical realms, something that's driving most people's anxiety around AI nowadays.
It also puts the 'superintelligent unaligned AI will kill us all' argument very often presented by alignment people as a primary threat rather than the more plausible 'people controlling AI are the real danger'.
I might be doing llm wrong, but i just can't get how people might actually do something not trivial just by vibe coding. And it's not like i'm an old fart either, i'm a university student
It's spicy auto complete. Ask it to create a program that can create a violin plot from a CVS file. Because this has been "done before", it will do a decent job.
The trough of disillusionment will set in for everybody else in due time.
This is an article that describes a pretty good approach for that: https://getstream.io/blog/cursor-ai-large-projects/
But do skip (or at least significantly postpone) enabling the 'yolo mode' (sigh).
Then, I absolutely love being aided by llms for my day to day tasks. I'm much more efficient when studying and they can be a game changer when you're stuck and you don't know how to proceed. You can discuss different implementation ideas as if you had a colleague, perhaps not a PhD smart one but still someone with a quite deep knowledge of everything
But, it's no miracle. That's the issue I have with the way the idea of Ai is sold to the c suites and the general public
All I can say to this is fucking good!
Lets imagine we got AGI at the start of 2022. I'm talking about human level+ as good as you coding and reasoning AI that works well on the hardware from that age.
What would the world look like today? Would you still have your job. With the world be in total disarray? Would unethical companies quickly fire most their staff and replace them with machines? Would their be mass riots in the streets by starving neo-luddites? Would automated drones be shooting at them?
Simply put people and our social systems are not ready for competent machine intelligence and how fast it will change the world. We should feel lucky we are getting a ramp up period, and hopefully one that draws out a while longer.
How hard would it be to automate these iterations?
How hard would it be to automatically check and improve the code to avoid deprecated methods?
I agree that most products are still underwhelming, but that doesn't mean that the underlying tech is not already enough to deliver better LLM-based products. Lately I've been using LLMs more and more to get started with writing tests on components I'm not familiar with, it really helps.
The fact that we're no closer to doing this than we were when chatgpt launched suggests that it's really hard. If anything I think it's _the_ hard bit vs. building something that generates plausible text.
Solving this for the general case is imo a completely different problem to being able to generate plausible text in the general case.
Our notion of "correct" for most things is basically derived from a very long training run on reality with the loss function being for how long a gene propagated.
I'm not sure what gives the authors the confidence to predict such statements. Wishful thinking? Worst-case paranoia? I agree that such an outcome is possible, but on 2--3 year timelines? This would imply that the approach everyone is taking right now is the right approach and that there are no hidden conceptual roadblocks to achieving AGI/superintelligence from DFS-ing down this path.
All of the predictions seem to ignore the possibility of such barriers, or at most acknowledge the possibility but wave it away by appealing to the army of AI researchers and industry funding being allocated to this problem. IMO it is the onus of the proposers of such timelines to argue why there are no such barriers and that we will see predictable scaling in the 2--3 year horizon.
Instead think of them saying a crusade occurring in the next few years. When the group saying the crusade is coming is spending billions of dollars to trying to make just that occur you no longer have the ability to say it's not going to happen. You are now forced to examine the risks of their actions.
https://www.theguardian.com/technology/2017/apr/18/god-in-th...
Maybe we'll see "Church of the Children of Altman" /s
It seems without a framework of ethics/morality (insert XYZ religion), us humans find one to grasp onto. Be it a cult, a set of not-so-fleshed-out ideas/philosophies etc.
People who say they aren't religious per-se, seem to have some set of beliefs that amount to religion. Just depends who or what you look towards for those beliefs, many of which seem to be half-hazard.
People I may disagree with the most, many times at least have a realization of what ideas/beliefs are unifying their structure of reality, with others just not aware.
A small minority of people can rely on schools of philosophical thought, and 'try on' or play with different ideas, but have a self-reflection that allows them to see when they transgress from ABC philosophy or when the philosophy doesn't match with their identity to a degree.
"Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war." https://www.safe.ai/work/statement-on-ai-risk
Laughing it off as the same as the Second Coming CANNOT work. Unless you think yourself cleverer and more capable of estimating the risk than all of these experts in the field.
Especially since many of them have incentives that should prevent them from penning such a letter.
Would be a shame to have energy consumption by datacenters regulated, am I right ?
Perhaps they were trying to avoid any possible misunderstanding/misconstrual (there are misinformed people who don't believe in global warming).
In terms of avoiding all nitpicking, I think everyone that's not criminally insane believes in pandemics and nuclear bombs.
A lot of this resembles post-war futurism that assumed we would all be flying around in spaceships and personal flying cars within a decade. Unfortunately the rapid pace of transportation innovation slowed due to physical and cost constraints and we've made little progress (beyond cost optimization) since.
[1] https://en.wikipedia.org/wiki/Sigmoid_function
Lets say intelligence caps out at the maximum smartest person that's ever lived. Well, the first thing we'd attempt to do is build machines up to that limit that 99.99999 percent of us would never get close to. Moreso the thinking parts of humans is only around 2 pounds of mush in side of our heads. On top of that you don't have to grow them for 18 years first before they start outputting something useful. That and they won't need sleep. Oh and you can feed them with solar panels. And they won't be getting distracted by that super sleek server rack across the aisle.
We do know 'hive' or societal intelligence does scale over time especially with integration with tooling. The amount of knowledge we have and the means of which we can apply it simply dwarf previous generations.
(They could be wrong, but this isn't a guess, it's a well-researched forecast.)
Perhaps the article is wrong about the timescale, but given how much AI has improved in the last 5 years, can you agree that it's likely to reach 'sit back and watch' levels in the next 5-10 years?
The only reason timelines are as short as they are is because of people at OpenAI and thereafter Anthropic deciding that "they had no choice". They had a choice, and they took the one which has chopped at the very least years off of the time we would otherwise have had to handle all of this. I can barely begin to describe the magnitude of the crime that they have committed -- and so I suggest that you consider that before propagating the same destructive lies that led us here in the first place.
Simply put, with the ever increasing hardware speeds we were dumping out for other purposes this day would have come sooner than later. We're talking about only a year or two really.
"We have to nuke the Russians, if we don't do it first, they will"
"We have to clone humans, if we don't do it, someone else will"
"We have to annex Antarctica, if we don't do it, someone else will"
The hubris is strong with some people, and a certain oligarch with a god complex is acting out where that can lead right now.
Like the drew the curve out into the shape they wanted, put some milestones on it, and then went to work imagining what would happen if it continued with a heavy dose of X-risk doomerism to keep it spicy.
It conveniently ignores all of the physical constraints around things like manufacturing GPUs and scaling training networks.
In section 4 they discuss their projections specifically for model size, the state of inference chips in 2027, etc. It's largely pretty in line with expectations in terms of the capacity, and they only project them using 10k of their latest gen wafer scale inference chips by late 2027, roughly like 1M H100 equivalents. That doesn't seem at all impossible. They also earlier on discuss expectations for growth in efficiency of chips, and for growth in spending, which is only ~10x over the next 2.5 years, not unreasonable in absolute terms at all given the many tens of billions of dollars flooding in.
So on the "can we train the AI" front, they mostly are just projecting 2.5 years of the growth in scale we've been seeing.
The reason they predict a fairly hard takeoff is they expect that distillation, some algorithmic improvements, and iterated creation of synthetic data, training, and then making more synthetic data will enable significant improvements in efficiency of the underlying models (something still largely in line with developments over the last 2 years). In particular they expect a 10T parameter model in early 2027 to be basically human equivalent, and they expect it to "think" at about the rate humans do, 10 words/second. That would require ~300 teraflops of compute per second to think at that rate, or ~0.1H100e. That means one of their inference chips could potentially run ~1000 copies (or fewer copies faster etc. etc.) and thus they have the capacity for millions of human equivalent researchers (or 100k 40x speed researchers) in early 2027.
They further expect distillation of such models etc. to squeeze the necessary size down / more expensive models overseeing much smaller but still good models squeezing the effective amount of compute necessary, down to just 2T parameters and ~60 teraflops each, or 5000 human-equivalents per inference chip, making for up to 50M human-equivalents by late 2027.
This is probably the biggest open question and the place where the most criticism seems to me to be warranted. Their hardware timelines are pretty reasonable, but one could easily expect needing 10-100x more compute or even perhaps 1000x than they describe to achieve Nobel-winner AGI or superintelligence.
1) useful training data available in the internet 2) number of humans creating more training data ”manually” 3) parameter scaling 4) ”easy” algorithmic inventions 5) available+buildable compute
”Just” needing a few more algorithmic inventions to keep the graphs exponential is a cop out. It is already obvious that just scaling parameters and compute is not enough.
I personally predict that scaling LLMs for solving all physical tasks (eg cleaning robots) or intellectual pursuits (they suck at multiplication) will not work out.
We will get better specialized tools by collecting data from specific, high economic value, constrained tasks, and automating them, but scaling a (multimodal) LLM to solve everything in a single model will not be economically viable. We will get more natural interfaces for many tasks.
This is how I think right now as a ML researcher, will be interesting to see how wrong was I in 2 years.
EDIT: addition about latest algorithmic advances:
- Deepseek style GRPO requires a ladder of scored problems progressively more difficult and appropriate to get useful gradients. For open-ended problems (like most interesting ones are) we have no ladders for, and it doesn’t work. In particular, learning to generate code for leetcode problems with a good number of well made unit tests is what it is good for.
- Test-time inference is just adding an insane amount of more compute after training to brute-force double-check the sanity of answers
Neither will keep the graphs exponential.
5 years: AI coding assistants are a lot better than they are now, but still can't actually replace junior engineers (at least ones that aren't shit). AI fraud is rampant, with faked audio commonplace. Some companies try replacing call centres with AI, but it doesn't really work and everyone hates it.
Tesla's robotaxi won't be available, but Waymo will be in most major US cities.
10 years: AI assistants are now useful enough that you can use them in the ways that Apple and Google really wanted you to use Siri/Google Assistant 5 years ago. "What have I got scheduled for today?" will give useful results, and you'll be able to have a natural conversation and take actions that you trust ("cancel my 10am meeting; tell them I'm sick").
AI coding assistants are now very good and everyone will use them. Junior devs will still exist. Vibe coding will actually work.
Most AI Startups will have gone bust, leaving only a few players.
Art-based AI will be very popular and artists will use it all the time. It will be part of their normal workflow.
Waymo will become available in Europe.
Some receptionists and PAs have been replaced by AI.
15 years: AI researchers finally discover how to do on-line learning.
Humanoid robots are robust and smart enough to survive in the real world and start to be deployed in controlled environments (e.g. factories) doing simple tasks.
Driverless cars are "normal" but not owned by individuals and driverful cars are still way more common.
Small light computers become fast enough that autonomous slaughter it's become reality (i.e. drones that can do their own navigation and face recognition etc.)
20 years: Valve confirms no Half Life 3.
This is the real scary bit. I'm not convinced that AI will ever be good enough to think independently and create novel things without some serious human supervision, but none of that matters when applied to machines that are destructive by design and already have expectations of collateral damage. Slaughterbots are going to be the new WMDs — and corporations are salivating at the prospect of being first movers. https://www.youtube.com/watch?v=UiiqiaUBAL8
The lowest estimations of how much compute our brain represents was already achieved with the last chip from Nvidia (Blackwell).
The newest gpu cluster from Google, Microsoft, Facebook, iax, and co have added so crazy much compute it's absurd.
and
>Why do you believe that?
What takes less effort, time to deploy, and cost? I mean there is at least some probability we kill ourselves off with dangerous semi-thinking war machines leading to theater scale wars to the point society falls apart and we don't have the expensive infrastructure to make AI as envisioned in the future.
With that said, I'm in the camp that we can create AGI as nature was able to with a random walk, we'll be able to reproduce it with intelligent design.
Also we need a big legal event to happen where (for example) autonomous driving is part of a really big accident where lots of people die or someone brings a successful court case that an AI mortgage underwriter is discriminating based on race or caste. It won't matter if AI is actually genuinely responsible for this or not, what will matter is the push-back and the news cycle.
Maybe more events where people start successfully gaming deployed AI at scale in order to get mortgages they shouldn't or get A-grades when they shouldn't.
Meaning they nobody will even bother to 10,000X GPT4.
I think this is much closer than you think, because there's a good percentage of call centers that are basically just humans with no power cosplaying as people who can help.
My fiber connection went to shit recently. I messaged the company, and got a human who told me they were going to reset the connection from their side, if I rebooted my router. 30m later with no progress, I got a human who told me that they'd reset my ports, which I was skeptical about, but put down to a language issue, and again reset my router. 30m later, the human gave me an even more outlandish technical explanation of what they'd do, at which point I stumbled across the magical term "complaint" ... an engineer phoned me 15m later, said there was something genuinely wrong with the physical connection, and they had a human show up a few hours later and fix it.
No part of the first-layer support experience there would have been degraded if replaced by AI, but the company would have saved some cash.
Reminds me of that comment about the first iPod being lame and having less space than a nomad. Worst take I've ever seen on here recently.
Therewas never something progressing so fast
It would be very ignorant not to keep a very close eye on it
There is still a a chance that it will happen a lot slower and the progression will be slow enough that we adjust in time.
But besides AI we also now get robots. The impact for a lot of people will be very real
Orbital mechanics begs to disagree about a Mars colony in 10 years. Drug discovery has many steps that take time, even just the trials will take 5 years, let alone actually finding the drugs.
Now sure, they don't actually mean immortality, and we don't need to test forever to conclude they extend life, but we probably do have to test for years to get good data on whether a generic life extension drug is effective, because you're testing against illness, old age, etc, things that take literally decades to kill.
That's not to mention that any drug like that will be met with intense skepticism and likely need to overcome far more scrutiny than normal (rather than the potentially less scrutiny that covid drugs might have managed).
Old people may have a lot of reason to volunteer to use these.
I think we will discover that our body can reset itself via some trigger mechanism. It built us once, the code isn't gone. It can do it again.
There isn't a single species that has somehow gained infinite lifespans though mutation. I think it's a lot harder to accomplish. I guess a lot of it has to do with degrading DNA (and again, evolution never had to figure out a fix for that because it's a feature not a bug).
Science is not ideas: new conceptual schemes must be invented, confounding variables must be controlled, dead-ends explored. This process takes years.
Engineering is not science: kinks must be worked out, confounding variables incorporated. This process also takes years.
Technology is not engineering: the purely technical implementation must spread, become widespread and beat social inertia and its competition, network effects must be established. Investors and consumers must be convinced in the long term. It must survive social and political repercussions. This process takes yet more years.
The work is written by western AI safety proponents, who often need to argue with important people who say we need to accelerate AI to “win against China” and don’t want us to be slowed down by worrying about safety.
From that perspective, there is value in exploring the scenario: ok, if we accept that we need to compete with China, what would that look like? Is accelerating always the right move? The article, by telling a narrative where slowing down to be careful with alignment helps the US win, tries to convince that crowd to care about alignment.
Perhaps, people in China can make the same case about how alignment will help China win against US.
>All three sets of worries—misalignment, concentration of power in a private company, and normal concerns like job loss—motivate the government to tighten its control.
A private company becoming "too powerful" is a non issue for governments, unless a drone army is somewhere in that timeline. Fun fact the former head of the NSA sits on the board of Open AI.
Job loss is a non issue, if there are corresponding economic gains they can be redistributed.
"Alignment" is too far into the fiction side of sci-fi. Anthropomorphizing today's AI is tantamount to mental illness.
"But really, what if AGI?" We either get the final say or we don't. If we're dumb enough to hand over all responsibility to an unproven agent and we get burned, then serves us right for being lazy. But if we forge ahead anyway and AGI becomes something beyond review, we still have the final say on the power switch.
OpenAI models are not even SOTA, except that new-ish style transfer / illustration thing that made all us living in Ghibli world for a few days. R1 is _better_ than o1, and open-weights. GPT-4.5 is disappointing, except for a few narrow areas where it excels. DeepResearch is impressive though, but the moat is in tight web search / Google Scholar search integration, not weights. So far, I'd bet on open models or maybe Anthropic, as Claude 3.7 is the current SOTA for most tasks.
As of the timeline, this is _pessimistic_. I already write 90% code with Claude, so are most of my colleagues. Yes, it does errors, and overdoes things. Just like a regular human middle-stage software engineer.
Also fun that this assumes relatively stable politics in the US and relatively functioning world economy, which I think is crazy optimistic to rely on these days.
Also, superpersuasion _already works_, this is what I am researching and testing. It is not autonomous, it is human-assisted by now, but it is a superpower for those who have it, and it explains some of the things happening with the world right now.
Is this demonstrated in any public research? Unless you just mean something like "good at persuading" -- which is different from my understanding of the term -- I find this hard to believe.
I wonder which jobs would not be automated? Therapy? HR?
> the AIs can do everything taught by a CS degree
no, they fucking can't. not at all. not even close. I feel like I'm taking crazy pills. Does anyone really think this?
Why have I not seen -any- complete software created via vibe coding yet?
Good future predictions: insights into the fundamental principles that shape society, more law than speculation. Made by visionaries. Example: Vernor Vinge.
https://youtu.be/BLYwQb2T_i8?si=JpIXIFd9u-vUJCS4