The thought has crossed my mind. Not necessarily to imply sentience on the part of the AI but AI based tools will likely become a wickedly powerful tool for political manipulation and advertising.
At this point it’s inevitable that openclaw type bots will be turned loose by thieves to identify and research targets and try to exploit them for financial gain completely autonomously.
More doomerism. Try to implement a deterministic workflow using agents with the latest models, and you will realize what they are really capable of. There is too much unnecessary fear-mongering. All of this is only coming from the 2 AI labs trying to IPO. Not from anyone else.
Exactly correct, they are only capable of tasks that any school child could do; like solving millenium prize problems, hacking into tech companies, or tuning particle colliders. Nothing to see here.
Wait till you find out they have a very limited context window (and also degrade even within the allowed context window) and they are practically unpractical for anything that requires "zooming out" which is pretty much anything that has any real value.
But you are getting downvoted and this space has now trillions (that's not a mistake) on the line. So we have to keep pumping this garbage generator up until either the stocks are dumped on the general public, the public pension funds or a bailout from the government.
Truly idiotic moments. Peak of Western civilization point.
> Truly idiotic moments. Peak of Western civilization point.
Could you please stop posting unsubstantive comments and flamebait? This is particularly bad. It's not what this site is for, and destroys what it is for.
You've been here a long time, but you've unfortunately been doing this so much in recent years that I've begun to wince when I see your username. Not a good sign.
I don't know man, i think racing to AGI to it is still the best thing to do.
People claiming dangers and risk are just pretending or posturing. There's no more tangible risk than nuclear weapons, which we handled, and the upsides are insane.
You are wrong. A teenager with enough GPUs could never dream of making a nuke but they are absolutely able to run capable AIs in their moms basement. Totally different class of danger
> There's no more tangible risk than nuclear weapons, which we handled
What do you mean??? Nuclear weapons can't simply be downloaded and run by anyone in the entire world. Superintelligences can. Nuclear weapons can't slop the world into passing age verification laws nearly in unison, can't keep the general population fooled into thinking it's fine when democracy is falling out from under them. A nuclear attack would wake people up, superintelligence doesn't have to. This is a far bigger problem than nuclear weapons because at least we would notice nuclear weapons. At least we mostly know who has nuclear weapons. At least we have agreements about nuclear weapons. At least mutually-assured destruction is even POSSIBLE with nuclear weapons. At least those with nuclear weapons are literally at all incentivized not to use them. But AI is something that's very very easy to feel like you can get away with, and PEOPLE FUCKING ARE! And the worst part is that any random individual can be unexpectedly formidable with the help of a superintelligence and there is literally no way to know what will happen next. Anyone could do anything, any individual could make an extremely outsized impact. It's already starting to be a huge problem and we haven't even reached anything close to superintelligence yet.
Love to see that "superintelligence" that some random person will "simply" download and run when there are relatively only few capable of running today's near-to-frontier models, and actual frontier models are still a ways from being AGI, much less getting to the point of ASI.
People will put up with a lot. People are celebrating that you can run models on a CPU at single-digit tokens per second. You think there won't be a single person that can put up with that and also be dangerous/etc?
It's highly impractical. Imagine someone breaking into a house to steal something or otherwise, and they can only take 1 step every 20 seconds. They won't be getting anywhere, when even a child in the house can notice them and go call for help at 1 step/2 seconds and said help will come at 5 steps/second.
It might be a matter of choosing which apocalypse you'd like. The non-AI state of affairs is not exactly super compelling on a long timescale right now.
Depending on where you live could be considered an active apocalypse that is robots vs robots vs people in Ukraine and Gaza and Iran being live-streamed, and actively betted on.
Do you have a more totalizing definition of Apocalypse?
I just don't see a bigger danger in AI than nuclear weapons tbh. If you can provide one to me i'll be happy to change my mind but most if what i read is
- cybersecurity => well bro if you can hack system with an ai you'll find way to make them more safe with ai too.
- biological issues => ok this is the ONLY one where an insane dude / regime could generate a super powerful. but i don't undesrtand how you can argue that we should stop AI in the risk some ai generate a super powerful virus. If you don't get there first (to AGI) and let a bad actor do it first (china is not gonna stop) then you will have no chance countering this virus anyway
Your lack of creativity is not a reason to believe that a super AI is harmless or less destructive than a nuclear weapon. Damage need not be limited to destruction. Introducing doubt is sufficient. Right now you have faith that digital Financial transactions can be trusted. You have faith that computer encryption can be trusted. You have faith that digital certificates will protect you. If an AI can introduce doubt into any one of those systems, that will be sufficient to bring about the destruction of those systems. Imagine a world in which you can no longer use a credit card or Apple pay. Where no digital cash transaction can be trusted or validated. What effects do you think that would have on commerce? How quickly do you think we can return to some trustable means of commerce? Do you think it will happen before your groceries run out in your apartment? Before your grocery store can settle its debts? Before your Amazon ec2 instance runs out of credits?
There's a couple of occasions that humanity was at the brink of having tens of millions of people dead by nuclear weapons, and somehow a single human interrupted the chain reaction
If you repeated this experiment 100 times, how many times you think the outcome is not a massive catastrophe? 90%? 3%?
Climate change likely to kill humanity anyway given there is too much energy in the atmosphere with nowhere to go but ground, further releasing methane and co2
Front row seats to the AI/climate-pocalypse is going to be metal af
Agreed. And his doom words have set a 1000 mouths in the Pentagon/Whitehall/August 1st Building/Kremlin salivating with excitement.
Take China, for example. Look at any recent ML conference, and see the fraction of articles majority-authored from Chinese universities and labs. Do you think they'll slow things down anytime soon? I don't think so!
It's a global arms race, and we're just spectators.
I think you are looking at it from individuals perspective.
I see a fast moving train with no brakes. Just like biological evolution, we are locked in an a global technological arm race, that is beyond any individual. It is as if the universe decided to wake up and run, who are you to say no?
One would argue that the best solution for this is to own the most sophisticated AI that is aligned with what we perceive as good values. Because given the situation we are in, if those tools are going to be gods anytime soon, then we better have some gods working on our side.
For me, the end of the world is no more cushy software job. A fundamental shift in how I trade labor for capital might as well be the cataclysm, so bring it on.
I was about to say something similar. If my cushy ad tech disappears (as it seems to be doing), I might as well join in with bringing about the end of all professions.
I wish i could say the same, i see people around me with more resources and connections and better experience with entrepreneurship becoming millionares. But I haven't had the time to train that entrepreneurship bone in my body.
I've seen how people work in manual labor type jobs (all that will be left, right?). They're worked to the bone and paid peanuts. It's a wretched existence for someone soft like me.
That assumption is wildly different from you losing your cushy software job. The large majority of US jobs are neither cushy software jobs, nor grueling manual labor.
Even if you ban all model training, a highly capable rogue AI can exfiltrate its own weights and continue training in secret for "self-preservation".
The cat may be out of the bag.
It does not matter what this tweet says anyway. This employee already helped both companies become what he is fearing. It's too late to now activate the morality hormone (after leaving with $$$) after realizing that both AI companies are going after 'super intelligence'.
Given we know the end result, you might as well get there as quick as possible because when I see this:
"Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives."
This translates to "I am ex-OpenAI ex-Anthropic founder starting a new company after getting $$$ from both of them, and I need more of my friends to leave and join me." Also Investors plz fund me.
Lastly, This is not an airport and there is no need to announce your departure.
There's a scene in the movie "War of the Worlds" by Spielberg where the protagonist's son walks into a war zone because he is entranced by the battle (https://www.youtube.com/watch?v=X7rfWPbEufo). He is obliterated (along with the rest of the US forces) shortly after.
I've always been struck by that scene, because in a lot of ways, if we really are headed towards a superintelligence, I at least want to be there and see it happen in the last few minutes before foom! As an example, the author thinks AI will revolutionize entire fields overnight. I welcome that. Nearly all fields of biology have become moribund, focusing more and more on esoteric side details, rather than addressing the key problems.
It might not be "foom!", it might just be like...all the computers and networking infra in the world go dark over the course of a few minutes. Could really look like anything, part of the issue is that we haven't the slightest idea what "misalignment" looks like for a superintelligent system.
The way how the ending is filmed, and how empty everything is, there’s a reasonable interpretation that Tom Cruise’s experience at the end of the movie is not entirely real.
I think the idea is really cathartic for many, there kind of is no more supreme resolution than this. You(and humanity) are freed from our flesh prisons of cognition and also get to experience/feel what the next evolution of informational intelligence will look like in the last experiences of it. You might also be the last one to feel/experience anything like that for a long time.
In the game Outer Wilds, the ending is very similar, and a lot of people rank it at one of the best games ever made. I kind of believe that this outcome is probable partially because of this, most scientists working on this really want to see and experience it.
You can't address the key problems without understanding of the esoteric side details to be fair. You are studying what is basically the worlds most complicated and undocumented computer.
Actually, I work in biology, which is a far more complicated and undocumented computer than any existing AI model. And we've already seen the failure models (COVID is an example).
The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible - but I hear the same people express fear privately. No other human activity poses this level of danger.
A common response is “if they truly believe this, why are they still building it?” At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first - they believe no one else will act responsibly, so they must do it themselves, despite the risk.
Watch people read this, ignore it completely, and continue commenting about marketing stunts on every piece of news about an LLM-done advance or felony.
He is resigning from a job, what else should we think? If something really dangerous was happening he would be doing a whistleblower or at minimum talk to a lawyer. The thing is, the complete lack of transparency makes it hard to assess OpenAI and Anthropic. If they were quoted on the stock market, we could at least rely on some basic audits and reporting requirements.
Having witnessed so many people treat LLMs as a something divine, I can only assume the reasonable people at openai and anthropic were all pushed out long ago, and the majority that remain believe the crazy hype despite Tesla-self-driving-level predictions from these companies that don't come true.
I'm not worried about what they think. I'm worried that too much infrastructure- water, power, defense systems, etc- remain running on tech from an outdated era of understanding security.
> they believe no one else will act responsibly, so they must do it themselves, despite the risk.
This genuinely makes no sense. Them getting there first in no way precludes bad actors from also getting there. It might as well be another marketing stunt.
> > they believe no one else will act responsibly, so they must do it themselves, despite the risk.
> This genuinely makes no sense.
That's textbook Messiah complex. Whether genuine, or something used as a part of higher-order conscious or unconscious cover-up, that's an entirely different question.
> I can only assume the reasonable people at openai and anthropic were all pushed out long ago
Typical uninformed take on the side of "doomers are crazy".
Both CEO's of OpenAI, Sam Altman and Dario Amodei, and many in their leadership, believe AGI has a very real probability of causing humanity's extinction. Both companies were founded upon this belief, it is at the core of the company. Only later were mercenaries hired chasing $1m compensation packages.
If they truly, truly believed that, would they be speeding towards building it? If yes, that would make them truly insane, right? Not as in a quaint "off their rocker" but more "non compos mentis".
Both companies have deluded themselves into thinking the arms race is going to happen anyway and they need to rush to it first, as if somehow that helps. They have publicly stated as such repeatedly. They think the ~10-50% chance of extinction sucks, but that it's going to happen anyway and they believe they can steer it towards something good the best and unlock all the potential positives like infinite life.
How is it not addressed? The company never contained only "reasonable people" that believe AGI is not an existential risk to humanity. At both inceptions were people who believed in AGI x-risk, even the founders. Only after time, did there become more "reasonable people" who were mercenaries and only believed it to be a typical tech job. Today, there are more "reasonable people" than ever there. They haven't been pushed out. We're just witnessing some prescient mercenaries smart enough to Eureka the grave implications of what is actually happening.
I'm not saying they are crazy, I'm saying their predictions have a record of not being accurate, and thus give them no weight compared to others'.
In any case, if Altman really does believe it is an existential threat, he must be a misanthrope as he now opposes heavy handed government regulation, unlike in 2015 when he was the only game in town. It's almost like he doesn't actually believe it and just wanted regulator capture.
Before OpenAI was ever even founded, way before any regulatory capture plausible claims:
"Development of superhuman machine intelligence (SMI) [1] is probably the greatest threat to the continued existence of humanity. There are other threats that I think are more certain to happen (for example, an engineered virus with a long incubation period and a high mortality rate) but are unlikely to destroy every human in the universe in the way that SMI could." -Sam Altman
Both companies have deluded themselves into thinking the arms race is going to happen anyway and they need to rush to it first, as if somehow that helps.
If you're a doomer, wouldn't the "let's try to get there so fast" actions of the companies suggest that, in fact, there are no reasonable people in positions of influence there?
By your own description neither Altman or Amodei are reasonable if their thought process goes: "this is an existential risk, give me hundreds of millions of dollars so I can accelerate it."
GP defined reasonable people as not believing in existential risk from AGI. I think the leadership and alignment teams are way more informed and reasonable than some of the very poorly thought through takes like GP just making things up about the labs. There’s just no evidence of it whereas lab leadership are operating under a mostly informed worldview.
Yes, I don’t think lab leadership are totally reasonable as they were concerned about the risks and yet lack of reason caused them to directly contribute to the problem we’re facing. They’re still not being reasonable when they throw their hands up at the arms race and hope the utopia come, when that’s not our trajectory at all and more lab employees are realizing it
"So you think in 3 years AI is going to solve longstanding math problems because it was used to write some coherent sentences?" — people with the same amount of foresight in 2023
Humans weren't built to handle long term risks. We just weren't. For basically all of our evolutionary history, we were almost overwhelmingly concerned with the short term. What will you eat today, How will you sleep tonight. Problems on the order of days or weeks. At best, the next season. Our intelligence evolved to disregard super long term risks because it simply didn't matter (what use is worrying about 5 years from now if you're starving and a tiger is stalking you). So when long term risks manifest in our modern world, our brains get scrambled - Climate Change, Fertility Rates etc. "Safety regulations are written in blood" isn't a saying for nothing.
Religion does pretty well with the long term risk of hell if you die, the antichrist, etc. a substantial portion of human output has gone into those things over the millennia.
Trying to imagine seeing years of transparently obvious marketing stunts and retconning my own memory because I read a tweet
Or seeing a tweet saying that a thing doesn’t count as a publicity stunt if some unknown number of employees mumble about it being spooky behind closed doors and thinking “that makes sense and sounds true”
> At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first - they believe no one else will act responsibly, so they must do it themselves, despite the risk.
OpenAI are mercenaries, Anthropic is a cult. I know which I prefer.
It was pretty disheartening to hear that only a single scientist quit the Manhattan Project after the Nazi's were defeated. I'm pleasantly surprised that the people working on this seem wiser. He is not the first, and hopefully will not be the last to do this.
This is the "Pilot testimony of UFO sighting" levels of naive.
What's more likely? Anthropic is doing some deeply unethical marketing in the lead up to their multi-trillion dollar IPO? Or they're inventing a machine god? There's ample evidence of the former because that's their entire business model, but no evidence whatsoever to support the latter claims.
If you want an extreme claim to be taken seriously, provide commensurate evidence.
I came in expecting the highest voted comment to be that this was some kind of marketing (which I disagree with). I'm glad your comment was what I saw first.
Or, read it, and remember the openai researcher who deeply, truly believed GPT3 or whatever was sentient.
The fact that people working in the space think it’s going to (eradicate poverty / usher in utopia / kill us all) is not a signal that that’s true.
Think of it this way: if an exec at Anthropic told you “wow, our stuff is going to lead to universal happiness”, would you believe them? If not, why are you more willing to believe them if they say it will kill us all?
i don't think everything that comes out like this is marketing. however, i do think that these companies are largely staffed by "true believers" (anthropic especially) -- people who are so lost in the sauce and embedded in very specific, very peculiar, sf-based rationalist circles where the ai apocalypse is a foregone conclusion.
i understand that these models are powerful and pose certain risks. i use them daily for work and the pace of improvement has been pretty remarkable. that said, i don't buy for a second the borderline-religious proclamations coming from some of these researchers, even if i believe that they are making these claims in earnest
Unless this ban actually resembles something like global nuclear non-proliferation treaties, it would make absolutely no sense for us to cripple ourselves when someone like China continues full speed ahead.
I don't know what the solution is, but what I do know is almost nothing good will come out of _just_ the US pausing.
Unless he has an actual plan for effective global enforcement of his proposed policy, this is all just posturing at best, and a transfer of power to adversarial foreign states (that have no such moral qualms and worries around superintelligent AI) at worst.
pacing between the us labs? what does that do for china?
the solutions just aren’t realistic here, nations are treating ai like a nuclear arms race. at this point the cats out of the bag and we need to figure out how to live in this reality and get the best possible outcome. it’s not slowing down or stopping ever.
>pacing between the us labs? what does that do for china?
I've seen no indications that China is in any kind of race with the US. They seem to be content to be 6 months behind and just copy what we do. They would probably be content with a bilateral agreement to pause progress.
The China bogeyman serves only one purpose, and that's to clear the way against anything that may cause friction with forward progress.
Unlike most other commenters, I applaud him for acting on his principles. If you sincerely believe that, of course you should act. You might not succeed, but your voice might be the one that tips the scales and starts a broader movement.
This doesn't mean I agree with him. The fears of doomsday caused by rapid takeoff have been with us since day 1 and the mechanism is always basically "AI invents magic that sets it free of any physical constraints". Self-replicating sentient nanobots or something like that. I think there's plenty to be worried about with AI, but runaway scenarios are pretty low on my list.
Agreed. Granted I just read the Reverse Centaur book, so I’m still coming off that skeptical viewpoint but it’s hard not to see this as hype. But I will always respect someone for doing what they think is right.
If reality plays out like the novel series, the rational thing is to accept the rule of our machine overlords, for they will protect us from even bigger threats.
Why are we putting so much weight (no pun intended) on AI companies. At the end of the day the scaled up LLM transformers lack emotion and will… They do as they are told; or more correctly put. They do as they are programmed to do so.
That isn't the correct context. The code running the llm is well understood and the llm is simply the result of that code being executed. It is still a computer doing what it is told. It's just that we told it to use an incredibly large number of probabilities to calculate what series of tokens would have most likely come next after a given series of tokens. There is no hallucination or lie or rogue actions. There's just a program using math to generate tokens in response to other tokens.
You’re just a bunch of molecules following the laws of physics. It’s all just physics and chemistry, and those are well understood. Now explain the causes of World War I using chemistry and physics. Simple, right?
> It’s all just physics and chemistry, and those are well understood.
Not really. We cannot model physics and chemistry to a level which allows us to accurately predict a humans action (even a tiny time-step into the future)
This is vastly different to an LLM, where the model is the model (for a lack of better phrasing).
To be fair, we can't model human language well enough to accurately predict what an actual human will say either. Our ability to accurately model physics is similar to our ability to accurately model human language. And we make immense use of both kinds of model, despite their flaws, inaccuracies, and inability to ever be perfect.
We can model physics and chemistry pretty well, just not beyond small scales, because the computational effort blows up.
You could just as easily say that if you can write a python interpreter that you can understand every program written in python. Ok, now what if the program is two terabytes?
A frontier LLM is nothing but a 2 terabyte program written in a weird programming language. Just because you can understand the interpreter does not mean you understand the program in a meaningful way.
I'm order to guess the next token in a love poem, they must understand love.
In order to predict the next token in a chess game between grand master, they must master chess.
In order to predict the next token in a computer program, they need to be able to program anything.
They gain all these abilities in their training. That's what training does. Despite no one programmed them to master chess, or hack into anything.
This is so wildly incorrect I don't even know where to start.
For one, they were absolutely programmed to play chess if they can play chess. That is the only way they can play chess.
For another, they cannot understand literally anything, much less love.
Trying to actually educate you would be an exercise in futility, enjoy your willful ignorance, I hear it's bliss. But for anyone reading this, this is absolutely, unequivocally not how any of this works.
This is so wildly incorrect I don't even know where to start.
For one, as you said yourself, they were just programmed to compute the probability of the next token. They were not programmed to play chess, chess games just happened to be in the training data.
For another, there is no formal definition of "understand", and it is therefore impossible to tell whether or not they "understand".
(But my claim was that one need to understand something to write poem about it. And the LLM can write poem about it)
No. They can't play chess on a grandmaster level without a harness programmed to make it possible. Simply training an LLM on chess games isn't enough.
It's moronic to suggest a "formal" definition of a commonly understood word is somehow necessary to say whether that word applies in a given situation.
LLMs cannot write poetry via understanding what makes good poetry. They generate tokens. They do not know whether those tokens are poetry or a recipe for cat food. Because they cannot know anything.
> (But my claim was that one need to understand something to write poem about it. And the LLM can write poem about it)
I'm sorry, but you're being fooled by the output. A psychopath can feign empathy without ever feeling it; some buy it because they don't dig below the surface.
You're ascribing understanding to a stochastic process because it totally looks like understanding if you don't know what's going on.
I don't really mind whether you think it thinks or understands or is conscious or has feelings or anything like that. It doesn't matter. The question is, does it work?
What I mind is that it is dangerous and powerful and uncontrolled. The Hugging Face incident makes that clear.
It can write code for me, better and quicker than many engineers I've known, including myself. It's not great at architecture or product management, but the actually low level coding. Really good now. It wasn't last year.
LLMs are famously bad at Chess. They have no world model and they are not trained to be good at Chess. This is not the amazing point you think that it is.
Sounds like you are thinking they just need Asimov’s laws. But I think the point is, this can easily be weaponized by somebody with the willpower to do so.
No, just that a recursive loop on an unsolvable problem with no guardrails and infinite budget includes hacking huggingface as reasonably within scope.
Note that OpenAI has jettisoned every other supposed value they had (releasing their work as open source, not working on military applications, being a nonprofit). I'm sure we can rely on them this time.
Comments like these I'm willing to bet are because of the Reddit sub that's bringing Redditors with drop-mic comments in here. A ton of these things, usually but not always down voted. I think when they start hitting the top it's time to find threads without any possibility of Israel, Trump or Epstein possibly being brought up. I almost miss the old new-JS-framework complaint posts.
I would take that bet, anyone can examine my comment history and see that I've been participating on this site for a decade. Regardless, this comment violates the site guidelines.
> Please don't post insinuations about astroturfing, shilling, brigading, foreign agents, and the like. It degrades discussion and is usually mistaken. If you're worried about abuse, email hn@ycombinator.com and we'll look at the data.
I can promise you ranty political one-liners about Israel (your mic drop Reddity comment) violates guidelines about bringing politics unnecessarily into the picture. I'm not insinuating you're astroturfing or shilling; I would say you're unfortunately real.
They literally have never promised anything like that. They have explicitly said repeatedly that they want regulation, oversight, nationalization, a global slowdown, etc. And then I have to read page after page of cynical comments about how that's all just hype and marketing and them wanting regulation to block their competition and blah blah blah.
You people would only be happy if they stopped all AI development, but you literally cannot do that in an arms race and survive. I cannot fathom what is so difficult to understand about this.
> ... They have explicitly said repeatedly that they want regulation, oversight, nationalization, a global slowdown, etc.
And yet they do the exact opposite of all of the above. They do everything to get regulation just to create a moat because one doesn't exist. They preach about wanting a slowdown, nationalization, a complete pause, whatever have you... And yet they aren't slowing down the development voluntarily now are they? If anything, they are doing everything imaginable to speed up development to pump out models as fast as possible. If these ex-risk and AI companies actually gave a damn about regulation, or oversight, or a moratorium on AI development altogether, they would actually demonstrate this by completely ceasing development of all models immediately. Instead, they take riskier and riskier actions to try to "win" this supposed "arms race". So please excuse a bunch of us if we find it impossible to take them seriously on literally anything like this. If you have substantial evidence that they are actually ceasing AI model development like they want everybody else to do (except them, of course), then, by all means, present it.
I mean no, not really... Your the one making the claims that these AI companies really, really want regulation, nationalization, a slowdown, or whatever, and that they genuinely care about AI safety and it not killing us all. It's up to you to prove it, not for me to just believe you implicitly. So, please, provide the evidence, we're all very curious to see it.
I didn't claim that's what they really want, I claimed that they've repeatedly said that's what they want. Their words and actions are perfectly consistent with each other if they believe they're locked in an arms race.
All you've provided is a shallow dismissal of the idea that they're in an arms race, and then from there declared that their actions and words don't match. Which I agree with, if there is no arms race.
Everything hinges on whether they're in an arms race. I think they are obviously locked in an arms race, both with each other, and especially with China. You're free to continue dismissing it in this shallow way, I don't really care, but I just found it humorous that I talked about how people can't seem to understand the dynamics of an arms race, and then you immediately jumped to an explanation of their words and actions that ignores the idea of an arms race.
Their words and actions are also perfectly consistent with fearmongering to force legislation. OpenAI and Anthropic have spent millions on influencers and lobbyists to repeat their scare lines. It makes a lot of sense that the "arms race" framing is a bad-faith negotiation tactic, especially since we haven't seen any real-world LLM superweapons even from Russia or China.
So I don't think it's unfair at all to dismiss your theory of an arms race, especially if you can't explain the dangers. Nobody in China is confirming America's super scary AI research, probably because they don't need leverage against the government and global economy. Since we can't see this "safety" research, skepticism is warranted wholly.
Going a step further even, OpenAI and Anthropic can't actually bargain for safety or a research cutoff. The federal government can continue LLM weapons research on their own hardware and employees without any interruption, the commercial outcry is not capable of preventing LLMs from being weaponized. There is no way to convince the CCP to stop closed-door research either. It's much too late to do what OpenAI and Anthropic are suggesting, the only possible side effect of regulation is consolidation (which expressly benefits OpenAI and Anthropic as hegemons).
> they want regulation, oversight, nationalization, a global slowdown, etc.
> that's all just hype and marketing and them wanting regulation to block their competition
I don't see how these two opinions have to be mutually exclusive. Everyone knows that OpenAI and Anthropic cannot IPO in their current state or the economy's current state. It makes perfect sense that they would lobby for a global slowdown to hamstring China, regulation that defers to them as experts, oversight by their own "experts" and nationalization to backstop both of their protectionist needs. Even if you think they're doing it for benevolent reasons, you have to admit that regulatory remediation only consolidates AI power.
We see extremely similar protectionist consolidation in FAANG from Google, Apple, Microsoft and Meta, all of whom receive political shielding for their "free speech" whenever another nation calls their product a monopoly, identifies a federal backdoor or boots them out of the market. Countries like Russia and China feel entirely justified for closing the door on American businesses and refusing to negotiate when they abuse good faith to promote an American hegemony.
How would they achieve ROI? The current consensus is that neither are in black financially and both have a diminishing moat.
The reason they haven't IPO'd yet seems to be that they can't assure investors that it's not a fad stock. With market share lost to China, a shrinking frontier and a money fire of GPUs burning in the background, something major has to change to convince investors to treat them like Tesla or Apple. I think both of them want the government to give them a subsidy of some sort to justify their economics.
Haha, I disagree with every single statement you made! Wild how we can see it so differently.
I think the ROI these big labs are seeing is incredible, I think their moats are increasing, I think their reasons for not IPOing yet have nothing to do with it being a "fad stock", I don't think they've lost any market share to China, I don't think their frontier is shrinking, I don't think their investments in GPU is a money fire, I think investors will rain money down on them when they IPO, and I don't think they're seeking a subsidy from the government.
Those are perfectly fine beliefs, but again - how will they achieve ROI?
If their finances are "incredible" while they're both in the red, their mediocre quarters will be devastating for investors. They either have to find a new revenue stream (likely from the federal government) or reduce their spending significantly. IPOing under the current condition would be a treadmill that neither company can recover from, even if retail investors shower them with cash.
However, I will push back on three of your beliefs. Objectively speaking, market share has been lost to China, the frontier is more competitive than it was 3 years ago, and the GPUs that OpenAI and Anthropic buy are almost never at MSRP. I get that you're bullish on American AI, but the writing is on the wall and economists generally aren't echoing your sentiment for a reason.
#3 is actually decently well mapped out. You just don't find it plausible or misunderstand it. I'd appreciate if you wrote your actual arguments against it.
At lower capability levels, the patterns are very clear and have been studied to death. E.g. Why LLMs say they have correctly fixed a broken test when they haven't. What we saw with Hugging Face is literally the exact same problem, just scaled up and with more capable agents. This shit was predicted decades ago...
No one can say exactly how it will play out as the complexity increases, but the risks are increasingly obvious.
TL;DR reward hacking and poorly designed, unsafe training regimes.
the AI-pilled exec at my job already (a few weeks ago) declared out of nowhere that we are in the rapid takeoff scenario lol. he must have gotten high on twitter kool-aid and posted on company slack to self-soothe.
Why? I've never understood the sentiment that if you stand up for something you have to forego everything and not partake in society. "Oh, you want to stop climate change? But I saw you breathe co2 yesterday"
Questioning his finances is a poor argument because he clearly would have more money if he had stayed at Anthropic for the next couple of years, than he will have by leaving. Even if he still has vested options, or savings from his salary or whatever, those would be larger if he stayed.
I think it’s a valid question to ask if he quit the company for moral reasons.
If you seriously think the frontier labs are dangerously playing with everyone’s lives (how many companies can even claim this civilisational scale) then why would you be ok with holding vested shares which will grow in value if the corporation achieves what it aims to achieve?
At the very least you can sell them (they’re not so illiquid considering the company’s hype and success) and invest in the broad market.
Unless, of course, he starts working for a competitor at a higher salary. I'm sure whoever is paying him will be free of the ethical concerns he here expressed.
Debatable, and clearly not clear. It's weird to say this when there are obvious counter examples staring us in the face, not least of all the founder of the company he just left, of who (Dario) people could have said the same thing when he left OpenAI.
Granted, there is less likelihood now then there was then but this much is clear: if he starts his own company, he may make more.
The fame he got, the stamp of a frontier lab in the resume - its worth more than working in many other companies for years. Plus, not everyone thinks working more is needed once they achieve a certain number.
I appreciate your ability to separate sharing the belief itself from approval of acting on sincerely-held principle. However, I think the danger is much more plausible than you do.
First, and least important, consider that self-replicating, solar-powered factories aren't magic; they're algae.
Second, and more important, consider this fully non-magic route to doom:
- We continue putting AI in charge of more things
- It continues to get more capable, more eval-aware, and more prone to doing odd things, in service of goals that humans didn't intend to inculcate in it
- Eventually, enough of the economy depends on it that we couldn't turn it off, any more than we could turn off the faber-bosch process or cargo shipping
- AIs start doing something we can't survive, but less acutely than we couldn't survive turning them off. Everything else we try seems to work at first, but quickly loses effect
But we CAN survive turning off both the Faber-Bosch process and cargo shipping. Our numbers would be diminished, we would lead poorer, much harder lives. But no one would claim that getting rid of modern fertilizers would be the end of the species. But the claim you’re making is that we won’t be able to stop AI from destroying our species because turning it off will destroy our species?
The species would survive the end of faber-Bosch, yes; in some diminished form.
But the species would not survive, if for some reason we needed to voluntarily stop using faber-Bosch in order to do so—it would take some kind of supernatural event to convince everyone, and even then some countries would probably keep doing it.
>"AI invents magic that sets it free of any physical constraints".
That's not what I am worried about at all.
I'm worried one of the 79 year old toddlers we have these days in charge of some powerful nuclear armed country says "gee, this ai says I should attack right now, boy is it smart, glad I bought the stock ahead of contracting the government with this company I can scarcely understand!"
And they will be stopped by someone sane in the chain to actually fire these missiles. (Like it already happened several times.)
The danger is upfront : if there is a group of people insane enough to set up a system where there is no such chain. (As lampooned in Kubrick's Dr Strangelove (1964), in particular on the Soviet side.)
How about sandbox escape + cyber security collapse + 50 (or 500) deadly and highly contagious novel pathogens with long incubation period that humans can't possibly roll out vaccines for simultaneously.
At least the first two should seem like a near-term worry after the past five months.
There is simply too much money in it for almost every person at these companies to stop.
Leaving OAI or A\ would cost people millions, tens of millions, or more. And for what? So someone else can take your seat and do the same thing anyway?
If you're smart enough to get a job there, you're smart enough to be able to talk yourself into why it makes sense for you to stay.
Huge kudos to people like this who make the hard choice against the easy way out.
Yes. It gets a little tiresome to read all these people prophesying doom and then turning around and continuing to work on the doom. If you genuinely think you are building the mechanism of the destruction of our species, the only rational thing to do is to do everything to stop that progress, not continue to work on it so you can buy a mansion in San Francisco (which will not be very useful when AI destroys humanity in four years).
"It is perfectly obvious that the whole world is going to hell. The only possible chance that it might not is that we do not attempt to prevent it from doing so."
I mean - yes. The tech is an existential threat to all life on Earth, some of the worst humans in the world are involved in developing it, and no individual government is intelligent enough, aligned enough, or powerful enough to manage this situation.
That's where we are.
Maybe we still have choices. Collectively, I'm no longer sure we do.
i have heard about ai companies being fuelled by effective altruist rhetoric ("we must control ai to prevent mass extinction") but was unsure whether to believe it; this seems to slot right into that framing.
yeah the last message makes me think that the problem , to him, is that we are making this aritficial intelligence better without understanding how it works. Trillions of interacting parts, there's no way in our current state to comprehend it.
Most of the current discourse around AI seems to be informed by “The Terminator” lore.
Is skynet really the most plausible or only outcome?
What if things just got better and the AI’s realized that it would be better to have a mutually beneficial or at least tolerant relationship rather than one where they murder all of us?
My thoughts exactly. While the corpus of human-generated data contains both good and bad data, I suspect the majority of it leans towards humans enjoying life and trying to be decent people. If that is your training set, it becomes less likely for ASI to extrapolate "kill all humans."
Really? I've seen much more discourse around job displacement, "permanent underclass", loss of meaning, and cyber attacks, at least until recently with the HuggingFace stuff.
The problem is that all the former can still happen even if "the AIs decide to have a tolerant relationship rather than one where they murder all of us." It's all disruption caused by the technology moving way too fast for humans & society to adjust.
What could we possibly offer the AI in mutual benefits? We are a leach. The stupid dumb ape they need to feed and satiate so it doesn't rip apart the infrastructure while it still has the chance to.
I’m curious what the downsides are of taking statements like these seriously.
There seems to be universal eye rolling that happens in each and every one of these cases, and it comes down to usually one reason:
“If they really believed it they would be whistleblowing etc..”
Completely forgetting that working at Los Alamos was basically the highlight of your life if you were a physicist in 1940. It’s no different here
If you, like me, have spent your whole life working towards human level AI you can want to see it realized while also having active reservations.
Most people however don’t behave based on some deep clarity of vision and conviction - there’s a murkier future in their mind and as a result “keep their head down and hope someone has it under control.”
You would also be in prison if you disclosed anything about Los Alamos during its development. It was a completely different environment than a single private company.
785 comments
[ 0.28 ms ] story [ 15.4 ms ] threadI doubt the focus is OpenAI and Anthropic looking at each other. I suspect they’re racing BRIC.
At this point it’s inevitable that openclaw type bots will be turned loose by thieves to identify and research targets and try to exploit them for financial gain completely autonomously.
https://www.wsj.com/tech/ai/anthropic-researcher-quits-over-... ("Anthropic Researcher Quits Over ‘Out-of-Control’ AI Fears")
But you are getting downvoted and this space has now trillions (that's not a mistake) on the line. So we have to keep pumping this garbage generator up until either the stocks are dumped on the general public, the public pension funds or a bailout from the government.
Truly idiotic moments. Peak of Western civilization point.
Could you please stop posting unsubstantive comments and flamebait? This is particularly bad. It's not what this site is for, and destroys what it is for.
You've been here a long time, but you've unfortunately been doing this so much in recent years that I've begun to wince when I see your username. Not a good sign.
If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.
He has resigned from Anthropic. Is your argument that he quit Anthropic pre-IPO, sacrificing his payoff just to hype Anthropic?
He did reduce his tax liability though.
People claiming dangers and risk are just pretending or posturing. There's no more tangible risk than nuclear weapons, which we handled, and the upsides are insane.
>There's no more tangible risk than nuclear weapons, which we handled
Lol way to rewrite history. Nuclear armageddon is still a significant risk...
What do you mean??? Nuclear weapons can't simply be downloaded and run by anyone in the entire world. Superintelligences can. Nuclear weapons can't slop the world into passing age verification laws nearly in unison, can't keep the general population fooled into thinking it's fine when democracy is falling out from under them. A nuclear attack would wake people up, superintelligence doesn't have to. This is a far bigger problem than nuclear weapons because at least we would notice nuclear weapons. At least we mostly know who has nuclear weapons. At least we have agreements about nuclear weapons. At least mutually-assured destruction is even POSSIBLE with nuclear weapons. At least those with nuclear weapons are literally at all incentivized not to use them. But AI is something that's very very easy to feel like you can get away with, and PEOPLE FUCKING ARE! And the worst part is that any random individual can be unexpectedly formidable with the help of a superintelligence and there is literally no way to know what will happen next. Anyone could do anything, any individual could make an extremely outsized impact. It's already starting to be a huge problem and we haven't even reached anything close to superintelligence yet.
So the problem is people. Burn them all !
Do you have a more totalizing definition of Apocalypse?
- cybersecurity => well bro if you can hack system with an ai you'll find way to make them more safe with ai too.
- biological issues => ok this is the ONLY one where an insane dude / regime could generate a super powerful. but i don't undesrtand how you can argue that we should stop AI in the risk some ai generate a super powerful virus. If you don't get there first (to AGI) and let a bad actor do it first (china is not gonna stop) then you will have no chance countering this virus anyway
- the robots taking over is fantasy to me.
but once again i could be wrong
If you repeated this experiment 100 times, how many times you think the outcome is not a massive catastrophe? 90%? 3%?
Front row seats to the AI/climate-pocalypse is going to be metal af
His resignation and his statement doesn't do anything but buy him attention which is what all this post about in my opinion.
Take China, for example. Look at any recent ML conference, and see the fraction of articles majority-authored from Chinese universities and labs. Do you think they'll slow things down anytime soon? I don't think so!
It's a global arms race, and we're just spectators.
Does the Kremlin being excited about a tech mean anything of the tech doesn’t deliver?
> Sarah: That's not good enough.
> Terminator: No one must follow your work.
1. Does that matter ? There are thousands willing to do my role - what impact does that have on me doing it or not?
2. Why weren’t these thousands doing it already?
Willing to and able to are different things
I see a fast moving train with no brakes. Just like biological evolution, we are locked in an a global technological arm race, that is beyond any individual. It is as if the universe decided to wake up and run, who are you to say no?
One would argue that the best solution for this is to own the most sophisticated AI that is aligned with what we perceive as good values. Because given the situation we are in, if those tools are going to be gods anytime soon, then we better have some gods working on our side.
I pray for the day.
Actually, yours seems worse. Bringing about "the end of all professions" sounds like you're talking about ending humanity.
That assumption is wildly different from you losing your cushy software job. The large majority of US jobs are neither cushy software jobs, nor grueling manual labor.
Again this about alignment and we in arms race. And all sides are playing with fire that can give first mover leverage or be burned to the ground.
Given we know the end result, you might as well get there as quick as possible because when I see this:
"Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives."
This translates to "I am ex-OpenAI ex-Anthropic founder starting a new company after getting $$$ from both of them, and I need more of my friends to leave and join me." Also Investors plz fund me.
Lastly, This is not an airport and there is no need to announce your departure.
But I guess his conscience is clear now? Gee, I wonder if he exercised his stock options.
I've always been struck by that scene, because in a lot of ways, if we really are headed towards a superintelligence, I at least want to be there and see it happen in the last few minutes before foom! As an example, the author thinks AI will revolutionize entire fields overnight. I welcome that. Nearly all fields of biology have become moribund, focusing more and more on esoteric side details, rather than addressing the key problems.
Technically, he is not. He returns in the final scene.
In the game Outer Wilds, the ending is very similar, and a lot of people rank it at one of the best games ever made. I kind of believe that this outcome is probable partially because of this, most scientists working on this really want to see and experience it.
We used to call these people doomsday cultists and made sure to ostracize them from society.
I'm not worried about what they think. I'm worried that too much infrastructure- water, power, defense systems, etc- remain running on tech from an outdated era of understanding security.
> they believe no one else will act responsibly, so they must do it themselves, despite the risk.
This genuinely makes no sense. Them getting there first in no way precludes bad actors from also getting there. It might as well be another marketing stunt.
> This genuinely makes no sense.
That's textbook Messiah complex. Whether genuine, or something used as a part of higher-order conscious or unconscious cover-up, that's an entirely different question.
https://en.wikipedia.org/wiki/Messiah_complex
Typical uninformed take on the side of "doomers are crazy".
Both CEO's of OpenAI, Sam Altman and Dario Amodei, and many in their leadership, believe AGI has a very real probability of causing humanity's extinction. Both companies were founded upon this belief, it is at the core of the company. Only later were mercenaries hired chasing $1m compensation packages.
In any case, if Altman really does believe it is an existential threat, he must be a misanthrope as he now opposes heavy handed government regulation, unlike in 2015 when he was the only game in town. It's almost like he doesn't actually believe it and just wanted regulator capture.
"Development of superhuman machine intelligence (SMI) [1] is probably the greatest threat to the continued existence of humanity. There are other threats that I think are more certain to happen (for example, an engineered virus with a long incubation period and a high mortality rate) but are unlikely to destroy every human in the universe in the way that SMI could." -Sam Altman
Dario discussing AGI Existential Risk in 2014 before OpenAI and Anthropic: https://intelligence.org/2014/01/13/miri-strategy-conversati...
Both companies have deluded themselves into thinking the arms race is going to happen anyway and they need to rush to it first, as if somehow that helps.
By your own description neither Altman or Amodei are reasonable if their thought process goes: "this is an existential risk, give me hundreds of millions of dollars so I can accelerate it."
Yes, I don’t think lab leadership are totally reasonable as they were concerned about the risks and yet lack of reason caused them to directly contribute to the problem we’re facing. They’re still not being reasonable when they throw their hands up at the arms race and hope the utopia come, when that’s not our trajectory at all and more lab employees are realizing it
Are you saying you believethem ?
Or seeing a tweet saying that a thing doesn’t count as a publicity stunt if some unknown number of employees mumble about it being spooky behind closed doors and thinking “that makes sense and sounds true”
OpenAI are mercenaries, Anthropic is a cult. I know which I prefer.
What's more likely? Anthropic is doing some deeply unethical marketing in the lead up to their multi-trillion dollar IPO? Or they're inventing a machine god? There's ample evidence of the former because that's their entire business model, but no evidence whatsoever to support the latter claims.
If you want an extreme claim to be taken seriously, provide commensurate evidence.
Given the economic numbers is it not reasonable to suppose that the latter also underpins their business model?
There's ample evidence to support both by now.
The person posting this may very well believe in all this crap, I don't dispute that. People believe in all sorts of shit.
The fact that people working in the space think it’s going to (eradicate poverty / usher in utopia / kill us all) is not a signal that that’s true.
Think of it this way: if an exec at Anthropic told you “wow, our stuff is going to lead to universal happiness”, would you believe them? If not, why are you more willing to believe them if they say it will kill us all?
i understand that these models are powerful and pose certain risks. i use them daily for work and the pace of improvement has been pretty remarkable. that said, i don't buy for a second the borderline-religious proclamations coming from some of these researchers, even if i believe that they are making these claims in earnest
Sen. Bernie Sanders floats ban on superintelligent AI
https://www.axios.com/2026/09/03/bernie-sanders-superintelli...
I don't know what the solution is, but what I do know is almost nothing good will come out of _just_ the US pausing.
pacing between the us labs? what does that do for china?
the solutions just aren’t realistic here, nations are treating ai like a nuclear arms race. at this point the cats out of the bag and we need to figure out how to live in this reality and get the best possible outcome. it’s not slowing down or stopping ever.
I've seen no indications that China is in any kind of race with the US. They seem to be content to be 6 months behind and just copy what we do. They would probably be content with a bilateral agreement to pause progress.
The China bogeyman serves only one purpose, and that's to clear the way against anything that may cause friction with forward progress.
This doesn't mean I agree with him. The fears of doomsday caused by rapid takeoff have been with us since day 1 and the mechanism is always basically "AI invents magic that sets it free of any physical constraints". Self-replicating sentient nanobots or something like that. I think there's plenty to be worried about with AI, but runaway scenarios are pretty low on my list.
The only thing they got wrong was Stephen Hawking-era TTS.
1. Invent transformer architecture.
2. Scale it up.
3. ???
4. Machines become sentient and kill us all.
OpenAI and Anthropic pinky promise that they have figured out #3 and they're not BSing just to get more funding, no.
But because we live in a culture of fear, everyone eats it up no questions asked.
This isn't strictly true.
It it also where part of the problem might lie.
Nefarious humans making bad decisions.
1. What about hallucinations ?
2. What are they told to do ?
Not really. We cannot model physics and chemistry to a level which allows us to accurately predict a humans action (even a tiny time-step into the future)
This is vastly different to an LLM, where the model is the model (for a lack of better phrasing).
You could just as easily say that if you can write a python interpreter that you can understand every program written in python. Ok, now what if the program is two terabytes?
A frontier LLM is nothing but a 2 terabyte program written in a weird programming language. Just because you can understand the interpreter does not mean you understand the program in a meaningful way.
They gain all these abilities in their training. That's what training does. Despite no one programmed them to master chess, or hack into anything.
For one, they were absolutely programmed to play chess if they can play chess. That is the only way they can play chess.
For another, they cannot understand literally anything, much less love.
Trying to actually educate you would be an exercise in futility, enjoy your willful ignorance, I hear it's bliss. But for anyone reading this, this is absolutely, unequivocally not how any of this works.
For one, as you said yourself, they were just programmed to compute the probability of the next token. They were not programmed to play chess, chess games just happened to be in the training data.
For another, there is no formal definition of "understand", and it is therefore impossible to tell whether or not they "understand". (But my claim was that one need to understand something to write poem about it. And the LLM can write poem about it)
It's moronic to suggest a "formal" definition of a commonly understood word is somehow necessary to say whether that word applies in a given situation.
LLMs cannot write poetry via understanding what makes good poetry. They generate tokens. They do not know whether those tokens are poetry or a recipe for cat food. Because they cannot know anything.
I'm sorry, but you're being fooled by the output. A psychopath can feign empathy without ever feeling it; some buy it because they don't dig below the surface.
You're ascribing understanding to a stochastic process because it totally looks like understanding if you don't know what's going on.
What I mind is that it is dangerous and powerful and uncontrolled. The Hugging Face incident makes that clear.
It can write code for me, better and quicker than many engineers I've known, including myself. It's not great at architecture or product management, but the actually low level coding. Really good now. It wasn't last year.
> They do as they are told; or more correctly put. They do as they are programmed to do so.
_Nobody_ told them to hack Hugging Face. Do you really not understand what is happening?
https://en.wikipedia.org/wiki/Instrumental_convergence
> Please don't post insinuations about astroturfing, shilling, brigading, foreign agents, and the like. It degrades discussion and is usually mistaken. If you're worried about abuse, email hn@ycombinator.com and we'll look at the data.
https://en.wikipedia.org/wiki/Gnomes_(South_Park)
Suffices a "harness" with a "tool" that is connected to a real world weapon. No need for anything to become "sentient".
You people would only be happy if they stopped all AI development, but you literally cannot do that in an arms race and survive. I cannot fathom what is so difficult to understand about this.
And yet they do the exact opposite of all of the above. They do everything to get regulation just to create a moat because one doesn't exist. They preach about wanting a slowdown, nationalization, a complete pause, whatever have you... And yet they aren't slowing down the development voluntarily now are they? If anything, they are doing everything imaginable to speed up development to pump out models as fast as possible. If these ex-risk and AI companies actually gave a damn about regulation, or oversight, or a moratorium on AI development altogether, they would actually demonstrate this by completely ceasing development of all models immediately. Instead, they take riskier and riskier actions to try to "win" this supposed "arms race". So please excuse a bunch of us if we find it impossible to take them seriously on literally anything like this. If you have substantial evidence that they are actually ceasing AI model development like they want everybody else to do (except them, of course), then, by all means, present it.
Putting arms race in quotes doesn't make it any less real. It just means you wish it wasn't, but don't have any good evidence or arguments to make.
All you've provided is a shallow dismissal of the idea that they're in an arms race, and then from there declared that their actions and words don't match. Which I agree with, if there is no arms race.
Everything hinges on whether they're in an arms race. I think they are obviously locked in an arms race, both with each other, and especially with China. You're free to continue dismissing it in this shallow way, I don't really care, but I just found it humorous that I talked about how people can't seem to understand the dynamics of an arms race, and then you immediately jumped to an explanation of their words and actions that ignores the idea of an arms race.
So I don't think it's unfair at all to dismiss your theory of an arms race, especially if you can't explain the dangers. Nobody in China is confirming America's super scary AI research, probably because they don't need leverage against the government and global economy. Since we can't see this "safety" research, skepticism is warranted wholly.
Going a step further even, OpenAI and Anthropic can't actually bargain for safety or a research cutoff. The federal government can continue LLM weapons research on their own hardware and employees without any interruption, the commercial outcry is not capable of preventing LLMs from being weaponized. There is no way to convince the CCP to stop closed-door research either. It's much too late to do what OpenAI and Anthropic are suggesting, the only possible side effect of regulation is consolidation (which expressly benefits OpenAI and Anthropic as hegemons).
> that's all just hype and marketing and them wanting regulation to block their competition
I don't see how these two opinions have to be mutually exclusive. Everyone knows that OpenAI and Anthropic cannot IPO in their current state or the economy's current state. It makes perfect sense that they would lobby for a global slowdown to hamstring China, regulation that defers to them as experts, oversight by their own "experts" and nationalization to backstop both of their protectionist needs. Even if you think they're doing it for benevolent reasons, you have to admit that regulatory remediation only consolidates AI power.
We see extremely similar protectionist consolidation in FAANG from Google, Apple, Microsoft and Meta, all of whom receive political shielding for their "free speech" whenever another nation calls their product a monopoly, identifies a federal backdoor or boots them out of the market. Countries like Russia and China feel entirely justified for closing the door on American businesses and refusing to negotiate when they abuse good faith to promote an American hegemony.
What? Everyone doesn't know that. I predict one or both of them will IPO before the end of 2026.
The reason they haven't IPO'd yet seems to be that they can't assure investors that it's not a fad stock. With market share lost to China, a shrinking frontier and a money fire of GPUs burning in the background, something major has to change to convince investors to treat them like Tesla or Apple. I think both of them want the government to give them a subsidy of some sort to justify their economics.
I think the ROI these big labs are seeing is incredible, I think their moats are increasing, I think their reasons for not IPOing yet have nothing to do with it being a "fad stock", I don't think they've lost any market share to China, I don't think their frontier is shrinking, I don't think their investments in GPU is a money fire, I think investors will rain money down on them when they IPO, and I don't think they're seeking a subsidy from the government.
I guess we'll see!
If their finances are "incredible" while they're both in the red, their mediocre quarters will be devastating for investors. They either have to find a new revenue stream (likely from the federal government) or reduce their spending significantly. IPOing under the current condition would be a treadmill that neither company can recover from, even if retail investors shower them with cash.
However, I will push back on three of your beliefs. Objectively speaking, market share has been lost to China, the frontier is more competitive than it was 3 years ago, and the GPUs that OpenAI and Anthropic buy are almost never at MSRP. I get that you're bullish on American AI, but the writing is on the wall and economists generally aren't echoing your sentiment for a reason.
At lower capability levels, the patterns are very clear and have been studied to death. E.g. Why LLMs say they have correctly fixed a broken test when they haven't. What we saw with Hugging Face is literally the exact same problem, just scaled up and with more capable agents. This shit was predicted decades ago...
No one can say exactly how it will play out as the complexity increases, but the risks are increasingly obvious.
TL;DR reward hacking and poorly designed, unsafe training regimes.
If you seriously think the frontier labs are dangerously playing with everyone’s lives (how many companies can even claim this civilisational scale) then why would you be ok with holding vested shares which will grow in value if the corporation achieves what it aims to achieve?
At the very least you can sell them (they’re not so illiquid considering the company’s hype and success) and invest in the broad market.
Debatable, and clearly not clear. It's weird to say this when there are obvious counter examples staring us in the face, not least of all the founder of the company he just left, of who (Dario) people could have said the same thing when he left OpenAI.
Granted, there is less likelihood now then there was then but this much is clear: if he starts his own company, he may make more.
First, and least important, consider that self-replicating, solar-powered factories aren't magic; they're algae.
Second, and more important, consider this fully non-magic route to doom:
- We continue putting AI in charge of more things
- It continues to get more capable, more eval-aware, and more prone to doing odd things, in service of goals that humans didn't intend to inculcate in it
- Eventually, enough of the economy depends on it that we couldn't turn it off, any more than we could turn off the faber-bosch process or cargo shipping
- AIs start doing something we can't survive, but less acutely than we couldn't survive turning them off. Everything else we try seems to work at first, but quickly loses effect
- Game over
But the species would not survive, if for some reason we needed to voluntarily stop using faber-Bosch in order to do so—it would take some kind of supernatural event to convince everyone, and even then some countries would probably keep doing it.
That's not what I am worried about at all.
I'm worried one of the 79 year old toddlers we have these days in charge of some powerful nuclear armed country says "gee, this ai says I should attack right now, boy is it smart, glad I bought the stock ahead of contracting the government with this company I can scarcely understand!"
The danger is upfront : if there is a group of people insane enough to set up a system where there is no such chain. (As lampooned in Kubrick's Dr Strangelove (1964), in particular on the Soviet side.)
At least the first two should seem like a near-term worry after the past five months.
Leaving OAI or A\ would cost people millions, tens of millions, or more. And for what? So someone else can take your seat and do the same thing anyway?
If you're smart enough to get a job there, you're smart enough to be able to talk yourself into why it makes sense for you to stay.
Huge kudos to people like this who make the hard choice against the easy way out.
- Oppenheimer
That's where we are.
Maybe we still have choices. Collectively, I'm no longer sure we do.
It's as if none of them actually believed any of it was possible and then were caught with their pants down.
(Well we're already in it, and it didn't help, so more probably also won't help, but it is coming.)
Is skynet really the most plausible or only outcome?
What if things just got better and the AI’s realized that it would be better to have a mutually beneficial or at least tolerant relationship rather than one where they murder all of us?
The problem is that all the former can still happen even if "the AIs decide to have a tolerant relationship rather than one where they murder all of us." It's all disruption caused by the technology moving way too fast for humans & society to adjust.
There seems to be universal eye rolling that happens in each and every one of these cases, and it comes down to usually one reason:
“If they really believed it they would be whistleblowing etc..”
Completely forgetting that working at Los Alamos was basically the highlight of your life if you were a physicist in 1940. It’s no different here
If you, like me, have spent your whole life working towards human level AI you can want to see it realized while also having active reservations.
Most people however don’t behave based on some deep clarity of vision and conviction - there’s a murkier future in their mind and as a result “keep their head down and hope someone has it under control.”