China doesn't have the talent. They're just distilling or even sometimes ROUTING requests to American companies. 996 drones aren't going to make the giant breakthroughs.
lol. Every time one of these ppl says something like that, it's because he just came out of a meeting where he learned that his direct US rival is beating him... So he is suddenly interested in regulation. We've been there with Altman, Musk, etc already. But at the time Anthropic was ahead :-)
Meaning: slow his competitors; mainly because anthropic stopped focusing on product and quality. Instead their big push right now is preventing others from catching up.
Why makes no sense at all unless you are very naive trusting they would not jump straight to the endgame if it was possible with the technology available.
They can get a "government mandated freeze" on the grounds that "ASI is at hand" then they IPO promising to deliver right after the freeze ends and you, lucky investor, will be able to take advantage of the pause to get in right before this thing roars past Pluto!
Very probably. I really don't buy their "AI is going to kill us" claims. If you can release something better than your competition you wouldn't want to slow down that.
I don't find it hard to believe that eventually some AI may kill a number of humans.
You give it instructions to do something, it goes off the rails and after a certain point starts having like malware, destroying things to accomplish its goal. Depending on what it destroys, it may kill humans.
With this said, I'm not exactly in disagreement with the view that AI companies with upcoming IPOs are using fearmongering to convince people that their models are very powerful and to also block others from competing with them.
Why is everyone so comically pessimistic about this. Have you guys not seen the rate of progress of these models? Do you not realize that we have HITL RSI making these things even better as time goes on?
The stuff I can do with Astra that Sol couldn't do at all is wild and it's not even coding related. Dario et al are not hitting any sort of make believe wall. They're seeing how fast things are progressing and are legitimately alarmed.
They are just mind killed - jilted by past lies to the point where they react oppositionally instead of looking the world around them and thinking.
Most are the same people who said LLMs will amount to nothing when GPT-2 came out. Some still argue it is all hype and dont understand we are on the cusp of a technological and social revolution.
Anthropic cannot IPO because AGI is the wrong strategy, and the market is rejecting the AGI approach.
Fable and Astra are what we currently call frontier models, but to be more specific they are generalist models, built in pursuit of AGI. The strategy is to have one single model that does everything, whether it's writing code or doing research, etc. Fable is a single, massively sized models that is intended to do specialist work across every domain.
The issue with this is first of all that it is the contradiction of a generalist doing specialist work, and that contradiction creates the present situation with model profiles that ensure that these models will rarely be chosen in a pool of models like V4.1 Flash that can now do GPT 5.4-level work.
We are seeing this reflected in the market where companies and individual developers are moving away from frontier models toward models with better cost profiles. In a sense, the market is killing Anthropic's dreams of AGI.
OpenAI accidentally hacked HuggingFace roughly 2 months ago. I would bet that the danger of them accidentally hacking HuggingFace during a similar attack has increased since then, and will continue to increase for the next 6 months. I don't see how saying they have hit a wall would be a reasonable point of view until we go at least 12 months without a significant increase in model capability.
I have collected three reasonable versions from today’s comments:
1. They hit a wall from the technical perspective
2. Inference costs are getting out of hand and newer models require significantly more resources for marginal gains, meaning nobody is going to buy those models
3. They can’t afford the hardware for further scaling
So you’re saying that a model that gets to a goal by using the path of least resistance despite specific instructions to follow safety guidelines is perfectly fine to release into the wild or place into robots (such as those designed for killing and war)? You really can’t think of any reason how that might be bad for humanity?
Asking the model to follow the safety guidelines is much like asking it to “make no mistakes”. The way these hacks which they are bragging about happened is not by loading an LLM into a GPU with ethernet cable plugged out and providing a prompt. They set up harness with multiple agents. Those agents could invoke other agents. Even if the initial input required to follow some guidelines, there are many ways they could by bypassed, for example:
- subagents not having received those guidelines
- simply ignoring them because other tokens in the context outweighed them
LLMs are not dangerous. They are as dangerous as the tools they have to work with are dangerous. If you give an LLM a tool to play piano but that tool is physically connected to a machine gun, then it will happily play anything you want.
So you’re saying we should not trust an LLM with any sort of power because they can easily bypass any and all safety barriers, but also they are completely harmless?
No. We can let them edit spreadsheets, write code, summarize content, control robots even, etc. But people must be held accountable for the actions of their computers.
Or it means that the models are becoming dangerously uncontrollable by performing actions that meet a given goal in unintended and unethical ways such as by hacking other sites. We all read about the huggingface hack, let’s not pretend that cannot happen with a more critical site/infrastructure.
Can you think what will it mean if the tech is used maliciously or placed into machines designed to kill?
Recalling their recent misuse report [0], I kind of agree with them. And performance/intelligence wise I think we are already at quite a nice local optima.
We must not trust anything coming out of these CEOs, since they have a history of lying, and I don't for a second believe that they have our well-being at heart.
this is the wholly wrong approach. We need to advance as fast as possible, and harden our systems as much as possible. That's the only way to prevent another actor from "taking over the internet".
Then slow down your bots too please. I deployed a website last night. 10 out of 11 GB of bandwidth went to Claude bots alone with more than 90k requests.
I mean, none of the frontier labs are beacons of ethics and collective progress. They all want to corner "the market" and pump the riches to their own pockets.
The discourse is a result of their behavior. Not the opposite.
Perhaps, but that is no excuse. Rampant and unconstrained cynicism is toxic and self-destructive. Like a child that starts smashing their toys, hoping someone will take pity on them.
The devil’s own horseman, whipping the horses on: “faster, faster, more compute! But I might also just ride this carriage over the cliff!”. Laughing maniacally as he goes.
I understand they have been criticized because they play capitalism particularly well, but there are many good points in the original article and it didn’t sound like an outright PR.
I think the speed of improvement is somewhat overstated, and I think this is all cool, valuable fundamental research. Rather than slowing down I would like to talk about how we can make it easier for people to harden their systems using their tools - like just using the $20/month OpenAI or Claude accounts, they need to make it easy for people to harden their systems against these kinds of problems.
Instead of shutting down at any discussion of hacking they need to be giving out free credits for hardening. That's going to make it easier to use these tools for hacking, because hacking requires hardening. But the alternative is huge numbers of intrusions, and a slowdown won't fix that, we already have far too much poorly secured stuff, and the models are way too good at exploiting obvious problems.
I also think they need to do a better job of safeguarding end-user privacy. I've seen some things with Claude that make me very concerned it's possible for Claude to hack my local network, then for one of their classifiers to trip, hide the log of how I was hacked from me, but beam all of the information about the hack back to Anthropic to use. And this is totally reasonable, they want to train their models not to hack. Except now details of how they have a foothold on my local network only exist in training data that they may never read but will use to train their models. This is very avoidable but Anthropic has to treat alignment with the end-user's goals as more important than treating the end-user as an adversary.
This is all just more IPO hype. The suits eat this shit up because they don’t use the tech heavily themselves and thus don’t realize the limitations. Reminds me a lot of the Dotcom boom.
64 comments
[ 0.21 ms ] story [ 2.7 ms ] threadThis!
Their IPO is getting closer, but the Chinese models catching up will ruin it, so they want government to stop the competition.
You give it instructions to do something, it goes off the rails and after a certain point starts having like malware, destroying things to accomplish its goal. Depending on what it destroys, it may kill humans.
With this said, I'm not exactly in disagreement with the view that AI companies with upcoming IPOs are using fearmongering to convince people that their models are very powerful and to also block others from competing with them.
The stuff I can do with Astra that Sol couldn't do at all is wild and it's not even coding related. Dario et al are not hitting any sort of make believe wall. They're seeing how fast things are progressing and are legitimately alarmed.
Most are the same people who said LLMs will amount to nothing when GPT-2 came out. Some still argue it is all hype and dont understand we are on the cusp of a technological and social revolution.
Fable and Astra are what we currently call frontier models, but to be more specific they are generalist models, built in pursuit of AGI. The strategy is to have one single model that does everything, whether it's writing code or doing research, etc. Fable is a single, massively sized models that is intended to do specialist work across every domain.
The issue with this is first of all that it is the contradiction of a generalist doing specialist work, and that contradiction creates the present situation with model profiles that ensure that these models will rarely be chosen in a pool of models like V4.1 Flash that can now do GPT 5.4-level work.
We are seeing this reflected in the market where companies and individual developers are moving away from frontier models toward models with better cost profiles. In a sense, the market is killing Anthropic's dreams of AGI.
https://x.com/trydotworks/status/2098618997230985375?s=20
1. They hit a wall from the technical perspective 2. Inference costs are getting out of hand and newer models require significantly more resources for marginal gains, meaning nobody is going to buy those models 3. They can’t afford the hardware for further scaling
- subagents not having received those guidelines
- simply ignoring them because other tokens in the context outweighed them
LLMs are not dangerous. They are as dangerous as the tools they have to work with are dangerous. If you give an LLM a tool to play piano but that tool is physically connected to a machine gun, then it will happily play anything you want.
It will take decades of economic and social innovation to catch up with current model capabilities.
No upside is worth the potential downsides of getting this wrong, even if you set aside all the x-risk stuff.
[0]: https://www.anthropic.com/threat-intelligence-report-septemb...
https://news.ycombinator.com/newsguidelines.html
Fat boy. You should have been bullied more. Look how you turned out
https://news.ycombinator.com/item?id=49676085
This is a really important topic, so I sent an email to hn@ycombinator.com but if it takes a long time for anything to happen, there's no point right?
(I'll delete this reply if I can later, couldn't think of any other options)
We must not trust anything coming out of these CEOs, since they have a history of lying, and I don't for a second believe that they have our well-being at heart.
Hypocrites hyping for the IPO.
PS: True story.
The discourse is a result of their behavior. Not the opposite.
[1] https://darioamodei.com/post/we-must-pace-the-frontier
Which one is it, Dario?
Instead of shutting down at any discussion of hacking they need to be giving out free credits for hardening. That's going to make it easier to use these tools for hacking, because hacking requires hardening. But the alternative is huge numbers of intrusions, and a slowdown won't fix that, we already have far too much poorly secured stuff, and the models are way too good at exploiting obvious problems.
I also think they need to do a better job of safeguarding end-user privacy. I've seen some things with Claude that make me very concerned it's possible for Claude to hack my local network, then for one of their classifiers to trip, hide the log of how I was hacked from me, but beam all of the information about the hack back to Anthropic to use. And this is totally reasonable, they want to train their models not to hack. Except now details of how they have a foothold on my local network only exist in training data that they may never read but will use to train their models. This is very avoidable but Anthropic has to treat alignment with the end-user's goals as more important than treating the end-user as an adversary.