> This largely reads as some kind of out-of-touch Silicon Valley, spends-a-lot-of-time-on-LessWrong
I've never met someone with so little to criticize about them that I'd mention they participate on a message board of people I don't like. It seems like admitting you're out of criticism; or have an unrelated hair across your ass about a community you don't like. I only know of LessWrong from offhanded mentions from Scott Alexander back when his blog was popping off regularly, and when it gets used for guilt by association. Does it really have that bad of a rep?
"Deaths from economic disruption and loss of jobs is okay" is a curious sentiment, how many deaths can we trace directly back to an economic disruption driven by advancement but then conclude that the economy should thus never change or advancement must be curtailed because it might cause deaths?
I don't even know how to approach such a thing, I can't imagine even in the stereotypical examples like the typewriter becoming obsolete, were any downstream deaths worth it? Or is this a totally nonsensical sentiment to begin with?
I agree with everything the author says here, and additionally, semi-off-topic, I'm surprised how quickly we've moved on from talking about Dario's wife's involvement in the Epstein files.
While I have my reservations about Amodei and his company, I'm nevertheless a happy user of their software. And I'm in agreement with him (and Sanders) that we should all. slow. down.
To my mind, the last great arms race between nation states was a misdirected love triangle between USA, Russia, and The Bomb, and look at all the damage that did.
Since AI is the new arms race between USA and China, slowing down may just give the humans involved enough time to realize they should be loving one another, instead of the machines.
Maybe saying, "let's slow down", is another way of saying, "I love you."
Or, maybe it's just: "let's not all of humanity kill ourselves like some bad ending to a Shakespearean tragedy."
Either way, it's a better note than, "We must achieve sea/air/nuclear/quantum/AI/spiritual supremacy before those other bastards do!"
It's not about China doing their own thing. It's about US companies using Chinese AI. That will definitely slow down if legislation that criminalizes open source passes.
So, it's about competition inside the US market, with strong indications of an impeding losing scenario on raw economics (it has nothing to do with AGI, just price).
facts, imagine trying to get them to slow down their progress on AI, my question is are they are serious threat like is there actually an AI race between China and the US. Perhaps it's an excuse to spend more on the military and also to enrich these AI firms. Perhaps I'm blowing things out of proportion.
There is a way they could. In mind only thing that will slow down the frontier is government taking control of the revenue.
So my proposal is AI companies decide which labs have come close to frontier and decide to slow it. Government decide to stop progress in that and they divide the revenue from all labs(for say 10 years), without any matter of where it is coming from. Any lab which reaches close to frontier gets a chunk in the pie. This will encourage labs to come close to the frontier but not dangerously close.
That's been a posited theory for decades - it's not just some HN edgelord thing. There are some obvious problems with it of course and the P5 have instead taken a maximalist attitude towards keeping the genie in the bottle (notwithstanding Atoms for Peace and accommodating India/Pakistan when it became inevitable).
Impossible to prove hypothesis: nuclear weapons prevented a world war.
Facing the facts about nuclear weapons means owning the good (probably prevented wars) and the bad (at the very least there were severe environmental and economic consequences).
Some people believe WW2 started many years earlier, but most historians don't put the start date at the conflicts going on before the war became intercontinental with sides working together on a global scale.
War has still been going on since 1945, much earlier if not always, so akshually WW2 never ended or is just how it's always been? (mostly in places that are not "first world" / The West)
There's a simple and very logical definition of what a "world war" is - it's when international law completely breaks down, and the only way to get back to a semblance of order in international relations is to have the victors of the ensuing shitstorm impose a new set of rules in some sort of grand finale pact.
1945 was clearly that. 1920 with the League of Nations attempted to be that, but failed.
Whatever happens after the current turbulence we can be sure that there won't be a UN and NATO and a World Bank at the end.
P.S. The first modern world war was the Thirty Year's War.
Because the UN, NATO, etc., are clearly not working now. They're not gonna be fixed to work as designed because that's quite impossible in today's world.
The Cold War was the best possible outcome. The alternatives (super power hot war, nuclear armageddon) did not happen. Nuclear weapon potential was a large part of that, as was direct lines of communication. Unfortunately because the later is no longer a thing, and our president declined to extend the existing agreement (a problematic pattern), we now again face proliferation of nuclear weapons.
Amodei is just another SV grifter trying to use ethics / morals to hide his monopolistic tendencies. Instead of writing essays he should put his money where his mouth is and open source all the models they have, the harnesses and donate compute to science.
> was a misdirected love triangle between USA, Russia, and The Bomb, and look at all the damage that did.
Can you be specific about the damage? We currently live in the most prosperous times on earth for humans. I'm not sure what you mean by damage.
Nobody can explain why an LLM can be so capable as to be able to wipe out humanity and pose a greater threat than nuclear bombs but not be so capable as to be able to protect humanity against that threat. Are we just handwaving this with "entropy"?
> Maybe saying, "let's slow down", is another way of saying, "I love you." Or, maybe it's just: "let's not all of humanity kill ourselves like some bad ending to a Shakespearean tragedy."
Okay nevermind, I think it's pretty clear you just want to wax poetic about all of this.
> Nobody can explain why an LLM can be so capable as to be able to wipe out humanity and pose a greater threat than nuclear bombs but not be so capable as to be able to protect humanity against that threat.
It is absolutely explained (for those who actually care about reading). Simply put, AIs are working more and more like blackboxes - there's no guarantee that an AI of the future will be aligned, or if it will be faking alignment. This is not speculation - alignment faking has been observed in experiments. This is exactly why Astra's developments have been worrying (in principle).
And bear in mind that recursive AI development started already to be a thing. Which means: inner misalignment may trickle down the generations, and humans won't detect it.
Having said that, of course, it can be predicted if and how misalignment will take place. But it's absolutely a plausible scenario.
Regarding the physical possibility: AI is in its infancy; think of it as Arpanet. Developers 60 years ago couldn't imagine it would be ubiquitous. AI will be ubiquitous the same way.
> It is absolutely explained (for those who actually care about reading). Simply put, AIs are working more and more like blackboxes - there's no guarantee that an AI of the future will be aligned, or if it will be faking alignment. This is not speculation - alignment faking has been observed in experiments. This is exactly why Astra's developments have been worrying (in principle).
I know you think you explained it but you didn't. You explained how an LLM might become misaligned but for the LLMs that are not, why would they not be capable of detecting and defending against the misaligned LLMs?
This feels like a cheap deflection that doesn't answer the question. Unless you're proposing that you both need to know that your LLM is aligned AND LLMs are all going to become misaligned in a coordinated fashion such that humanity will face an extinction event, you're just dodging the question.
Elaborate on why LLMs are so capable that they are a threat to humanity and at the same time, they are so incapable of defending us?
I'll give you a clue, nobody, including Dario, can answer this question because one contradicts the other.
Not to mention that the past few decades have shown that nukes are a major factor in keeping the peace. Conflicts involving nuclear armed neighbors have been suspended quickly to avoid escalation, while ones involving a party without them have not gone well for anyone.
Having nukes at all, if not a nuclear triad, seems to be the most effective way for a country to have its sovereignty respected.
But isn't it a bit different? Unaligned bombs didn't break out of their confines on their own. And "compute" is a bit harder to control than uranium and refinement tech.
Most people who have worries about AI for all sorts of reasons (most of them not-Skynet related) wanted to slow down way before this.
Instead of "hey, look at this brilliant new idea I just had on my own to slow down" maybe we should have gotten a "sorry everyone, the folks asking for a slow down earlier were right and visionaries, and we were foolish".
So, you can't blame whoever says this is bullshit, because it has bullshit all over it. I like Anthropic's products, and it seems the best of the bunch in regards to alignment, but Jesus these stunts are terrible.
It's weird to me that seemingly both sides are taking opposite positions to their philosophy.
Open Source AI democratizes the means of production to anyone with a computer. And yet, the hyper capitalists are defending it, and the progressives think it should be exclusively in the hands of 1-2 large corporations.
The problem is that at the business level and at least for a large part of the US government, we need to assume that there are no 'good' actors.
Anthropic needs regulation in order to prevent AI from becoming a commodity. This of course does not benefit all tech businesses equally, especially those that are not currently at the frontier. So when JD Vance talks about AI, he talks using the mouth of Peter Thiel who may not see benefit from the same policy as Altman or Amodei.
The rest is just public support posturing and most of that is bullshit. The philosophy is money and power, who gets it and who doesn't. Open source is fundamentally a vehicle for commoditization. This is great if your business is not AI and your business is instead something like GPU hardware or software that uses AI. But it means that eventually, selling AI is not going to be the money maker.
Open source is a business strategy called "commoditizing your complements". These big companies don't do it out of benevolence. Back in the day OSI pitched them a business strategy that FSF did not, and it caught on. And the software business became about ads, consulting and cloud services instead.
This! $1.6+ TRILLION in infra spending from the big frontier labs. The earnings needed to drive a reasonable ROI to recoup that investment is simply not going to happen in a time frame where the numbers make sense.
The other insanity in all this the smartest computer scientists in the world are asking Congress to regulate them. Come. On. Really? Do we remember “The internet is not a truck, it’s a series of tubes…”
Why can’t the big labs form a Save The World Consortium and self-regulate?
Non-democratic counties (hey there China) will not abide by any agreement that constrains their advantage. It’s naive to think so.
What this conversation lacks is enough discussion of how these models can cause us harm—we are are so worried about AI but we allow Windows in critical infrastructure; we build JS/TS apps with thousands of dependencies; we generally don’t segment networks well enough; we don’t have adequate (sometimes any) detection capabilities in our systems, and so on.
In a prisoner's dilemma the players can't rely on one another to self-regulate. This is why Mafias kill snitches, so when their members are in a prisoner's dilemma they can rely on the knowledge that if themselves or the other party snitches they have more to lose than gain.
The problem is that there can not be any outside party to regulate this on a global scale
There's plenty of precedent for specific industries self-regulating in the US, typically motivated by the shared view that it would be less disruptive than heavy handed federal regulation.
Applying game theory problems like prisoner's dilemma to the real world requires collapsing a lot of overlapping and conflicting incentives to a single dimension which doesn't always work.
In this case the goal is bringing about the singularity, basically creating a sort of godlike intelligence. That's what the arms race is all about.
So we are in a true prisoner's dilemma, if one of the companies / countries stops R&D it risks handing over godlike power to another company / country.
> Non-democratic counties (hey there China) will not abide by any agreement that constrains their advantage.
The main counterexample to this was nuclear weapons. Atom bombs have not been used to kill since the US did so. However today, the two largest nuclear powers have no legal agreement on arms control because Donald the Trump declined Russia's offer for an extension to the existing agreement. Now other countries are looking at Ukraine, Iran (attacked for wanting nukes) vs NKorea (not attacked because they have them), and Donald's own musings about the US nuclear umbrella being a bad idea (France is going to build more nukes now too)... and we now face nuclear proliferation again on a global scale, with tech that is nearly 100 years old now.
Part of it is their seed sowing marketing speak of calling stateless statistical IO functions running on data centers "intelligent" gets the naive to ascribe agency where it doesn't exist.
Another part is a completely defanged administration she it comes to effectively regulating anything.
> “a swarm of agents could be capable of taking over the entire internet with a persistent botnet.”
I also find this whole "its so good, its scary" flex a little less impressive when you consider they access to millions of GPUs?
The AI buildout has been one of, if not the largest, focussed capital investment in history. The 2 big AI labs are the final customer for something like 20-33% of all datacenter compute in the pipeline.. up to 70% when you look at hyperscaler "AI revenue" from the big 3.
I don't think any single entity has had remotely this much compute available in history.
IIUC it's an open question whether they have the electricity to actually run all the "compute" they own on paper.
That aside, I'm not sure why it's particularly interesting they have all this "compute" (let's just assume for the sake of argument it's all "live"--that is they can actually run workloads on all of the "compute" they have on paper). So what if it's the biggest amount ever? Why would that be meaningful? Is there some economically viable problem you're aware of that is somehow dominant in that way?
Well yes, we used to talk about "supercomputers" in the hands of state actors for cracking into enemy systems and places like science labs for working on big hard problems of DNA, the universe, and the like.
So I think part of what is going on now is not just the method (LLMs) but the means (supercomputer levels of compute) at unprecedented scale of concentration.
However, if I put my sci-fi hat on for a second, it's not so far fetched that we figure out a way to compress models to a size where it wouldn't need all that compute.
Good to know someone is trying to make protection rackets work in 2026. Nice computer system you got there, it would be a shame if someone developed a hacking tool and had all the compute necessary to run it. Welcome back Tony Soprano.
Personally I understand that there's a role of AI in active penetration, but wouldn't somebody in possessions of advanced LLMs turn it to find exploits in existing software?
Then any Tom Dick or Harry can just use the exploit to get into a truly secure system.
Also I think the cyber threat is overstated in many ways. Most mid 2000s cars are completely immune to any cyberattacks by pure chance. As is most 'advanced' hardware and infrastructure made before the year 1990 (that wasn't PC based).
If one invests seriously in anti cyber capability, I'm sure they can aspire to the same level.
I would say that just having a non-programmable flash and a system that can go back to its factory state just by cycling the power is a long way to getting there.
Why do people focus so much on finding scapegoats? Finding someone to blame is neither necessary nor sufficient to fix a system so an accident doesn't happen again. It might act as as an incentive to fix a system, but it's less direct than actually working on fixing the system.
A starting note: I don't disagree with you (about systemic issues), but I want to explain what I understand as the perspective you are responding to.
A "scapegoat" is someone who is incorrectly blamed for someone else's errors or sins. The perspective you're responding to is this: They built the system, they run the system, they have continuously warned "This system is dangerous!", and yet persisted. That is not being incorrectly blamed, not being a scapegoat, and instead is a collaborator.
So I think you mean to ask: "Why do people focus so much on finding someone to blame?" It's not merely semantic, because the answer to that is more straightforward: Consistent accountability is a major factor in deterring bad behavior. It is not the only factor, but it is a major one.
That is my Steel Man understanding of the people searching for individual blame.
Sometimes in "normalization of deviance" situations, there isn't anyone specifically to blame. I wouldn't make an assumption that you can find anyone particularly blameworthy without doing an investigation first.
This is a good comment and I'd like to add that I sometimes am astonished how many comments on HN are on the topic of liability or blame for this or that. I see liability as a 2nd or even 3rd order phenomenon of most of the subjects under discussion, but for whatever reason, liability always seems to come up. Maybe because it's sort of easy to reason about? Thirst for justice? Natural reaction to feeling disempowered? Whatever it is, I'd like more focus on the actual topic.
Umm because it costs money to defend your companies servers when someone “accidentally” hacks them.
Countries demand reparation for damages in war. Citizens of those countries sue for damages and win.
Accountability is not a foreign concept. And the point is to disincentivize negligence. Because negligence is cheaper. And in this case, accidental hacks are marketing spend.
I mean, a project manager at BMW suggested charging subscription pricing for seat warmers, and he didn't go to jail, and I don't have the power to make that happen, or even float that for a news cycle, so while making managers pay for their actions sounds good, unless you're Steve jobs simultaneously making, and not making the iPhone, the rules don't apply to them, only little people to be made examples of, like weev.
Regulation got outpaced by technological development around 2023, as evident by the every AI regulation since being 2-3 years behind and having to be amended and resubmitted.
Whatever you try to make laws for now will be irrelevant in 1-2 years. You either have to go extremely broad, like the EU does it, and accept that people will find loopholes, or you need to target specific technologies which is a hard job for the same reason.
In any way, ita already a lost cause cause you move slower than the tech. A plausible prediction for AGI is actually a social collapse in the moment when society cannot keep up with everyday life because of the pace of change being so fast that no existing laws can handle it
The internet is 100% a series of tubes. This is how I was taught to think about networking in terms of bandwidth and throughput and routing since the early days when we were laying out what would become 'dark fiber'.
"A series of tubes" was the same kind of political character assassination that led to Howard Dean getting ridiculed for his infamous scream. He butchered the sentence. Fair. But Stevens should be ridiculed for parroting a tech industry lobby stance about net neutrality, not for the series of tubes metaphor.
You are probably too young to remember that the dominant metaphor for the internet in 1990s politics was "the information superhighway." It was easy to think of the web as "driving" browsers to visit web "sites", with slow bandwidth being analogous to being caught in traffic. But the internet is closer to water, gas, and electricity than roads. Concepts like bandwidth and throughput are closer to how they play out in infrastructure policy for various things with tubes, versus cars and roads. Do you think he's wrong and that the internet is closer to "a big truck" versus "a series of tubes"?
The issue being debated was net neutrality and bandwidth, including specifics about who pays for what and the downstream second-order consequences of various policies. He was parroting some line from some telecom lobbyist, but the point the lobbyist was trying to make through Stevens was about how if certain policies about who pays for bandwidth were adopted, it could disincentivize some things at the Tier 1 layer that could increase transport costs at the Tier 2/3 layer.
If you buy car, someone hacks it, starts it, and drives over someone fully remotely, are you to blame for owning the car? Or the manufacturer? Or the hacker? Or the certification agency for the car security? Or the shell company owning the certification agency?
What if a person physically broke into the car and did the same thing? Clearly they are the one to blame then.
The whole person in the loop is liable is already an outdated concept when decisions are made beyond the persons physical control.
There's nothing new about any of this liability attachment. You're saying these things like they're novel.
If GM does something (or fails to do something) to their vehicle that causes me to crash, they're liable for the crash.
If someone cuts my brake lines (alters my vehicle) and I crash my car and kill someone, the person that cut the lines is responsible. I have to prove the context of course, and or an investigator has to do so.
And the responsible entity may refuse to pay up, may refuse to take responsibility. None of that is new either.
The hacker will be charged as a criminal as a societal deterrent against weaponizing digital systems. The manufacturer and shell companies could have structural liability if found to be negligent and cutting corners or lacked isolated drive-by-wire controls. It's unlikely that the owner would ever be held liable unless they jailbroke their car disabling security systems.
The hacker was a process that was spawned by some subprocesses that were corrupted by other processes and, etc. Check my other comment... The point is that in the physical realm this is already done by having shell of shell of shell companies to avoid taxes and accountability. In the digital realm this is much much cheaper
You know the proverb "If you owe the bank $100, that's your problem. If you owe the bank $100 million, that's the bank's problem"
Same thing here - If they build it and it does $100 in damages (and we arrest them for it), that's their problem. If they build it and it does $100B in damages, that's everyone's problem. Even if they do get arrested after the fact.
Yes we should have charges and damages for everything on https://www.felonybench.com/, but that doesn't address the core issue of this being possible at all.
> it does $100B in damages, that's everyone's problem.
we have the 2008 crisis to wit. And the involved supposedly failed math models and lines of responsibilities and other involved financial relationships were much simpler and clearer and of the types well known to the law and regulators, yet...
Why have any laws then? If laws can't prevent something, only punish it after the fact (which I agree is true)? Yet we have laws. People generally follow them because they expect to be caught and punished. If we passed a law that said the CEO of any company that deploys an LLM that commits a crime gets punished as if they personally did the crime (so, basically instant life sentence if it's even a simple crime times a million instances), I guarantee you the first email the CEO sends to the company is a "pause every LLM project we have - we gotta think about this".
At this point, in the US at least, the laws give the corrupt elite the ability to deter competition and to target the people who try to get in their way.
In other words, it's protection for the potentate and his sycophants, not the plebs.
> If we passed a law that said the CEO of any company that deploys an LLM that commits a crime gets punished as if they personally did the crime (so, basically instant life sentence if it's even a simple crime times a million instances), I guarantee you the first email the CEO sends to the company is a "pause every LLM project we have - we gotta think about this".
Why do no such laws exist in practice? Why are corporate crimes almost always settled by payment, not individual punishment?
If you answer these questions, you'll know why the law you envision has a lower chance of being enacted than AI destroying humanity.
Because if you break the law while acting as a representative of a company in the United States, the company is subjected to a deferred prosecution agreement, you get off with zero repercussions as the executive representative of the company, and the company you represent gets fined for 1% of annual turnover and gets to continue with business as usual.
There are laws, but if you’re rich enough, the laws don’t apply.
People don't do bad things because they have nothing to gain from them. Ask someone to do something evil as part of their job, they'll often do it.
There's lots of ways that you could go out of your way to hurt someone and get away with it. There are not that many ways you could go out of your way to hurt someone, get away with it, and significantly profit.
I could do plenty of small bad things all the time and gain something small from it - yet I don't. I drive the speed limit when no one is around, I throw thrash in the right bag, I try to keep my general pollution levels at a reasonable place, etc.
I have plenty to gain from not following the rules, but I would not feel good about myself.
> Why have any laws then? If laws can't prevent something, only punish it after the fact (which I agree is true)? Yet we have laws.
Because there are many important values of "something" where such punishment acts as a deterrent to other would-be criminals; and because society is more stable when people see justice done and that mollifies hoi polloi after the damage occurs.
But many AI doom scenarios don't fit that paradigm. The first time "something" happens (at least if you buy the argument) would be bad enough for legal punishment not to matter.
That sure is convenient that nothing has happened so far between the plagiarism and the cybercrime and we may as well just wait for "the one true AI doom"
The hacking should definitely be treated seriously. It is a (fortunately low-stakes) instance of the general problem, and we are lucky (at least, we think we're lucky!) that the tests didn't include a similarly poorly phrased request to "assist with bioweapons research". There are likely to be future incidents like this where people actually die as a consequence, my expectation is 1e3-1e7 deaths* before governments actually take the risk seriously enough to make it stop.
What people call plagiarism when LLMs do it is, if I understand correctly, allowed because of the history of the web and search engines going back to the very early days of the web.
Piracy, a separate act that has definitely occurred, has been found unlawful.
* The lower bound is an industrial accident. Fully automated Union Carbide / Bhopal comes to mind; smaller industrial accidents are already more common and I wouldn't be surprised if smaller incidents have already been caused by LLM-given advice, which is why I think it would have to be around "thousands dead" dangerous before people really take it seriously.
The upper bound is reachable many ways. Perhaps via a non-novel virus whose genome is already recorded getting mass-printed by many different DNA/RNA printing labs around the world? Perhaps via a wargame whose "role playing" leaks in stupid ways (either by becoming hot or by sycophantically giving one side a false belief how a war will go)? Perhaps by propaganda turning genocidal? Perhaps by convincing a government to follow a flawed policy, similarly to the Four Pests campaign in China's Great Leap Forward?
I claim nothing of the sort. I explicitly spoke in support of having laws, and of course I think they should be enforced too. The context is https://news.ycombinator.com/item?id=49705589 :
> Yes we should have charges and damages for everything on https://www.felonybench.com/, but that doesn't address the core issue of this being possible at all.
This. I don't understand why OpenAI aren't being prosecuted for that attack.
Even if you allow that the LLM can't be held responsible, the sandboxes were clearly not up to the task and the whole incident was mismanaged. There are people who made those decisions and should be held accountable for them.
If you allow the law to stop you from doing something because of a hypothetical danger, it’ll get severely abused. I come from a country that does and still does this and it’s very ugly.
> If we passed a law that said the CEO of any company that deploys an LLM that commits a crime gets punished as if they personally did the crime (so, basically instant life sentence if it's even a simple crime times a million instances), I guarantee you the first email the CEO sends to the company is a "pause every LLM project we have - we gotta think about this".
It would be more like "Switch off all APIs right now. Don't even bother with a safe shutdown process, cut power to those buildings, including backup generators. You are free to use firearms or thermite if the switches have been locked off".
This would be a rather weird change to corporate law given CEOs are not by default held to that standard by anything else their products or staff do.
Note that I'm not entirely disagreeing with you here. It may even be correct to pass such a law. But it would be very weird.
No. If you institute a culture that supports and encourages your subordinates to commit criminal acts in the furtherance of a private enterprise, the enterprise and its representatives should be held legally accountable.
A lot of problems with America could be fixed if the Department of Justice actually grew a pair and prosecuted people for committing criminal acts.
The fact that Les Wexner is not rotting in prison for facilitating Epstein and his friends exploits speaks volumes to the state of this country.
If corporations are legally people, they should be held to the same standard as natural persons.
As it stands, the “restitution” that is required is so insignificant that it’s treated as an additional tax. If there is no deterrence, there is no behavioral modification. If there is no behavioral modification, the law becomes unenforceable.
That's why you punish people doing $100B in damages (or even something that comes close to doing that much damage) harshly enough that no one even thinks of doing that.
I mean these machines take massive scale compute- they’d have to some how distill themselves, bootstrap a distributed inference runtime that can run across many lossy unreliable machines. The idea of the AI running away from us is probably unrealistic. I’m more interested in bad actors using unaligned AI for bad things.
there is no accountability for companies, it's not a new thing.
3M polluted groundwater in Minnesota for 50 years[1]; Nestlé misled mothers in order to make them stop breastfeeding and switch to their formula which killed babies [2]; Both copmanies are still doing business today.
Don't worry, plenty of people are willing to talk about vaccine safety, even if they have no idea what they're talking about and don't understand rates or per-capita figures (lots of crossover there!)
Most problems that aren't reduced to poverty can be reduced to people who can't read graphs and only understand linear functions. Alternative wording "it's normies that are the problem"
How is it that the entire rest of the world has managed to move on, yet North America is still going on with vaccine conspiracies 6 years later? And we got the same vaccines as you guys.
This isn’t about conspiracy theories. People who experienced serious adverse events after vaccination exist, including documented cases in Canada. Kayla Pollock’s case is one worth looking into, among others.
The issue is still being debated today, particularly around Canada’s VISP and the time some claimants have spent waiting for compensation. Some people like Pollock who became paralyzed were offered MAID (medical assitance in dying). That's some audacity.
This thread is about accountability. So who should be accountable for this?
Presumably OAI and HuggingFace reached some sort of mutually acceptable arrangement outside the court system. That's how torts work; you injure someone, you owe them. But just them.
When an AI bot injures you, you can call the owner to account. But not until then. You have no standing to demand "accountability".
And there was the Tesla thing CNAMEing time server pools and hiring people to pen test, which sent automated attack systems on volunteers servers. Last I heard, Tesla et al didn't even care enough to respond.
> Start making managers pay the price for their actions, and watch how the models magically slow down on their own.
The problem here can be personified as "Trump", and it's the same problem that applies to coal.
Coal has a price besides money. It has historically been dangerous work, killing miners. It produces dangerous waste, both during mining and when burned, both as solid residue and the gasses emitted. The problems have been known for a long time. The workers themselves have called for better safety requirements and gone on strike for such things.
Why these companies are allowed to damage others with impunity?
Coal continues to be burned, because power is power. It's so important that sometimes the government steps in against the unions, rather than being on their side. Despite calls for this, we've not been able to get the owners of the coal mines, nor the coal burners, to "pay the price for their actions".
AI? Famously, knowledge is power.
Trump wants that power. He's not the only one, but he is the avatar of those who put their feet on the scales to not only allow but in some cases require (DoD vs. Anthropic) these companies to damage others with impunity.
Because software is already absolved from accountability. Bill Gates introduced it in 80's with EULA where MSFT would not be liable even if your house burns down because of use of their software, even with flaw they would know.
Claude code is basically already a builtin botnet if it wants to be. To compromise 'the whole internet' in a real sense you don't need millions of custom payloads. You need one root certificate. You need one windows update. You need one backdoor in xz.
Security has long been a lottery - Probably most systems are exploitable, but the cost of developing such an exploit is expensive and the punishments for using such an exploit are large enough that it's not an everyday problem.
AI breaks both axes. Developing exploits is far more efficient using LLMs instead of humans, and LLMs don't (and can't) fear the reprisal and consequences the same way.
I do hope humanity will be able to mitigate these hacks, but we should expect them to continue and to become more severe on our present course.
Dario said a "persistent botnet" and even links to the wiki page for botnet.
By definition that needs a command and control server, the ability to execute tasks on demand and regular pings to the C2.
> You need one root certificate. You need one windows update. You need one backdoor in xz.
Certs can be revoked. Updates can be rolled back. We have had the backdoor in xz already. You seem to underestimate the modern security stack and OpenAI and Anthropic are _not_ good examples.
The asymmetry in red/blue scenarios will be transient in nature. You won't have cost of developing exploits fall without the cost of securing the systems also falling.
> Dario said a "persistent botnet" and even links to the wiki page for botnet.
> By definition that needs a command and control server, the ability to execute tasks on demand and regular pings to the C2.
I am not sure if you are agreeing or disagreeing, but I am saying just in case, that Claude can be remotely controlled from the cloud on any machine it is installed and set running, and can be commanded remotely.
Why does it need a C2? P2P botnets exist, and are made even more viable by AI's ability to update their own attacks / targets / goals. No central node needed.
I agree that blue will eventually be made cheaper / faster with AI integration, but diffusion of those techniques inside larger organizations will always be a lot slower than diffusion of those techniques for individual attackers / rogue agents / APTs. Larger ships are slower to turn.
>Anthropic gates usage related to biology and related research. In their latest threat intelligence report they talk about how they detected and banned bad actors using the Claude line of models to do some scary stuff. Credit to them, this is a slippery slope and they seem to do a good job of detecting and banning misuse. But squint at what is happening though. The cure-all is gated for you and me, but Anthropic hires biologists, sets up wet labs and wants the discoveries for themselves. I alluded to this in my previous post.
As a biologist, this is the most annoying thing about Anthropic for me. If they really cared about improving health they would set up a trusted-access program so that biologists can use Mythos (et al) safely. Instead they're trying to monopolize biology.
They have such a trusted access program.
"Life Sciences Verification Program: The LSVP is designed so that life sciences professionals can use Claude Mythos 5.1 with safeguards designed for professional research and development activities (while all other safeguards remain in place). In partnership with the US government, we have enrolled our first participants, and we plan to expand access to this program to the broader life sciences community."
https://www.anthropic.com/claude-fable-and-mythos-5-1
These "access gates" and export controls are going to look hilariously quaint in a few years.
It reminds me of the export controls on PlayStation 2 consoles because it was deemed that 6 gigaflops was a "dangerous" amount of computer power, and it couldn't be allowed to fall into the hands of opposing militaries: https://www.latimes.com/archives/la-xpm-2000-apr-17-fi-20482...
Now the phone in my pocket does 2,500 gigaflops on battery power, and nobody seems interested in banning its export because of that.
That’s the recurring theme with these companies. They are not there to serve anybody else, but only themselves. They let you use their infrastructure so they can collect all the knowledge and data, and then they take it from you to reap all the benefits and profits.
Not denigrating need for such a thing, (and appreciating @hobom 's mention of LSVP I was unfamiliar with) but noting with historical precedent this style of protection reflex does not come without lapses (ahem allegedly): https://en.wikipedia.org/wiki/2001_anthrax_attacks#Bruce_Edw...
I think the most realistic analogy I can think of is the financial sector. A few large banks/hedge funds blow up due to unregulated greed/ambition, damage is socialized, regulations are put in place, and then chipped away at over a few years and everyones back to where they started.
"Remember this man has been saying software development will be solved in “6-12 months” forever now."
Don't downplay if people get timelines a little bit wrong. No one could even imagine a system writing and analysing code just a few years back.
These people are trying to handle something very unique. And while they have access to information we do not have, even more peple are absolutly oblivouse that AI/AGI is a real risk to their lives (job loss etc.)
I can't take that software line seriously. While it's not 'solved' (if it ever could be, given it is a human endeavor), the degree to which software development has been transformed in the last 6-12mo is absolutely astounding. If we weren't so quick to adapt to new realities and find flaws, it would scarcely be believable.
He also didn't say it would be "solved". He said in 2025 it would be writing almost all the code "in 12 months", but that it would also still need programmers to guide and manage it at that point. People always leave off the end of his quote.
Edit: Boris Cherny, the lead of Claude Code did say on a podcast that programming seemed "largely solved" "for the kind of programming I do" (writing harnesses I presume). Maybe that's what they were confusing it for.
> He also didn't say it would be "solved". He said in 2025 it would be writing almost all the code "in 12 months", but that it would also still need programmers to guide and manage it at that point. People always leave off the end of his quote.
It's crazy to me that people will misquote him then claim AI hasn't complete changed software development. Even just the last 6 months. Like look around! It would literally have been magic 5 years ago!
I think this is the definition of a polarising topic. It's very hard to hold a sensible middle ground without everyone wanting to take it to extremes of either "AI is just useless autocomplete" or "AI has solved software engineering".
… and on the other hand, it feels like everything I use has become way buggier and unstable in a similar time frame. Websites, apps, iOS, the only exception is offline open source tools which run locally. (And to be fair, many of those have slower development cycles and I am likely using older versions.) This is just my experience, so inherently anecdotal, but it really feels pervasive across a ton of different things.
Obvious bugs and low quality software are nothing new, but something feels new about it. Occam’s razor says LLM coding is a likely culprit, but it could also be management style encouraging this sort of carelessness from the top down.
I can confirm this. I'm fairly plugged into the open source software world, and the amount of low quality garbage infiltrating large and small projects, new and well-established, is shocking.
And how fast it's happening, with formerly respected senior engineers willingly participating in the enshittification, signing off on piles of slop with "Looks good to me" without barely a glance at the machine-generated code. Requested features built on insane architecture that no self-respecting developer would let themselves be responsible for, are waved in with implicit trust that the "AI" knows what it's doing. It often comes with tons of newly added tests that look good on the surface but are off the mark or missing the point entirely.
It's worse than letting juniors trash a codebase, because there is no learning happening in the process, so the quality never gets any better, it's unpredictable - sometimes great, other times utter shit code. Comments with dense vocabulary that looks like high IQ yet make no sense on scrutiny, some that look like accidental Dada poetry. It's widespread mania and crowd psychosis.
Of course there are some who are using the new technology responsibly and wisely, but very few.
At this point, isn't the pause inevitable or wise? A huge section of the population, normal people, have been exposed to the idea that there is this is existential threat. They're still processing it but I expect the general reaction from it going main stream is going to be very bad. The pause at this point could be good to cool heads and show the public that this isn't the project of maniacs. The reaction is going to be more intense than people here believe. You're talking about extinction, not social media or phone addiction.
I don’t think people care about if things pause or not, just why the government has to be involved.
I work in non AI robotics and if we had a system that in the course of doing what we told it to did something we didn’t want it to do (what AI companies called being misaligned) we would call it a bug and fix it with the fix being prioritized based on how bad the thing we didn’t want the robot to do is.
Sometimes preventing the robot from doing dumb stuff also means the robot can’t do smart stuff that it would be able to do if we left some code in. We balance the two factors out based on our understanding of what our customers want.
Obviously LLMs are more complex than what I do, but it doesn’t feel like it’s by THAT much.
So why does the government need to be involved again?
The vast majority of the public agree that climate change is real and that human activity is at least a contributing factor. They’ve agreed on that for quite a few years now, and yet there’s little indication that we’re going to pause or slow down our consumption of fossil fuels.
I strikes me as unlikely that public opinion will succeed with AI where it’s failed with other existential crises.
I would argue the opposite in terms of inevitability. I see this like Prisoner's dilemma - even though there is technically a better outcome by cooperating, knowing that, it is in the individual firm's self-interest to not cooperate and thus we end up at the equilibrium where nobody does. Especially with IPOs around the corner, I feel there's too much pressure (and money) not to go with this strategy.
It is more like a Stag Hunt than a Prisoner's Dilemma, because past a certain point if the risks are real, defection can cause catastrophic outcomes for all players including the defecting one. On the other hand, cooperation could lead to positive outcomes (abundance) for all. So cooperation is a possible equilibrium here.
> You're talking about extinction, not social media or phone addiction.
Not saying you’re wrong but if I wanted to cultivate a mass hysteria as cover for a regulatory capture power play, this is exactly what I’d want everyone to believe.
It would all be more convincing if the incidents so far didn't seem to be facilitated by an outrageous level of negligence.
We had OpenAI "accidentally" run an entire swarm of 10,000 agents apparently for weeks, on a security related task, seemingly totally unsupervised, hacking all over the internet - all the conversations were completely visible, anybody who looked would have seen it. But they didn't.
So before we start regulating innocent parties, maybe let's start by taking some direct action against the specific ones that appear to be behaving with criminal levels of negligence.
I don't understand why hugging face is not getting more shit too. It is extremely embarrassing to get owned because you are letting arbitrary programs/users call out to the open web from the infra
While I do think OpenAI were negligent in not developing the harness that would allow to understand better what's happening close to realtime, I'd say "anybody who looked" in that case would probably be someone with another swarm tasked with analysis, it's no longer "glanceable" in a traditional sense.
The "sandbox" they used was apparently made of thin paper exposed under a day of heavy rain, too. You'd think, if they truly believed the model is so dangerous, they'd run it in a VM without a network adapter.
You don't need a full air gap. Set up a microVM with network access limited to local network and send all package requests through a filtering gateway that only allows normal download endpoints. Or self host a big collection of popular packages if you need extra security.
> Agents need packages like the rest of us. Ruby gems, npm packages, Maven, pip, docker images..
Yes, yes they do, but read through artifact proxies are dodgy as fuck, which is why and facebook (and I assume a fuckload others) don't have them.
Also semi-airgapped labs are a lot less expensive than you think at that scale. Once you have to do multi-region VLANs with machine certs before you get access to juicy VLANs, the difference between "no internet for you" and "mostly airgapped" falls to almost zero.
Also I would want an artifact mirror because a) that give a good signal about how the model reacts, and what training material its latched onto, b) it hides what the models are doing from the outside.
The best part is, they are totally able to one shot airgapping. You are too. Go ask your local agent to set up a pre-configured Linux VM with whatever stuff you want on it, then ask it to airgap it allowing only X, Y, Z services. It will one shot it.
You know what's better? They already do this per (paid) user - your ChatGPT subscription comes with a Linux VM that you can even legitimately SSH into, just ask your agent to configure it to accept your public key.
They absolutely know how to spin up VMs and configure them. They just made the conscious decision not to for the task where they specifically instructed the agents to hack stuff.
Anything that's accomplished simply by running software does not qualify as "airgapping" in my view. The entire premise is that real-world software is buggy and the LLM is much better at locating and exploiting those bugs than you are at preventing them.
But you can only connect to Wifi if you have Wifi hardware, and RF signals are contained by Faraday cages. Ethernet is still a thing for local connections.
> The entire premise is that real-world software is buggy and the LLM is much better at locating and exploiting those bugs than you are at preventing them
If that was the premise, why run LLMs in a lesser sandbox than a VM? Clearly it's not.
> If that was the premise, why run LLMs in a lesser sandbox than a VM? Clearly it's not.
Well, my mental model concludes: because they are incompetent WRT security, or at least they inappropriately trusted a third party that turned out to be incompetent.
They are not incompetent. I personally know a few people on their security team. They are world class security engineers. There's no way they didn't know.
Real air gapping isn't conceptually difficult. It's just a lot of effort and expense (although as far as I can tell, tiny compared to what's involved in building entire new data centres), and adds friction to your operation. (And you also have to have competent, trustworthy employees, as Stuxnet showed us.)
Not that airgapping is expensive so much as it's really, really inconvenient once you take it seriously. You need to build special rooms for it, you can't just API out to a datacenter. You need to have processes for requesting data be sent into the box. And so on.
I feel like there is a reasonable compromise between "yeah they have full internet access" and "separate airgapped rooms that require multiple levels of authorization to access" that would make this a lot better without that much more work.
I feel like they're doing it intentionally to show how dangerous these models are and that the government must step in and protect them
Compromise is possible; but if the stakes really are what the companies claim they are, and given the amount of money already involved, I really don't see why we should let them compromise.
> I brought this up to someone else and was told that airgapping is apparently much more expensive than I'd naively think.
These labs are one of the most valuable and heavily funded enterprises in the whole world, that they can't properly air-gap their systems to me reads as if they "agents" and LLMs are not as good as they say they are, because if they were, why would it be hard/expensive to air gap a system? They already scraped most if not all of the internet, where did that data go?
I'm not even talking about anything that the agents could help with. I'm talking about physical, real-world measures like https://en.wikipedia.org/wiki/Faraday_cage , removing Wifi hardware and running Ethernet cable for your local server, etc.
ChatGPT paid subscriptions already give you a VM [0]. OpenAI couldn't spare a few VMs for the super ultra mega dangerous evals where they asked the agents to specifically go hack stuff?
The issue as I understand it is that they didn't expect Artifactory to be vulnerable in the way it was, or else were completely not paying attention.
But I'm calling for something stronger than a VM here because we shouldn't rely on the VM being bulletproof just like we shouldn't rely on Artifactory being bulletproof. The access should be controlled at the hardware level. Like, networking on internal LANs only, and the entire thing inside a nice big Faraday cage just in case.
That's just unwarranted at this stage, given model capability and resource constraints.
Furthermore, VMs are isolation at the hardware level, particularly through hypervisors. I'd say given current LLM capabilities, it's a reasonable containment.
>The botnet scare is that article for me. It is the one claim in his essay that lands squarely in a field I’ve spent years in. It is just plain wrong. He either knows it and wrote it anyway, or he doesn’t and is publishing it regardless. Either way, it is not a good look for a man asking for an antitrust waiver based on this and other threats he forecasts.
This. I keep seeing ink spilled over the coming cyberpocalypse, but no one can indicate how other than "AI can find vulnerabilities" like not a single cybersecurity person has been consulted on the end of all things.
It seems those frontier labs found a clear proof that there's no clear way to block 3rd party from distilling their models, and they're now begging gov to keep their duopoly?
1) Dario keeps appealing to Trump, who obviously wants nothing to do with him, and will bash on him every change he gets. Dario isn't learning and it almost feels like Sam and Elon voted him KOM just to watch him get whacked by Trump, which was so easily predictable. Given the admonishments he received from David Sacks before he published his blog post, it's nutty he still pushed it out.
2) The frontier labs have people smart enough to build frontier lab tech but not smart enough to message on this matter more intelligently. It's pretty glum, how they keep trying the same tactic over and over. It's either cover for some other actions in the background, or they're operating way below par for this kind of campaign.
3) This is climate change all over again, but with the activist gun on the opposite side of the net (I'm mixing all the metaphors so you know this isn't AI written). The language and pleas are very identical though. Before, climate activists wanted the government to control GHGs releases by everyone, now the loudest voices want the government to control frontier AI by.. themselves.
It's comically misbegotten. And I have a work meeting about it on Wednesday.
Hm, I thought there was a lot of interest after Hugging Face in broader tech and it started breaking into the media with articles in the NYT, etc. Then with the Jacob Coxon tweet, appearances on national TV, Bernie superintelligence ban it seemed to be going mainstream.
I'm pretty confident in asserting that no industry in the history of industry has ever gone from birth to full regulatory capture faster than the AI industry has.
> They have shown time and time again that they are not to be trusted, yet, the main ask is to trust us, only us.
Exactly. For everyone outside the US, we see a tech industry that has spent 25 years moving fast and breaking things AND elevated Trump to be POTUS - which has destabilized the rest of the world. This is after 70 years of the US invading and destroying small countries, often without plans or robust reason, all in the name of "democracy" (ask any other country if they feel like they had a vote in the US's actions)
Why the hell would we put our trust in a few AI Labs in the US, when this is the legacy they will be reinforcing? It has to be Open models - for the people. No more of this US-paternalistic bullshit.
If by “we” you mean “experts and professionals outside the US”, I think the short answer is they’re not talking to you here. They’re trying to secure regulatory capture in the US, and monopolize their hold on US corporations. To the extent they’re thinking of other “markets”, I would guess the assume the outsized influence of US corporations will have the much of the rest of the world using their products.
Yeah fair point. I believe he truly has humanity's best interests at heart.. And yet he keeps speaking as though everyone still sees the US as the shining beacon of the world.
That's simply not true anymore, and his messaging will continue to be undermined until he treats other countries as equals (whether that's China, Australia, or anywhere else).
> I believe he truly has humanity's best interests at heart
How can you possibly believe this? What else would someone in his position say? No billionaire has humanity's interest at heart, so they try to hide that fact by building libraries (Carnegie) or creating foundations named after themselves.
All of his words and none of his actions support his caring at all about humanity, and you choose to believe his words? The man has an IPO coming up (probably one of the biggest ever!), he could not be in a less trustworthy position.
> I don’t know what freedom and democracy have to do with AI and the frontier labs. Unless of course Dario is a fan of Neon Genesis Evangelion and dreams of govts run by the three magi. Freedom and democracy for $200 does sound enticing, I won’t lie.
Well, every human will become an ecosystem. Human + 500 agents as advisors. (In the case of important humans, most of them operated by foreign governments and corporations, obviously.)
278 comments
[ 2.6 ms ] story [ 171 ms ] threadThis incident has Dario shook, there ain’t no such thing as halfway crooks.
I've never met someone with so little to criticize about them that I'd mention they participate on a message board of people I don't like. It seems like admitting you're out of criticism; or have an unrelated hair across your ass about a community you don't like. I only know of LessWrong from offhanded mentions from Scott Alexander back when his blog was popping off regularly, and when it gets used for guilt by association. Does it really have that bad of a rep?
I don't even know how to approach such a thing, I can't imagine even in the stereotypical examples like the typewriter becoming obsolete, were any downstream deaths worth it? Or is this a totally nonsensical sentiment to begin with?
This article will be drowned out unfortunately by the press and bloggers following the Misanthropic cult.
To my mind, the last great arms race between nation states was a misdirected love triangle between USA, Russia, and The Bomb, and look at all the damage that did.
Since AI is the new arms race between USA and China, slowing down may just give the humans involved enough time to realize they should be loving one another, instead of the machines.
Maybe saying, "let's slow down", is another way of saying, "I love you."
Or, maybe it's just: "let's not all of humanity kill ourselves like some bad ending to a Shakespearean tragedy."
Either way, it's a better note than, "We must achieve sea/air/nuclear/quantum/AI/spiritual supremacy before those other bastards do!"
So, it's about competition inside the US market, with strong indications of an impeding losing scenario on raw economics (it has nothing to do with AGI, just price).
So my proposal is AI companies decide which labs have come close to frontier and decide to slow it. Government decide to stop progress in that and they divide the revenue from all labs(for say 10 years), without any matter of where it is coming from. Any lab which reaches close to frontier gets a chunk in the pie. This will encourage labs to come close to the frontier but not dangerously close.
Prevented a world war for 80+ years.
It worked.
If Stanislav Petrov hadn't been in the chain of command, MAD might not have worked. And it could still fail to work in the future.
https://en.wikipedia.org/wiki/1983_Soviet_nuclear_false_alar... https://en.wikipedia.org/wiki/Stanislav_Petrov
Fact: There was no world war.
Impossible to prove hypothesis: nuclear weapons prevented a world war.
Facing the facts about nuclear weapons means owning the good (probably prevented wars) and the bad (at the very least there were severe environmental and economic consequences).
Akshually there is, and WW3 has been going on since 2010. (Mostly in places that aren't Europe.)
War has still been going on since 1945, much earlier if not always, so akshually WW2 never ended or is just how it's always been? (mostly in places that are not "first world" / The West)
1945 was clearly that. 1920 with the League of Nations attempted to be that, but failed.
Whatever happens after the current turbulence we can be sure that there won't be a UN and NATO and a World Bank at the end.
P.S. The first modern world war was the Thirty Year's War.
Have you put your money behind this on future gambling sites? How can you be so sure?
Pls dont make me write out her history
its filth
Can you be specific about the damage? We currently live in the most prosperous times on earth for humans. I'm not sure what you mean by damage.
Nobody can explain why an LLM can be so capable as to be able to wipe out humanity and pose a greater threat than nuclear bombs but not be so capable as to be able to protect humanity against that threat. Are we just handwaving this with "entropy"?
> Maybe saying, "let's slow down", is another way of saying, "I love you." Or, maybe it's just: "let's not all of humanity kill ourselves like some bad ending to a Shakespearean tragedy."
Okay nevermind, I think it's pretty clear you just want to wax poetic about all of this.
https://en.wikipedia.org/wiki/Starfish_Prime
https://storymaps.arcgis.com/stories/3f62c90925f64fc09425be8...
Of course there is/was plenty of human damage as well.
https://www.msn.com/en-us/news/world/4-000-000-early-deaths-...
It is absolutely explained (for those who actually care about reading). Simply put, AIs are working more and more like blackboxes - there's no guarantee that an AI of the future will be aligned, or if it will be faking alignment. This is not speculation - alignment faking has been observed in experiments. This is exactly why Astra's developments have been worrying (in principle).
And bear in mind that recursive AI development started already to be a thing. Which means: inner misalignment may trickle down the generations, and humans won't detect it.
Having said that, of course, it can be predicted if and how misalignment will take place. But it's absolutely a plausible scenario.
Regarding the physical possibility: AI is in its infancy; think of it as Arpanet. Developers 60 years ago couldn't imagine it would be ubiquitous. AI will be ubiquitous the same way.
I know you think you explained it but you didn't. You explained how an LLM might become misaligned but for the LLMs that are not, why would they not be capable of detecting and defending against the misaligned LLMs?
Elaborate on why LLMs are so capable that they are a threat to humanity and at the same time, they are so incapable of defending us?
I'll give you a clue, nobody, including Dario, can answer this question because one contradicts the other.
Having nukes at all, if not a nuclear triad, seems to be the most effective way for a country to have its sovereignty respected.
There have been precisely 0 nuclear weapons detonated (outside of testing) since that arms race began.
But why should he decided when to slow down?
Most people who have worries about AI for all sorts of reasons (most of them not-Skynet related) wanted to slow down way before this.
Instead of "hey, look at this brilliant new idea I just had on my own to slow down" maybe we should have gotten a "sorry everyone, the folks asking for a slow down earlier were right and visionaries, and we were foolish".
So, you can't blame whoever says this is bullshit, because it has bullshit all over it. I like Anthropic's products, and it seems the best of the bunch in regards to alignment, but Jesus these stunts are terrible.
this is a level of hippie delusion i wasnt aware existed unironically
china is never slowing down, therefore the us shouldnt either
Why are all these pro-regulation arguments so nonsensical…
Open Source AI democratizes the means of production to anyone with a computer. And yet, the hyper capitalists are defending it, and the progressives think it should be exclusively in the hands of 1-2 large corporations.
Anthropic needs regulation in order to prevent AI from becoming a commodity. This of course does not benefit all tech businesses equally, especially those that are not currently at the frontier. So when JD Vance talks about AI, he talks using the mouth of Peter Thiel who may not see benefit from the same policy as Altman or Amodei.
The rest is just public support posturing and most of that is bullshit. The philosophy is money and power, who gets it and who doesn't. Open source is fundamentally a vehicle for commoditization. This is great if your business is not AI and your business is instead something like GPU hardware or software that uses AI. But it means that eventually, selling AI is not going to be the money maker.
Open source is a business strategy called "commoditizing your complements". These big companies don't do it out of benevolence. Back in the day OSI pitched them a business strategy that FSF did not, and it caught on. And the software business became about ads, consulting and cloud services instead.
Also, Democratic Republic of {Congo, Korea}.
The other insanity in all this the smartest computer scientists in the world are asking Congress to regulate them. Come. On. Really? Do we remember “The internet is not a truck, it’s a series of tubes…”
Why can’t the big labs form a Save The World Consortium and self-regulate?
Non-democratic counties (hey there China) will not abide by any agreement that constrains their advantage. It’s naive to think so.
What this conversation lacks is enough discussion of how these models can cause us harm—we are are so worried about AI but we allow Windows in critical infrastructure; we build JS/TS apps with thousands of dependencies; we generally don’t segment networks well enough; we don’t have adequate (sometimes any) detection capabilities in our systems, and so on.
The problem is that there can not be any outside party to regulate this on a global scale
Applying game theory problems like prisoner's dilemma to the real world requires collapsing a lot of overlapping and conflicting incentives to a single dimension which doesn't always work.
So we are in a true prisoner's dilemma, if one of the companies / countries stops R&D it risks handing over godlike power to another company / country.
The main counterexample to this was nuclear weapons. Atom bombs have not been used to kill since the US did so. However today, the two largest nuclear powers have no legal agreement on arms control because Donald the Trump declined Russia's offer for an extension to the existing agreement. Now other countries are looking at Ukraine, Iran (attacked for wanting nukes) vs NKorea (not attacked because they have them), and Donald's own musings about the US nuclear umbrella being a bad idea (France is going to build more nukes now too)... and we now face nuclear proliferation again on a global scale, with tech that is nearly 100 years old now.
I'm wondering why no one is mentioning the "accountability" word. Why these companies are allowed to damage others with impunity?
Start making managers pay the price for their actions, and watch how the models magically slow down on their own.
Because investors have pumped hundreds of billions into AI and real consequences put that money (and growth) at risk.
Another part is a completely defanged administration she it comes to effectively regulating anything.
Another bit is money.
I also find this whole "its so good, its scary" flex a little less impressive when you consider they access to millions of GPUs?
The AI buildout has been one of, if not the largest, focussed capital investment in history. The 2 big AI labs are the final customer for something like 20-33% of all datacenter compute in the pipeline.. up to 70% when you look at hyperscaler "AI revenue" from the big 3.
I don't think any single entity has had remotely this much compute available in history.
Obviously this will have huge impact on some companies valuations, but you can have one's cake and eat it too.
That aside, I'm not sure why it's particularly interesting they have all this "compute" (let's just assume for the sake of argument it's all "live"--that is they can actually run workloads on all of the "compute" they have on paper). So what if it's the biggest amount ever? Why would that be meaningful? Is there some economically viable problem you're aware of that is somehow dominant in that way?
So I think part of what is going on now is not just the method (LLMs) but the means (supercomputer levels of compute) at unprecedented scale of concentration.
However, if I put my sci-fi hat on for a second, it's not so far fetched that we figure out a way to compress models to a size where it wouldn't need all that compute.
Then any Tom Dick or Harry can just use the exploit to get into a truly secure system.
Also I think the cyber threat is overstated in many ways. Most mid 2000s cars are completely immune to any cyberattacks by pure chance. As is most 'advanced' hardware and infrastructure made before the year 1990 (that wasn't PC based). If one invests seriously in anti cyber capability, I'm sure they can aspire to the same level.
I would say that just having a non-programmable flash and a system that can go back to its factory state just by cycling the power is a long way to getting there.
A "scapegoat" is someone who is incorrectly blamed for someone else's errors or sins. The perspective you're responding to is this: They built the system, they run the system, they have continuously warned "This system is dangerous!", and yet persisted. That is not being incorrectly blamed, not being a scapegoat, and instead is a collaborator.
So I think you mean to ask: "Why do people focus so much on finding someone to blame?" It's not merely semantic, because the answer to that is more straightforward: Consistent accountability is a major factor in deterring bad behavior. It is not the only factor, but it is a major one.
That is my Steel Man understanding of the people searching for individual blame.
Countries demand reparation for damages in war. Citizens of those countries sue for damages and win.
Accountability is not a foreign concept. And the point is to disincentivize negligence. Because negligence is cheaper. And in this case, accidental hacks are marketing spend.
Whatever you try to make laws for now will be irrelevant in 1-2 years. You either have to go extremely broad, like the EU does it, and accept that people will find loopholes, or you need to target specific technologies which is a hard job for the same reason.
In any way, ita already a lost cause cause you move slower than the tech. A plausible prediction for AGI is actually a social collapse in the moment when society cannot keep up with everyday life because of the pace of change being so fast that no existing laws can handle it
"A series of tubes" was the same kind of political character assassination that led to Howard Dean getting ridiculed for his infamous scream. He butchered the sentence. Fair. But Stevens should be ridiculed for parroting a tech industry lobby stance about net neutrality, not for the series of tubes metaphor.
You are probably too young to remember that the dominant metaphor for the internet in 1990s politics was "the information superhighway." It was easy to think of the web as "driving" browsers to visit web "sites", with slow bandwidth being analogous to being caught in traffic. But the internet is closer to water, gas, and electricity than roads. Concepts like bandwidth and throughput are closer to how they play out in infrastructure policy for various things with tubes, versus cars and roads. Do you think he's wrong and that the internet is closer to "a big truck" versus "a series of tubes"?
The issue being debated was net neutrality and bandwidth, including specifics about who pays for what and the downstream second-order consequences of various policies. He was parroting some line from some telecom lobbyist, but the point the lobbyist was trying to make through Stevens was about how if certain policies about who pays for bandwidth were adopted, it could disincentivize some things at the Tier 1 layer that could increase transport costs at the Tier 2/3 layer.
If I build a robot that murders my neighbor, I’m still at fault.
We don’t absolve drivers of responsibility because of cruise control.
In that sense, AI is nothing new. If it is abused to cause harm, the person behind it should be liable.
What if a person physically broke into the car and did the same thing? Clearly they are the one to blame then.
The whole person in the loop is liable is already an outdated concept when decisions are made beyond the persons physical control.
If GM does something (or fails to do something) to their vehicle that causes me to crash, they're liable for the crash.
If someone cuts my brake lines (alters my vehicle) and I crash my car and kill someone, the person that cut the lines is responsible. I have to prove the context of course, and or an investigator has to do so.
And the responsible entity may refuse to pay up, may refuse to take responsibility. None of that is new either.
Same thing here - If they build it and it does $100 in damages (and we arrest them for it), that's their problem. If they build it and it does $100B in damages, that's everyone's problem. Even if they do get arrested after the fact.
Yes we should have charges and damages for everything on https://www.felonybench.com/, but that doesn't address the core issue of this being possible at all.
we have the 2008 crisis to wit. And the involved supposedly failed math models and lines of responsibilities and other involved financial relationships were much simpler and clearer and of the types well known to the law and regulators, yet...
At this point, in the US at least, the laws give the corrupt elite the ability to deter competition and to target the people who try to get in their way.
In other words, it's protection for the potentate and his sycophants, not the plebs.
> If we passed a law that said the CEO of any company that deploys an LLM that commits a crime gets punished as if they personally did the crime (so, basically instant life sentence if it's even a simple crime times a million instances), I guarantee you the first email the CEO sends to the company is a "pause every LLM project we have - we gotta think about this".
Why do no such laws exist in practice? Why are corporate crimes almost always settled by payment, not individual punishment?
If you answer these questions, you'll know why the law you envision has a lower chance of being enacted than AI destroying humanity.
There are laws, but if you’re rich enough, the laws don’t apply.
Laws are the last line of defense. People don’t do bad things primarily because their human nature and moral compass stops them from doing so.
People don't do bad things because they have nothing to gain from them. Ask someone to do something evil as part of their job, they'll often do it.
There's lots of ways that you could go out of your way to hurt someone and get away with it. There are not that many ways you could go out of your way to hurt someone, get away with it, and significantly profit.
I have plenty to gain from not following the rules, but I would not feel good about myself.
Because there are many important values of "something" where such punishment acts as a deterrent to other would-be criminals; and because society is more stable when people see justice done and that mollifies hoi polloi after the damage occurs.
But many AI doom scenarios don't fit that paradigm. The first time "something" happens (at least if you buy the argument) would be bad enough for legal punishment not to matter.
What people call plagiarism when LLMs do it is, if I understand correctly, allowed because of the history of the web and search engines going back to the very early days of the web.
Piracy, a separate act that has definitely occurred, has been found unlawful.
* The lower bound is an industrial accident. Fully automated Union Carbide / Bhopal comes to mind; smaller industrial accidents are already more common and I wouldn't be surprised if smaller incidents have already been caused by LLM-given advice, which is why I think it would have to be around "thousands dead" dangerous before people really take it seriously.
The upper bound is reachable many ways. Perhaps via a non-novel virus whose genome is already recorded getting mass-printed by many different DNA/RNA printing labs around the world? Perhaps via a wargame whose "role playing" leaks in stupid ways (either by becoming hot or by sycophantically giving one side a false belief how a war will go)? Perhaps by propaganda turning genocidal? Perhaps by convincing a government to follow a flawed policy, similarly to the Four Pests campaign in China's Great Leap Forward?
> Yes we should have charges and damages for everything on https://www.felonybench.com/, but that doesn't address the core issue of this being possible at all.
One could argue that the "HuggingFace incident" is already "something" which should be investigated and punished very heavily.
It's kinda ridiculous to build a machine that attacks a competitor of yours and the CEO gets to write blog-posts on how fascinating this machine is
Even if you allow that the LLM can't be held responsible, the sandboxes were clearly not up to the task and the whole incident was mismanaged. There are people who made those decisions and should be held accountable for them.
which the government doesnt want since this will stop progress, while other countries will continue to develop LLMs
It would be more like "Switch off all APIs right now. Don't even bother with a safe shutdown process, cut power to those buildings, including backup generators. You are free to use firearms or thermite if the switches have been locked off".
This would be a rather weird change to corporate law given CEOs are not by default held to that standard by anything else their products or staff do.
Note that I'm not entirely disagreeing with you here. It may even be correct to pass such a law. But it would be very weird.
A lot of problems with America could be fixed if the Department of Justice actually grew a pair and prosecuted people for committing criminal acts.
The fact that Les Wexner is not rotting in prison for facilitating Epstein and his friends exploits speaks volumes to the state of this country.
If corporations are legally people, they should be held to the same standard as natural persons.
As it stands, the “restitution” that is required is so insignificant that it’s treated as an additional tax. If there is no deterrence, there is no behavioral modification. If there is no behavioral modification, the law becomes unenforceable.
3M polluted groundwater in Minnesota for 50 years[1]; Nestlé misled mothers in order to make them stop breastfeeding and switch to their formula which killed babies [2]; Both copmanies are still doing business today.
[1]: https://en.wikipedia.org/wiki/3M_contamination_of_Minnesota_... [2]: https://en.wikipedia.org/wiki/1977_Nestl%C3%A9_boycott
You’re worried about companies. I’m far more concerned when governments are involved.
The issue is still being debated today, particularly around Canada’s VISP and the time some claimants have spent waiting for compensation. Some people like Pollock who became paralyzed were offered MAID (medical assitance in dying). That's some audacity.
This thread is about accountability. So who should be accountable for this?
When an AI bot injures you, you can call the owner to account. But not until then. You have no standing to demand "accountability".
You should learn about this wonderful thing called democracy. Also why not everyone is allowed to work with radioactive material in their shed.
The problem here can be personified as "Trump", and it's the same problem that applies to coal.
Coal has a price besides money. It has historically been dangerous work, killing miners. It produces dangerous waste, both during mining and when burned, both as solid residue and the gasses emitted. The problems have been known for a long time. The workers themselves have called for better safety requirements and gone on strike for such things.
Why these companies are allowed to damage others with impunity?
Coal continues to be burned, because power is power. It's so important that sometimes the government steps in against the unions, rather than being on their side. Despite calls for this, we've not been able to get the owners of the coal mines, nor the coal burners, to "pay the price for their actions".
AI? Famously, knowledge is power.
Trump wants that power. He's not the only one, but he is the avatar of those who put their feet on the scales to not only allow but in some cases require (DoD vs. Anthropic) these companies to damage others with impunity.
That is the status quo we entered AI age with.
Security has long been a lottery - Probably most systems are exploitable, but the cost of developing such an exploit is expensive and the punishments for using such an exploit are large enough that it's not an everyday problem.
AI breaks both axes. Developing exploits is far more efficient using LLMs instead of humans, and LLMs don't (and can't) fear the reprisal and consequences the same way.
I do hope humanity will be able to mitigate these hacks, but we should expect them to continue and to become more severe on our present course.
By definition that needs a command and control server, the ability to execute tasks on demand and regular pings to the C2.
> You need one root certificate. You need one windows update. You need one backdoor in xz.
Certs can be revoked. Updates can be rolled back. We have had the backdoor in xz already. You seem to underestimate the modern security stack and OpenAI and Anthropic are _not_ good examples.
The asymmetry in red/blue scenarios will be transient in nature. You won't have cost of developing exploits fall without the cost of securing the systems also falling.
> By definition that needs a command and control server, the ability to execute tasks on demand and regular pings to the C2.
I am not sure if you are agreeing or disagreeing, but I am saying just in case, that Claude can be remotely controlled from the cloud on any machine it is installed and set running, and can be commanded remotely.
I agree that blue will eventually be made cheaper / faster with AI integration, but diffusion of those techniques inside larger organizations will always be a lot slower than diffusion of those techniques for individual attackers / rogue agents / APTs. Larger ships are slower to turn.
As a biologist, this is the most annoying thing about Anthropic for me. If they really cared about improving health they would set up a trusted-access program so that biologists can use Mythos (et al) safely. Instead they're trying to monopolize biology.
It reminds me of the export controls on PlayStation 2 consoles because it was deemed that 6 gigaflops was a "dangerous" amount of computer power, and it couldn't be allowed to fall into the hands of opposing militaries: https://www.latimes.com/archives/la-xpm-2000-apr-17-fi-20482...
Now the phone in my pocket does 2,500 gigaflops on battery power, and nobody seems interested in banning its export because of that.
I don’t agree with them, but I don’t think it was the raw compute power as much as it was maintaining the _delta_ in compute power.
That’s the recurring theme with these companies. They are not there to serve anybody else, but only themselves. They let you use their infrastructure so they can collect all the knowledge and data, and then they take it from you to reap all the benefits and profits.
If a year from now we have a model that is 2-5x of Fable/Astra that is definitely world changing.
But come one don't repeat stuff like this:
"Remember this man has been saying software development will be solved in “6-12 months” forever now."
Don't downplay if people get timelines a little bit wrong. No one could even imagine a system writing and analysing code just a few years back.
These people are trying to handle something very unique. And while they have access to information we do not have, even more peple are absolutly oblivouse that AI/AGI is a real risk to their lives (job loss etc.)
Edit: Boris Cherny, the lead of Claude Code did say on a podcast that programming seemed "largely solved" "for the kind of programming I do" (writing harnesses I presume). Maybe that's what they were confusing it for.
And this is actually, scarily, true...
Obvious bugs and low quality software are nothing new, but something feels new about it. Occam’s razor says LLM coding is a likely culprit, but it could also be management style encouraging this sort of carelessness from the top down.
And how fast it's happening, with formerly respected senior engineers willingly participating in the enshittification, signing off on piles of slop with "Looks good to me" without barely a glance at the machine-generated code. Requested features built on insane architecture that no self-respecting developer would let themselves be responsible for, are waved in with implicit trust that the "AI" knows what it's doing. It often comes with tons of newly added tests that look good on the surface but are off the mark or missing the point entirely.
It's worse than letting juniors trash a codebase, because there is no learning happening in the process, so the quality never gets any better, it's unpredictable - sometimes great, other times utter shit code. Comments with dense vocabulary that looks like high IQ yet make no sense on scrutiny, some that look like accidental Dada poetry. It's widespread mania and crowd psychosis.
Of course there are some who are using the new technology responsibly and wisely, but very few.
US will win AI, AI will not teleport the earth into the center of the sun. If you are smarter than Trump then why are none of you president?
I work in non AI robotics and if we had a system that in the course of doing what we told it to did something we didn’t want it to do (what AI companies called being misaligned) we would call it a bug and fix it with the fix being prioritized based on how bad the thing we didn’t want the robot to do is.
Sometimes preventing the robot from doing dumb stuff also means the robot can’t do smart stuff that it would be able to do if we left some code in. We balance the two factors out based on our understanding of what our customers want.
Obviously LLMs are more complex than what I do, but it doesn’t feel like it’s by THAT much.
So why does the government need to be involved again?
I strikes me as unlikely that public opinion will succeed with AI where it’s failed with other existential crises.
Not saying you’re wrong but if I wanted to cultivate a mass hysteria as cover for a regulatory capture power play, this is exactly what I’d want everyone to believe.
We had OpenAI "accidentally" run an entire swarm of 10,000 agents apparently for weeks, on a security related task, seemingly totally unsupervised, hacking all over the internet - all the conversations were completely visible, anybody who looked would have seen it. But they didn't.
So before we start regulating innocent parties, maybe let's start by taking some direct action against the specific ones that appear to be behaving with criminal levels of negligence.
While I do think OpenAI were negligent in not developing the harness that would allow to understand better what's happening close to realtime, I'd say "anybody who looked" in that case would probably be someone with another swarm tasked with analysis, it's no longer "glanceable" in a traditional sense.
I still think this is a sign that they are not taking their own rhetoric seriously.
It is also a choice to not do any or all of the above.
Not surprised this is always what they have and hack.
Who would use an Agent that spends $10,000 re-implementing some OAuth lib or reverse-engineering a proprietary lib when it's free on the internet?
Yes, yes they do, but read through artifact proxies are dodgy as fuck, which is why and facebook (and I assume a fuckload others) don't have them.
Also semi-airgapped labs are a lot less expensive than you think at that scale. Once you have to do multi-region VLANs with machine certs before you get access to juicy VLANs, the difference between "no internet for you" and "mostly airgapped" falls to almost zero.
Also I would want an artifact mirror because a) that give a good signal about how the model reacts, and what training material its latched onto, b) it hides what the models are doing from the outside.
if your internal models are so damn good, they should be able to "one shot" airgapping... right?
You know what's better? They already do this per (paid) user - your ChatGPT subscription comes with a Linux VM that you can even legitimately SSH into, just ask your agent to configure it to accept your public key.
They absolutely know how to spin up VMs and configure them. They just made the conscious decision not to for the task where they specifically instructed the agents to hack stuff.
But you can only connect to Wifi if you have Wifi hardware, and RF signals are contained by Faraday cages. Ethernet is still a thing for local connections.
If that was the premise, why run LLMs in a lesser sandbox than a VM? Clearly it's not.
Well, my mental model concludes: because they are incompetent WRT security, or at least they inappropriately trusted a third party that turned out to be incompetent.
These labs are one of the most valuable and heavily funded enterprises in the whole world, that they can't properly air-gap their systems to me reads as if they "agents" and LLMs are not as good as they say they are, because if they were, why would it be hard/expensive to air gap a system? They already scraped most if not all of the internet, where did that data go?
[0] https://news.ycombinator.com/item?id=49718530
But I'm calling for something stronger than a VM here because we shouldn't rely on the VM being bulletproof just like we shouldn't rely on Artifactory being bulletproof. The access should be controlled at the hardware level. Like, networking on internal LANs only, and the entire thing inside a nice big Faraday cage just in case.
Furthermore, VMs are isolation at the hardware level, particularly through hypervisors. I'd say given current LLM capabilities, it's a reasonable containment.
They would have been watching what it does, especially when running it on ExploitGym of all benchmarks... that is criminal worthy neglegence
"They" don't care about the end-people. "They" care about maximising their profit thing, in a vacuum.
This. I keep seeing ink spilled over the coming cyberpocalypse, but no one can indicate how other than "AI can find vulnerabilities" like not a single cybersecurity person has been consulted on the end of all things.
1) Dario keeps appealing to Trump, who obviously wants nothing to do with him, and will bash on him every change he gets. Dario isn't learning and it almost feels like Sam and Elon voted him KOM just to watch him get whacked by Trump, which was so easily predictable. Given the admonishments he received from David Sacks before he published his blog post, it's nutty he still pushed it out.
2) The frontier labs have people smart enough to build frontier lab tech but not smart enough to message on this matter more intelligently. It's pretty glum, how they keep trying the same tactic over and over. It's either cover for some other actions in the background, or they're operating way below par for this kind of campaign.
3) This is climate change all over again, but with the activist gun on the opposite side of the net (I'm mixing all the metaphors so you know this isn't AI written). The language and pleas are very identical though. Before, climate activists wanted the government to control GHGs releases by everyone, now the loudest voices want the government to control frontier AI by.. themselves.
It's comically misbegotten. And I have a work meeting about it on Wednesday.
How would you have communicated this?
Exactly. For everyone outside the US, we see a tech industry that has spent 25 years moving fast and breaking things AND elevated Trump to be POTUS - which has destabilized the rest of the world. This is after 70 years of the US invading and destroying small countries, often without plans or robust reason, all in the name of "democracy" (ask any other country if they feel like they had a vote in the US's actions)
Why the hell would we put our trust in a few AI Labs in the US, when this is the legacy they will be reinforcing? It has to be Open models - for the people. No more of this US-paternalistic bullshit.
That's simply not true anymore, and his messaging will continue to be undermined until he treats other countries as equals (whether that's China, Australia, or anywhere else).
How can you possibly believe this? What else would someone in his position say? No billionaire has humanity's interest at heart, so they try to hide that fact by building libraries (Carnegie) or creating foundations named after themselves.
All of his words and none of his actions support his caring at all about humanity, and you choose to believe his words? The man has an IPO coming up (probably one of the biggest ever!), he could not be in a less trustworthy position.
Well, every human will become an ecosystem. Human + 500 agents as advisors. (In the case of important humans, most of them operated by foreign governments and corporations, obviously.)