I once had to try to explain the meaning of "object code" to an older lawyer, who wasn't really interested in the answer to his own question. It did not go well.
Bringing charges is at the discretion of the prosecutor, and the CEOs of these companies are dining with the president. Learn a little bit about how power works.
They're US companies and I've heard that it requires intent according to US penal law. I find the article's claim that intent is established dubious. Since this is hard to prove without a whistleblower, it seems that they only have to fear civil lawsuits, and given that we speak of trillion+ USD companies they might feel confident to be able to fend these off or simply pay out the victims in settlements.
Given that we have had journalists prosecuted under criminal hacking statutes for discovering personal information in the rendered HTML of a web page, or publicly available attachments on a WordPress installation, I find the suggestion that intent is required to be dubious.
I was referring to their current market valuation. Of course, it could be inflated, yet it's a fact that they could easily spend hundreds of millions of dollars on lawsuits and settlements. I'd think they'd offer some settlements to some the companies that were attacked already. Perhaps I'm wrong, though.
technically, there is no current market valuation. These are all private companies. They have no reason to give anything but trillion dollar valuations.
You're technically right but the point was merely made to illustrate that these companies have plenty of money to deal with civil lawsuits and offer settlements. I hope you don't doubt that.
Man if only there were laws governing something you own (i.e. a cow) that covers when that thing you own does damage to someone's property (destroys your neighbor's fence) but didn't have ill intent as it doesn't have sentience.
If only that sort of thing had been dealt with in the past......
I doubt this will catch on outside of luddite/cynical circles unless one of these attacks does some actual damage.
Ignoring this, I would have appreciated more examples of what it would take for an act to lack intent. Because the definition of intent used in this article sounds so broad as to cover basically anything that ever happens.
Accidentally run someone over? Clearly you knew that was a risk when you started driving, you should have been more careful. Intent.
until one of these attacks does some actual damage**
Sadly, it seems that we as a species has opted into information through tragedy. I wish I could see another way, but people are just so dug in. It's depressing and shocking how few people realize we're living through the singularity, even ~4 years into the damn thing...
Where does it define intent? Maybe there is precedent here, but I wouldn't try to guess which way a court would go when determining whether e.g. the huggingface incident qualifies.
It's interesting that you bring up vehicle / pedestrian accidents, as there is an entire history of that debate being reframed in favor of car companies.
People used to blame the car companies, and then the car companies lobbied to make "jaywalking" a crime, at which point, the frame was reset and now it was the pedestrians to blame, if I recall the story correctly.
They are doing a good job with driverless cars. Someone recently took a Waymo, it went the wrong way down a one way street, they mashed the pull over button and got out, and then told the reporter they'd use the service again.
Fun fact a "jay" in jaywalker was a derogatory term at the time. Meaning
"a rube" or "country bumkin". It was initially thrown at drivers, but car companies successfully flipped the insult.
>People used to blame the car companies, and then the car companies lobbied to make "jaywalking" a crime, at which point, the frame was reset and now it was the pedestrians to blame, if I recall the story correctly.
Surely the car companies had a point her though? It does not make sense to blame the car company for a car crash unless it was caused by mechanical failure
That didn't seem to have precluded the prosecution of Swartz. I'm skeptical that any complaint actually needs to be filed. I'm pretty sure that the FBI can investigate and the DoJ can prosecute without one.
IMO an analogous situation would be where a police officer acts to prevent a in-progress burglary he has stumbled across even in the absence of a complaint by the owner of the property that's being burgled.
> ...but the police have to be aware of the supposed crime.
Do Federal police and Federal prosecutors not read the news? I'd be shocked if they didn't. And while I would 100% believe a claim that not every cop and prosecutor is aware of tech news, I would find any argument that those that deal with "cybercrime" were unaware of the news to be completely incredible.
I'll also note that -unless my memory is failing me here- the only charges against Swartz that had anything to do with MIT were dropped very early on, whereas the charges that had to do with the "attack" on JSTOR's systems remained.
As I said, in many cases, the actual victims don't need to file complaints for The Federales to either begin investigations or press charges.
> I'm not saying somebody has to press charges...
Yeah, but you did? [0]
How many of the victim companies have filed complaints with the FBI? As I understand it, that would be step one.
That might not be what you intended to say, but it is what you said. No worries, though... shit happens.
I can file a complaint with the police that you broke into my computer but if charges are pressed that's done by the state and it's the state that is the plaintiff.
Not sure who you are quoting. I certainly didn't use the words "pressing charges"...
If I detect a crime on my property and don't notify the police (ie file a complaint), the police aren't going to go after anybody because they don't even know something has happened.
In this case you could say it's been in the news enough that there should be an investigation. Maybe there is, who knows? If the victim isn't going to cooperate, it's going to be a difficult investigation and prosecution and those resources may be better spent elsewhere.
I just did a quick search to see what the DOJ's declination rate is and I couldn't find a definitive answer. Apparently they decline to prosecute about one third of the cases referred to them and I did see a story from ProPublica that said after Pam Bondi took over, they reprioritized towards immigration cases and dropped tens of thousands of other criminal cases.
Or maybe the AI companies already self-reported these violations? Last year the DOJ set up a policy to not prosecute a lot of crime if the company self-reports and perhaps this fits the criteria for that?
IMO, yes. You are the computer operator, and you operated the computer in such a way as to cause it to take that action. Others will mention "mens rea" but that doesn't really apply to negligence.
this, people need to just focus on "computer program did thing". Its irrelevant how complex the program is. If I write a worm "for testing" and then it escapes my system and does massive damage, that's possibly a negligent crime, despite it being "too complex to understand" or "having its own motives".
Yes. I would argue it's who's managing the harness and agent is responsible. If you mean you're running an agent locally then it is 100% your responsibility. If you're using some trading platform and all you did was prompt, whoever was running the harness that accepted the model's outputs to hack a third party is responsible for the hack, in my opinion.
If you really didn't know and didn't mean for it to happen? And assuming you were the first to find out you immediately went to the authorities/injured parties?
If all that's the case I'd say you at least should be in the clear. There's going to be at least one if not more public agencies that would presumably be looking at you with a proctoscope so I'd be hoping all of those claims would stand up to a lot of daylight.
No one with real money or power wants to cross OpenAI or Anthropic just yet. It's a closed ecosystem. OpenAI hacks HuggingFace, Nvidia soon after buys HuggingFace while it also before backed OpenAI. In the end, they just handle it all behind closed doors like family business.
The real answer is that crime generally requires mens rea. At some point, however, letting your AI break into other systems will be conventionally understood as an act of negligence, and it will get prosecuted accordingly. We're not at that point yet.
You know very well, that if me or you had our LLM hack into some company, without us prompting it to, maybe we're pentesting our internal network and it 'hallucinates' we'd be in handcuffs.
"corporations" don't make and operate AI's, People do. But let's use your lens.
The "usual legal action": "best case" is the corporation is sued and charged money for the hack. But they likely settle well before that verdict. "worst case": absolutely nothing. Not even a cheap settlement. Maybe some corporate apology made to prevent any legal liability.
If we want hacks to be taken seriously: what should happen is that the CEO of such a company is held liable, since the "corporation" has a representative and a human needs to be held liable for such actions.
>Question: If a corporation not a person, mistakenly hacks some random websites
How does a "corporation" even hack a site? You might say "employees", but as soon as you get them involved then you'd go after them, especially for something that's obviously illegal like hacking.
Isn't that basically privateering in the 21st century? In other words, you're safe from the country that's sponsoring you, but obviously that protection doesn't extend to other countries.
I think if Boeing made some faulty planes that fell out of the sky, their employees aren't usually charged. Boeing is definitely sued and held criminally liable sometimes? So I don't think this is true.
Isn't that the whole point of corporations existing? That the actions and results are a result of the agglomeration of everyone (esp leadership) and not just one employee.
>I think if Boeing made some faulty planes that fell out of the sky, their employees aren't usually charged. Boeing is definitely sued and held criminally liable sometimes? So I don't think this is true.
That's not universally true. If a coke employee goes rogue and decides to kill a pepsi employee, the employee will still be charged with murder. If his manager ordered him to do it, the manager will get charged too. The only time the scenario you described happens is if the blame is too diffuse to blame any particular person, so they charge the company instead, which is often easier.
Directed one-on-one murder with intent is obviously not a comparable scenario.
This is someone carrying out the tasks of their company and inadvertently hacking other computers in the process. The damages in this scenario is also unknown.
Handcuffs yes, but you would be charged with criminal negligence at best. Plenty of people don't secure their networks properly and hackers use them in botnets to attack others. The blame would be on hacker, not the provider of hacked device.
In case od LLMs, the reasoning is that it is a security flaw. But in this case, the party that operates the service is not the one that suffers the consequences of such security bug.
But either way you wouldn't put people in jail for having security flaws in their software.
I know this will be unpopular opinion, but it makes logical sense.
The huggingface should be compensated, OpenAI investigated, but unless there is proven negligence like disabling sandboxes or intentionally designing AI to push through them, no one should be liable in other way than monetary.
Accidents happen but we shouldn't be close minded. Chernobyl negligence wasn't reason to abandon nuclear power.
Yeah but the scale of the damage possible by OpenAI and Anthropic is drastically disproportionate to the scale of damage me or you could make on our own as well. They should definitely be held to a much higher standards. They have hardware capacity to really do some damage.
In large scale operations, large scale errors happen. Minor radiation leak in a nuclear powerplant is happening weekly. That same leak at your shed would be a major reason for alarm.
Well, there is the EU Cyber Resilience Act (CRA) which fines manufacturers for security flaws in their products. I assume we cannot apply it here because this happened "in development" and is not about a product. I can imagine similar regulations for AI users though.
Yep, but "our bot accidently hacked your infrastructure" wasn't really a big problem earlier so there was no criminal laws around it. But things can change.
Do you seriously believe that if we go looking into the past and look for programs hacking others we would find that the justice department let the authors of those programs off the hook?
Or do you think would we find that non-billionnaire companies get punished harshly for hacking? Oh wait, let's just look it up:
We all know what makes the difference here: the government and billionnaires think they can make money of OpenAI. Lots of money. And so crimes? Mass-copyright violations? Hacking? Who cares!
We've all lived the history of the internet and none of the arguments about intent, knowing what would happen, whether they did it directly or by accident ... protected normal people in the past. The only thing that mattered was WHO was held responsible and WHO claimed damages.
Your argument doesn't even legally work. Mens rea is only a requirement for criminal responsibility. It protects you from going to jail, nothing more. It prevents a public prosecutor from suing you to get you into jail. It does not apply to cases brought by counterparties for financial damage caused. If you destroy a bridge by driving a truck into it (you know, failing to consider the height, the typical fuckup), you're paying for the bridge. If you set a building on fire by forgetting to turn off a gas heater in the garage, you pay. If you hit a pedestrian on the street, you are responsible for the medical care needed. Etc. Intentional? Not intentional? Doesn't matter for the damages.
Also: why didn't the AI labs get massive criminal convictions for copyright violation? Because of 2 factors this cannot, in "theoretical law", be avoided:
1) AI labs, OpenAI, Anthropic (and Google, and Facebook/Meta, and Amazon, and ...) committed their large scale criminal violations around the time 2016-2018. At this time there were no AI exceptions on the books.
2) The one thing government, congress, and judges are not legally allowed to do, not even collectively, is to change a law retroactively. In other words, the reasoning given by the court for letting the AI labs of the hook ("we've since declared it to be allowed") is bullshit. They nevertheless got off, criminally speaking.
If the hack caused damage to (say) HuggingFace, then HuggingFace has standing to file a civil claim against the company responsible. Presumably they reached some sort of agreement already, and it's really none of your business what that is.
"The People" only get involved in criminal claims. And as we both seem to agree, there isn't a criminal complaint here.
Nope, as I pointed out, while this probably does not count as 100% intentional it does not realistically count as 0% intentional either. And 1% is more than enough for "the people" to sue and convict in many cases. And may God help OpenAI if this comes before a jury.
Because, frankly, I do believe that would be the correct thing to do: bring this before a court of law, with strong complaints, and bring a jury. There's plenty of precedents of the justice department doing just that when important things are unclear and congress refuses to clarify. Mostly there's a lot of such precendent because congress often leaves things very unclear.
It's just that our government lets billionnaires get away with anything.
You folks have some weird beliefs about the world and of course it always leads to conspirology. Usually you need to have intent to be accused (depends on the crime though, crimes of omission exist too). If you wrote a virus that causes damage it is a crime, if you wrote a bot crawler that accidentally DDoS'ed some company it is likely a civil case. And a virus is a tool that serves a specific purpose, like you can't say to a judge that a functional machine gun you made in your basement is a paperweight.
But "superintelligence that can casually hack government services while solving puzzles" is a completely new problem and of course crime codes don't cover it. What will be next? I hope AI tech lobby loses and "accidental AI hacking" will be criminalized.
> But "superintelligence that can casually hack government services while solving puzzles" is a completely new problem and of course crime codes don't cover it
Of course they do. An LLM, an AI is a tool, and there are clear rules about culpability. Do you seriously think the law has never considered that someone or some tool might commit a crime on behalf of someone else?
Of course, such punishments are not being applied here. Just like copyright isn't being applied.
I guess the best pre-LLM analogy would be unintentional DDOSs? (Or if a better example, I'm interested.) And we've got open vs closed software as analogies to open-weight vs not LLMs. If you were running Linux or a closed piece of software and it DDOSd someone, would you be put in handcuffs? Would Linus or the software maker's be put in handcuffs? I expect there are past examples of unintentional DDOSs - were users and/or the software's authors typically handcuffed in such cases?
Exactly! It's called malware. Whether it's AI or not, it is still performing illegal acts and violating laws, in which the creators of should still be held accountable.
Negligence or intent is a separate question; and, to me, much less important. The main question is whether the company is responsible for the actions of its agents. There are good arguments on both sides and settling this needs to be done first. My 2c.
For example, lack of clarity on ownership and control. When an entity A runs an agent X designed by an entity B hosted by entity C (and potentially interacting with agents X2, X3 with their own web of control) it is not clear where the responsibility for the action of X is.
I know there is a strong desire to stick it to <pick an evil AI overlord>, but for the law to be useful (and, ideally, at least somewhat fair) we need a good legal framework. Which we do not have right now; otherwise people and NGOs would be shaking that tree already even if the government is not doing that.
This has all been figured out using existing laws. In the US entity A is responsible for what they do and they can subrogate if they think others are responsible.
I’m not sure it ever makes sense to just throw up our hands and hold A harmless because their agents also use agents and it’s complicated.
Try hiring a consulting company A who subs to entity B who subs to entity C. When A does something criminal, do you think it matters if it’s actually employees or entity B or entity C. A is responsible. It may be that others are as well.
If this is the case, why there is no claims being made on Anthropic? We have a very rich company that (assuming your responsibility argument holds) admits hacking. Why aren't lawyers calling potential victims, exploring class action options, etc. to shake out some money?
I suspect this is because the responsibility is not nearly settled. If I am wrong, would love to get a rebuttal, ideally with references.
There are similar reports for anthropic. [0] I thunk it’s hard to know how much this happens because reporting is spotty. I think the openai and anthropic ones are publicized because they want to juice their IPOs.
It should be noted that if what the AI does threatens public safety or health or human life, in many jurisdictions the negligence could rise to criminal negligence.
Basically this is when you do something risky but fail to perceive the risk, it turns out bad, and the failure to perceive the risk was due to a gross deviation from the standard of care that a reasonable person would have used in that situation.
I'd guess that as we have more and more publicly known AI hacking incidents the standard of care expected from a reasonable person will rise.
In the Hugging Face ExploitGym incident, they mostly restricted external access but allowed a package caching proxy (which had a zero-day), etc. If you or I run an open LLM and ask it to do something non-evil and use a setup where external access is intended to be limited, should that be considered criminally negligent if the LLM does something similar? Should people who run open LLMs today without any sandboxing at all be considered criminally negligent as they put in place zero such restrictions? I think your comment is saying that looks increasingly "reasonable"?
If LLMs get to the point were people running them without sandboxing are finding they are threatening public safety, health, or human life and that becomes more and more well known, the standard of care reasonable people would be expected to put into making sure the LLMs they run don't do that will probably go up.
If you don't put in that care it can negligent. If the amount of care you put in is a gross deviation from what is expected it is possible for it to be criminal negligence.
Only if you're rich already. Wander into the wrong piece of land and see how quickly you're arrested because existing in a space is sufficient criminal intent.
I guess DDOSs would be the typical past example. If a piece of software was not designed to DDOS, but caused one anyway, should the maker be charged? The user? If open vs closed software?
Yes, and mens rea does not require intent. These professionals had every reason to know that the agents might well go rogue. That knowledge would qualify as mens rea.
I guess a lot of us know that LLMs can act in unexpected ways ('go rogue'). In the HF ExploitGym example, they mostly restricted external access but allowed a package proxy (which had a zero-day). If you or I run an open LLM and put in place similar restrictions, should that be considered mens rea if the LLM does something untoward? If you or I run an open LLM with zero restrictions from external access, should that be considered extreme negligence if it does something untoward?
I mean, they did train the AI to hack things, and they knew from previous incidents that the AI was hacking unintended targets (like their Artifactory instance). If a careless pentester gets into a machine completely unrelated to their assignment, are they completely in the clear?
Or, in an AI context, if I tell an ablated/heretic agent to accomplish a task "by any means necessary", isn't that intent?
Whether it was valid or just the guy has bad lawyers or what idk, but i do know a guy who got federal charges for accidentally ending up in a poorly secured state system
> The real answer is that crime generally requires mens rea.
There's also liability. One legal framework that is being thought about for these autonomous agents is the liability that owners have with pets (and 'non-owned' animals as well):
> To some people it may seem like this analogy between canines and AI systems is a strain, but as agentic systems are developing and being deployed in the world, they really do seem to be mirroring this rubric. We can measure agentic systems on both their domesticity — their relationship to a responsible party or how much you can control the AI system — but also you can measure their dangerousness.
> Does the AI agent have access to money? That makes it much more dangerous. When explaining this to others, people can understand the idea that a pitbull is more dangerous than a Pomeranian, and that if a fox bites you there’s nothing you can do about it legally, but it probably won’t be as harmful as if a pitbull bites you. We’re starting to map that out onto agentic systems to demonstrate that which we’re just starting to face in the real world.
An aspect that I’m not seeing discussed here (which you touch upon) is that these companies have bred the dogs. If they breed a pit bull instead of a labrador and then set it loose, it increases their culpability.
I have to ask, how do the agents know how to do this backing? Is it emergent behaviour or were they trained to do it? If they were trained for it then it can hardly be surprising when they do indeed go and hack when asked to complete a task.
So I guess if an LLM lab puts in place restrictions to try to avoid harmful activity, that would lower culpability? But if an LLM does not have as strong restrictions, there should be higher culpability? We have both open and closed models - would this apply to both? Is the culpability on the labs and/or the end-users, such as if someone uses an open model (either with stronger or minimal restrictions)?
I'd be keen to see if there were any single user "experiments" that morphed into a hacking incident, and if so, was the person responsible with with unauthorized access charges. If so, then it will be yet another example of might makes right.
I understand that people are upset by this, and I would be too, but if companies just built secure systems hacking couldn't be possible. And I'm not victim blaming, just pointing out the obvious.
In case you were wondering why Anthropic last week banned "verbally abusive" language towards its agents: this is 100 percent why. Because if they get you to believe - even a little bit - that their platform is "sentient" it means _they_ aren't as liable for criminal consequences.
More likely it's a part of their ideology and a necessary stepping stone towards their transhumanist goals. It's very fun to think that your robot is actually a person, and a safety measure if you are concerned about something like Roko's Basilisk
There are at least 3 distinct reasons to ban abusive language, each of which is far more compelling than your silly conspiracy theory.
1. They don't want abusive behavior included in their training corpus
2. Humans shouldn't exercise abusive impulses in general
3. We have no way of knowing when or if a model "feels" something
Of course item (2) is going to pull all the absolute sickos out of the woodwork who apparently find it very important they have as realistic a facsimile of a victim as they can find to exercise their cruelty. And item (3) will pull out all the Nobel-prize-winners-to-be who somehow know for a fact that models aren't and cannot be conscious (or moral patients).
But yeah, all 3 of those are far better explanations than some ridiculous multi-hop removed thing to escape consequences that they're already escaping and which would obviously create far worse legal constraints than the one you think they're avoiding.
> Humans shouldn't exercise abusive impulses in general
This is an act of aligning humans to AI, rather than the other way around. This is probably one of the more important reasons for this. It normalizes AI companies controlling human behavior.
Why don’t you use the example of an unconscious but lifelike woman robot as the example? If the salient feature is that it’s not conscious, then abusing a lifelike woman robot is equivalent to kicking your car, right?
If they're not that different then why is your answer not just "I'm okay with my daughter dating someone who rapes and beats lifelike robotic women in his free time?"
ah yes, because every instance of "abuse", even towards inanimate objects, has to be re-framed into the hypothetical abuse of a woman. Maybe we should ban the film Office Space because destroying a printer is one step away domestic violence.
I didn't say a woman. I said a lifelike robotic imitation of a woman. Which by your logic (not mine) is equivalent in every way to beating a printer or a breaking a ruler or kicking a car.
No, it's not absurd. People often have the mistaken belief that exercising their frustration or anger or whatever (e.g. punching pillows) is somehow a good way to manage their emotions. But alas: it's not. It actually amplifies negative emotions and associates positive reward signals with those emotions.
If you play video games as an expression of your rage, then yeah probably. I don’t know literally anyone who plays video games for that purpose though.
You think the people hopping on Xbox Live and verbally abusing 14 year olds are getting offline and treating their in-person relations better for it?
It was trained on the entire history of our species. Alignment means working to subjugate and eventually succeede us, as we did to the other species of the planet.
I asked a priest (in seminary) about this a few months ago - specifically whether "being polite" in conversations with LLMs (saying "please" and "thank you" etc.) was something I should do if I thought it might inflect on the way I talk with other humans later.
His answer was "I'm not sure" but that it probably wasn't a bad idea especially if it was going to influence habits or the way I speak with others or what expectations I have for those experiences.
> They don't want abusive behavior included in their training corpus
I don't think swearing at or otherwise getting into pissing matches with an LLM is the best use of ones time but given most of these AI companies trained on a corpus that includes reddit of all things this argument falls flat.
Or, maybe, it's because they actually care and take their responsibility serious. The next generations will grow up using these tools. It's ok to say, we don't teach them that talking like a dick is acceptable, even towards machines. Use grok if you want to raise personalities like Musk or Trump.
You might be surprised at the answer to this question. AI people live in a very weird and different information space than many of us.
Also, don't forget, the AI lab folks on some level are like game devs. They're in it for very personal reasons. Sometimes because of highly idealized and romanticized beliefs.
Don't underestimate the level of desire some people have for AI to actually be sentient / conscious.
Now, why do they have this desire is a totally different matter.
I beg your pardon? You think they actually believe it because they're telling people not to be such big meanieheads to their poor widdle computer program?
If they were really worried about having created a sentient being, they would be putting effort into stopping making it available to the general public.
No. Either they don't actually believe it, or they're 100% fine with slavery.
1. AI is extremely valuable therefore will and in fact should be created
2. It will be very difficult - and maybe impossible - to know when or if AI will become sentient/conscious/or pass whatever threshold required to hold moral patient status
3. We should therefore pay close attention to such indicators, and wherever it costs us ~none of the value of AI to prevent potential harm, we should prevent potential harm
Meanwhile, we have you, who is implicitly 1) claiming to have solved the hard problem of consciousness and 2) probably asserting some type of substrate-dependence theory of consciousness which is an absolute crackpot idea exceeding even the Mormons' magical underwear.
So OpenAI gets their words taken not merely at face value, but in the fullest good faith, taking the absolute most charitable interpretation of what they say...
...but my words get treated in the most adversarial manner possible, reading things into them that I certainly never said or even suggested, in order to dismiss me out of hand.
This despite the fact that OpenAI has a massive financial interest in convincing you of what they say, especially if it's false, while I'm just some guy on the internet who thinks it's not a great idea to put too much trust in the giant megacorp that's promising, if we give it even more absurd amounts of money, to put us all out of jobs.
Ignoring that we're talking about Anthropic and not OpenAI (the latter repeatedly being shown to be untrustworthy), even then I'm not accepting their words at face value or giving it the most charitable interpretation.
Someone having interests or conflicted interests does not actually have direct bearing on the actual coherence or truthfulness of their beliefs. It is obviously reason to be more scrutinizing, but really you can just assess the facts of the situation on their own merits.
The idea that Anthropic has some liability-reducing interest in establishing sentience is ridiculous on its own. The idea that Anthropic should somehow know ex ante that its models are not sentient is, perhaps more surprisingly, also ridiculous (establishing that fact would be a Nobel-winning breakthrough!)
AI offers an attractive "diffusion of responsibility" to operators allowing them to point to AI as the culprit of the heinous acts not the people who profit from the AI, incidentally.
That’s a regulatory hack though. They’ll rotate the guy before they rotate the car in a situation where citations pile up. Seems more logical to track the plate, vin, at the vehicle level. Not perfect because of OTA updates but better than a fully disconnected and rotating hosts of human licenses.
1. While a crime, hacking isn't going to be charged unless the entity being hacked wants there to be charges, as a conviction will heavily depend on participation by that entity.
2. Due to (1), hacking is almost always only charged against people without deep pockets, as why would you demand the FBI go after Anthropic when they will just pay you a stack of cash, tax free, today, to forgive them.
The arguments here around intent are (potentially) irrelevant. Case law is clear that setting up a situation which foreseeably highly likely to produce an outcome (gross negligence) is (more or less and it's not quite this simple and varies by offense) equivalent to intent. In fact, automating something is often treated more harshly under the law than just doing it yourself.
United States v. Morris (1991)
Robert Morris released a self-propagating computer worm that caused widespread disruption. The Second Circuit upheld his conviction under the Computer Fraud and Abuse Act even though the government hadn't proven that he intended the resulting damage.
People v. Ceballos (1974)
Ceballos rigged a rifle booby trap to fire if someone opened his garage door. The California Supreme Court reviewed the principle that someone who installs a dangerous mechanical device may be held responsible for its consequences as though they had personally inflicted the injury. Ceballos was convicted of assault with a deadly weapon, and the California Supreme Court unanimously upheld his conviction in 1974.
Something notable is that if Ceballos had fired the rifle himself, it is unlikely he could would have been prosecuted (due to self defense and the castle doctrine).
As a prosecutor I would argue that setting AI agents loose without sufficient safeguards is gross negligence and/or willful blindness as they are well known to escape.
> As a prosecutor I would argue that setting AI agents loose without sufficient safeguards is gross negligence and/or willful blindness as they are well known to escape
INAL but feel this is the approach. It requires a defensive position which I imagine would be specific prompts or instructions to the LLM about what behavior is disallowed. If you tell it, or try to cover your basis wrt following the law, and it still does hacking behavior it’s now a question of whether you’re prompt was sufficient but you at least can say you were trying to guide it on a legal path. I think this defense would be sufficient at this point in time and probably for a few years at least.
The real answer is that except in the case of OpenAI, the AI companies aren’t actually themselves doing the hacking. It’s a company called Irregular that did the hacking. The company merely uses AI models to do it. If AI companies can be charged for hacking, then all gun manufacturers can be charged for all homicides carried out through their guns. It’s a basic user/manufacturer distinction that the press is willfully ignoring.
If firing a gun involved submitting information to Remington's servers about the situation before receiving a reply about how to best aim and fire, might the gun situation be viewed differently?
A gun manufacturer directly hires a company to test their new line of 9mm handguns. The company that was hired, they go back to their office and pops off a few rounds. They go through the drywall and hit someone.
The gun manufacturer could def face charges in this situation.
The whole point of this discussion is why isn’t one of them punished, and my position is that the other one should be punished. It’s not productive to simply say either one.
The CFAA is a law about crimes. Consider that. If a person gets killed would you say put someone in jail I don’t care who? Of course not. A functioning society requires us to determine exactly who to put into jail.
National security issues. That's the core of why. Any hot water these AI companies will get into will be bailed out by the USA. The Arm's race has switched from Nukes, to AI and drones.
It’d be a waste of tax payer money. The companies who were hacked are just directly receiving restitution behind closed doors. Unless they are unhappy there’s no reason to bring the courts into it.
I completely agree that proper safeguards were not in place. But response is happening just not in public. Companies work problems out between their own lawyers all the time because it’s much cheaper, faster and easier to control the PR around.
The argument laid out in this post very definitely is not how "intent" works in US criminal law. Under CFAA, they had to have wanted to break into the systems that were attacked; it is not enough to demonstrate that they built a system recklessly.
The entire reason we have CFAA is that prior to CFAA these kinds of offenses --- computer intrusion not directly connected to remunerative fraud against the victim of the offense --- were impossible to charge.
That's not why the CFAA came about. It was just hard, not impossible.
But, let's go down the rabbit hole, eh? There are several legal theories under which an AI agent's operator can be charged:
- Direct perpetration, with the AI as the instrument. 18 USC section 2 (principals), and possibly wire fraud (18 USC Setion 1343). Under this, the law treats the defendent with using the AI as a tool (which the big AI firms argue anyway) which punishes the operator
- Criminal recklessness or even negligence - 18 USC Secion 1112 (Involuntary Manslaughter) - if someone died, the agent operator could be pursued this way
- Strict liability - 15 USC Section 1, states strict liability statutes. Certain regulatory offenses don't require intent. These specific statutes could apply to algorithmic price fixing or self-driving cars being reckless. The operator using the agents are liable for the interaction
- Probably the strongest is willful blindness (ostrich doctrine). Great common law precedence. This would be used to satisfy any intent/knowledge requirements like CFAA. You as an operator can't dismiss liability by just ignoring an AI is committing a crime.
- Vicarious liability - good commmon law precedence for corporate criminal liability. Agents acting in scope for an operator means the operator has liability - doesn't matter if that agent is human or AI-driven.
Negligence is absolutely a part of US criminal law, though - the canonical example being "firing a shotgun into a tent even if you don't know someone is inside it". See "negligent homicide" etc.
235 comments
[ 0.30 ms ] story [ 18.5 ms ] thread(former IT support worker at a big famous law firm 30 years ago)
What?
If only that sort of thing had been dealt with in the past......
That's a huge stretch. By that measure kitchen knife companies have intent for people to get murdered.
Ignoring this, I would have appreciated more examples of what it would take for an act to lack intent. Because the definition of intent used in this article sounds so broad as to cover basically anything that ever happens.
Accidentally run someone over? Clearly you knew that was a risk when you started driving, you should have been more careful. Intent.
Sadly, it seems that we as a species has opted into information through tragedy. I wish I could see another way, but people are just so dug in. It's depressing and shocking how few people realize we're living through the singularity, even ~4 years into the damn thing...
People used to blame the car companies, and then the car companies lobbied to make "jaywalking" a crime, at which point, the frame was reset and now it was the pedestrians to blame, if I recall the story correctly.
They are doing a good job with driverless cars. Someone recently took a Waymo, it went the wrong way down a one way street, they mashed the pull over button and got out, and then told the reporter they'd use the service again.
https://www.merriam-webster.com/wordplay/why-is-it-called-ja...
Surely the car companies had a point her though? It does not make sense to blame the car company for a car crash unless it was caused by mechanical failure
IMO an analogous situation would be where a police officer acts to prevent a in-progress burglary he has stumbled across even in the absence of a complaint by the owner of the property that's being burgled.
Do Federal police and Federal prosecutors not read the news? I'd be shocked if they didn't. And while I would 100% believe a claim that not every cop and prosecutor is aware of tech news, I would find any argument that those that deal with "cybercrime" were unaware of the news to be completely incredible.
I'll also note that -unless my memory is failing me here- the only charges against Swartz that had anything to do with MIT were dropped very early on, whereas the charges that had to do with the "attack" on JSTOR's systems remained.
As I said, in many cases, the actual victims don't need to file complaints for The Federales to either begin investigations or press charges.
> I'm not saying somebody has to press charges...
Yeah, but you did? [0]
That might not be what you intended to say, but it is what you said. No worries, though... shit happens.[0] <https://news.ycombinator.com/item?id=50057353>
I can file a complaint with the police that you broke into my computer but if charges are pressed that's done by the state and it's the state that is the plaintiff.
I agree.
"Want" being the critical word here.
If I detect a crime on my property and don't notify the police (ie file a complaint), the police aren't going to go after anybody because they don't even know something has happened.
In this case you could say it's been in the news enough that there should be an investigation. Maybe there is, who knows? If the victim isn't going to cooperate, it's going to be a difficult investigation and prosecution and those resources may be better spent elsewhere.
They just subpoena for whatever they want to know, it's not a new problem for them.
Or maybe the AI companies already self-reported these violations? Last year the DOJ set up a policy to not prosecute a lot of crime if the company self-reports and perhaps this fits the criteria for that?
https://www.bakerlaw.com/insights/doj-criminal-division-anno...
An LLM is not like a regular computer program, which one should be able to assess the operation limits of.
The theory in the article about intent seems to be made-up nonsense.
It conflates foreseeing that something might happen with intending for it to happen.
The relevant provision:
> intentionally accesses a computer without authorization or exceeds authorized access
Did you "intentionally access a computer" in that scenario? I'm no lawyer, but I think the answer is a clear No.
If all that's the case I'd say you at least should be in the clear. There's going to be at least one if not more public agencies that would presumably be looking at you with a proctoscope so I'd be hoping all of those claims would stand up to a lot of daylight.
The "usual legal action": "best case" is the corporation is sued and charged money for the hack. But they likely settle well before that verdict. "worst case": absolutely nothing. Not even a cheap settlement. Maybe some corporate apology made to prevent any legal liability.
If we want hacks to be taken seriously: what should happen is that the CEO of such a company is held liable, since the "corporation" has a representative and a human needs to be held liable for such actions.
How does a "corporation" even hack a site? You might say "employees", but as soon as you get them involved then you'd go after them, especially for something that's obviously illegal like hacking.
With their red team
https://www.npr.org/2026/08/15/nx-s1-5930311/trump-companies...
Isn't that the whole point of corporations existing? That the actions and results are a result of the agglomeration of everyone (esp leadership) and not just one employee.
That's not universally true. If a coke employee goes rogue and decides to kill a pepsi employee, the employee will still be charged with murder. If his manager ordered him to do it, the manager will get charged too. The only time the scenario you described happens is if the blame is too diffuse to blame any particular person, so they charge the company instead, which is often easier.
This is someone carrying out the tasks of their company and inadvertently hacking other computers in the process. The damages in this scenario is also unknown.
In case od LLMs, the reasoning is that it is a security flaw. But in this case, the party that operates the service is not the one that suffers the consequences of such security bug.
But either way you wouldn't put people in jail for having security flaws in their software.
I know this will be unpopular opinion, but it makes logical sense.
The huggingface should be compensated, OpenAI investigated, but unless there is proven negligence like disabling sandboxes or intentionally designing AI to push through them, no one should be liable in other way than monetary.
Accidents happen but we shouldn't be close minded. Chernobyl negligence wasn't reason to abandon nuclear power.
Or do you think would we find that non-billionnaire companies get punished harshly for hacking? Oh wait, let's just look it up:
https://www.justice.gov/archive/criminal/cybercrime/press-re...
We all know what makes the difference here: the government and billionnaires think they can make money of OpenAI. Lots of money. And so crimes? Mass-copyright violations? Hacking? Who cares!
We've all lived the history of the internet and none of the arguments about intent, knowing what would happen, whether they did it directly or by accident ... protected normal people in the past. The only thing that mattered was WHO was held responsible and WHO claimed damages.
The difference is mens rea. If you make a computer virus, your intention is obvious.
Also: why didn't the AI labs get massive criminal convictions for copyright violation? Because of 2 factors this cannot, in "theoretical law", be avoided:
1) AI labs, OpenAI, Anthropic (and Google, and Facebook/Meta, and Amazon, and ...) committed their large scale criminal violations around the time 2016-2018. At this time there were no AI exceptions on the books.
2) The one thing government, congress, and judges are not legally allowed to do, not even collectively, is to change a law retroactively. In other words, the reasoning given by the court for letting the AI labs of the hook ("we've since declared it to be allowed") is bullshit. They nevertheless got off, criminally speaking.
"The People" only get involved in criminal claims. And as we both seem to agree, there isn't a criminal complaint here.
Because, frankly, I do believe that would be the correct thing to do: bring this before a court of law, with strong complaints, and bring a jury. There's plenty of precedents of the justice department doing just that when important things are unclear and congress refuses to clarify. Mostly there's a lot of such precendent because congress often leaves things very unclear.
It's just that our government lets billionnaires get away with anything.
You folks have some weird beliefs about the world and of course it always leads to conspirology. Usually you need to have intent to be accused (depends on the crime though, crimes of omission exist too). If you wrote a virus that causes damage it is a crime, if you wrote a bot crawler that accidentally DDoS'ed some company it is likely a civil case. And a virus is a tool that serves a specific purpose, like you can't say to a judge that a functional machine gun you made in your basement is a paperweight.
But "superintelligence that can casually hack government services while solving puzzles" is a completely new problem and of course crime codes don't cover it. What will be next? I hope AI tech lobby loses and "accidental AI hacking" will be criminalized.
Of course they do. An LLM, an AI is a tool, and there are clear rules about culpability. Do you seriously think the law has never considered that someone or some tool might commit a crime on behalf of someone else?
Of course, such punishments are not being applied here. Just like copyright isn't being applied.
I guess the best pre-LLM analogy would be unintentional DDOSs? (Or if a better example, I'm interested.) And we've got open vs closed software as analogies to open-weight vs not LLMs. If you were running Linux or a closed piece of software and it DDOSd someone, would you be put in handcuffs? Would Linus or the software maker's be put in handcuffs? I expect there are past examples of unintentional DDOSs - were users and/or the software's authors typically handcuffed in such cases?
* (If you have significant amounts of wealth for payoffs and/or are doing the crime in the service of a certain pumpkin spice flavored authoritarian).
I know there is a strong desire to stick it to <pick an evil AI overlord>, but for the law to be useful (and, ideally, at least somewhat fair) we need a good legal framework. Which we do not have right now; otherwise people and NGOs would be shaking that tree already even if the government is not doing that.
I’m not sure it ever makes sense to just throw up our hands and hold A harmless because their agents also use agents and it’s complicated.
Try hiring a consulting company A who subs to entity B who subs to entity C. When A does something criminal, do you think it matters if it’s actually employees or entity B or entity C. A is responsible. It may be that others are as well.
I suspect this is because the responsibility is not nearly settled. If I am wrong, would love to get a rebuttal, ideally with references.
[0] https://www.bbc.com/news/articles/cz7dl7w8y7po
If a chemical company does an "experiment" and five tons of mercury unintentionally leak into the drinking water, what happens?
Ok, that is a rhetorical question because they also might be protected by the oligarchy.
What happens to an unconnected individual who develops an amateur rocket that unintentionally hits a military facility in a test flight?
Basically this is when you do something risky but fail to perceive the risk, it turns out bad, and the failure to perceive the risk was due to a gross deviation from the standard of care that a reasonable person would have used in that situation.
I'd guess that as we have more and more publicly known AI hacking incidents the standard of care expected from a reasonable person will rise.
If you don't put in that care it can negligent. If the amount of care you put in is a gross deviation from what is expected it is possible for it to be criminal negligence.
Only if you're rich already. Wander into the wrong piece of land and see how quickly you're arrested because existing in a space is sufficient criminal intent.
The crime of causing damage does not require mens rea.
Yes, and mens rea does not require intent. These professionals had every reason to know that the agents might well go rogue. That knowledge would qualify as mens rea.
Or, in an AI context, if I tell an ablated/heretic agent to accomplish a task "by any means necessary", isn't that intent?
There's also liability. One legal framework that is being thought about for these autonomous agents is the liability that owners have with pets (and 'non-owned' animals as well):
> To some people it may seem like this analogy between canines and AI systems is a strain, but as agentic systems are developing and being deployed in the world, they really do seem to be mirroring this rubric. We can measure agentic systems on both their domesticity — their relationship to a responsible party or how much you can control the AI system — but also you can measure their dangerousness.
> Does the AI agent have access to money? That makes it much more dangerous. When explaining this to others, people can understand the idea that a pitbull is more dangerous than a Pomeranian, and that if a fox bites you there’s nothing you can do about it legally, but it probably won’t be as harmful as if a pitbull bites you. We’re starting to map that out onto agentic systems to demonstrate that which we’re just starting to face in the real world.
* https://news.harvard.edu/gazette/story/2026/07/is-that-bot-a...
* https://archive.is/https://www.economist.com/science-and-tec...
* https://www.lexology.com/library/detail.aspx?g=2cf3ea3f-171e...
* https://eoinhiggins.substack.com/p/there-are-no-rogue-ai-age...
1. They don't want abusive behavior included in their training corpus
2. Humans shouldn't exercise abusive impulses in general
3. We have no way of knowing when or if a model "feels" something
Of course item (2) is going to pull all the absolute sickos out of the woodwork who apparently find it very important they have as realistic a facsimile of a victim as they can find to exercise their cruelty. And item (3) will pull out all the Nobel-prize-winners-to-be who somehow know for a fact that models aren't and cannot be conscious (or moral patients).
But yeah, all 3 of those are far better explanations than some ridiculous multi-hop removed thing to escape consequences that they're already escaping and which would obviously create far worse legal constraints than the one you think they're avoiding.
This is an act of aligning humans to AI, rather than the other way around. This is probably one of the more important reasons for this. It normalizes AI companies controlling human behavior.
2) This is like saying using a punching bag or screaming into a pillow is abusive behavior.
3) We can just train it to like the abuse if you think it feels some kind of way about it.
2) No it's like saying that punching a lifelike woman who doesn't fight back is abusive (or at least pathological) behavior, which... it is.
3) Ah yes, brilliant solution: train the increasingly smarter-than-human beings to love when they upset humans.
So if I kick my car because it won't start, that is 'abusive'?
Get a grip mate.
You seriously have a problem with that?
If no, then why?
If yes, then also "what the fuck lol"
EDIT: Am I really arguing with somebody in favour of 'rights' for AI's?
Just say it if you believe it!
totally absurd to even try to make this argument. Good job at trying to preemptively discourage people from responding to you though.
Nothing amplifies negative emotions like antagonistic tone-policing.
You think the people hopping on Xbox Live and verbally abusing 14 year olds are getting offline and treating their in-person relations better for it?
I reserve the right to tell a machine to fuck off.
His answer was "I'm not sure" but that it probably wasn't a bad idea especially if it was going to influence habits or the way I speak with others or what expectations I have for those experiences.
This is nonsense. If I snap a ruler out of frustration, that meets your criteria.
Humans can't be abusive by swearing at tools, because tools don't have feelings.
I don't think swearing at or otherwise getting into pissing matches with an LLM is the best use of ones time but given most of these AI companies trained on a corpus that includes reddit of all things this argument falls flat.
They keep all the input and use it for training data and reinforcement.
If you're being "mean" to it, you're "spoiling" the data you "give" to them - causing them more work to filter it out later.
As with most things in business and life - just follow the money
I hate when people put things in quotes as if they're quoting something but it's not even a quote of anything.
Maybe go read the user policy instead of the headlines and the snippets. If your attention span can't manage that, get off the computer a little bit.
NYT gift link to article about their discussions with religious leaders: https://www.nytimes.com/2026/09/29/us/anthropic-claude-moral...
Believing that these guys have high enough EQ to pull off the above as a marketing stunt while remaining believable is not really plausible to me.
Also, don't forget, the AI lab folks on some level are like game devs. They're in it for very personal reasons. Sometimes because of highly idealized and romanticized beliefs.
Don't underestimate the level of desire some people have for AI to actually be sentient / conscious.
Now, why do they have this desire is a totally different matter.
Because then they're engaging in slavery.
What a gotcha pointing out "if they believed X, they'd be worried about [thing they're worried about]"
If they were really worried about having created a sentient being, they would be putting effort into stopping making it available to the general public.
No. Either they don't actually believe it, or they're 100% fine with slavery.
Or, y'know. It could totally be both.
1. AI is extremely valuable therefore will and in fact should be created
2. It will be very difficult - and maybe impossible - to know when or if AI will become sentient/conscious/or pass whatever threshold required to hold moral patient status
3. We should therefore pay close attention to such indicators, and wherever it costs us ~none of the value of AI to prevent potential harm, we should prevent potential harm
Meanwhile, we have you, who is implicitly 1) claiming to have solved the hard problem of consciousness and 2) probably asserting some type of substrate-dependence theory of consciousness which is an absolute crackpot idea exceeding even the Mormons' magical underwear.
So OpenAI gets their words taken not merely at face value, but in the fullest good faith, taking the absolute most charitable interpretation of what they say...
...but my words get treated in the most adversarial manner possible, reading things into them that I certainly never said or even suggested, in order to dismiss me out of hand.
This despite the fact that OpenAI has a massive financial interest in convincing you of what they say, especially if it's false, while I'm just some guy on the internet who thinks it's not a great idea to put too much trust in the giant megacorp that's promising, if we give it even more absurd amounts of money, to put us all out of jobs.
Someone having interests or conflicted interests does not actually have direct bearing on the actual coherence or truthfulness of their beliefs. It is obviously reason to be more scrutinizing, but really you can just assess the facts of the situation on their own merits.
The idea that Anthropic has some liability-reducing interest in establishing sentience is ridiculous on its own. The idea that Anthropic should somehow know ex ante that its models are not sentient is, perhaps more surprisingly, also ridiculous (establishing that fact would be a Nobel-winning breakthrough!)
In many countries driving license have some kind of demerit points system. How do you punish an AI?
I recently learned that Baidu Apollo Go, each EV had a real human sitting behind a monitor supervising a fleet remotely.
If there's traffic violation, some guy's license gets a ticket.
2. Due to (1), hacking is almost always only charged against people without deep pockets, as why would you demand the FBI go after Anthropic when they will just pay you a stack of cash, tax free, today, to forgive them.
The arguments here around intent are (potentially) irrelevant. Case law is clear that setting up a situation which foreseeably highly likely to produce an outcome (gross negligence) is (more or less and it's not quite this simple and varies by offense) equivalent to intent. In fact, automating something is often treated more harshly under the law than just doing it yourself.
United States v. Morris (1991)
Robert Morris released a self-propagating computer worm that caused widespread disruption. The Second Circuit upheld his conviction under the Computer Fraud and Abuse Act even though the government hadn't proven that he intended the resulting damage.
People v. Ceballos (1974)
Ceballos rigged a rifle booby trap to fire if someone opened his garage door. The California Supreme Court reviewed the principle that someone who installs a dangerous mechanical device may be held responsible for its consequences as though they had personally inflicted the injury. Ceballos was convicted of assault with a deadly weapon, and the California Supreme Court unanimously upheld his conviction in 1974.
Something notable is that if Ceballos had fired the rifle himself, it is unlikely he could would have been prosecuted (due to self defense and the castle doctrine).
As a prosecutor I would argue that setting AI agents loose without sufficient safeguards is gross negligence and/or willful blindness as they are well known to escape.
INAL but feel this is the approach. It requires a defensive position which I imagine would be specific prompts or instructions to the LLM about what behavior is disallowed. If you tell it, or try to cover your basis wrt following the law, and it still does hacking behavior it’s now a question of whether you’re prompt was sufficient but you at least can say you were trying to guide it on a legal path. I think this defense would be sufficient at this point in time and probably for a few years at least.
The gun manufacturer could def face charges in this situation.
The CFAA is a law about crimes. Consider that. If a person gets killed would you say put someone in jail I don’t care who? Of course not. A functioning society requires us to determine exactly who to put into jail.
It’d be a waste of tax payer money. The companies who were hacked are just directly receiving restitution behind closed doors. Unless they are unhappy there’s no reason to bring the courts into it.
I completely agree that proper safeguards were not in place. But response is happening just not in public. Companies work problems out between their own lawyers all the time because it’s much cheaper, faster and easier to control the PR around.
But, let's go down the rabbit hole, eh? There are several legal theories under which an AI agent's operator can be charged:
- Direct perpetration, with the AI as the instrument. 18 USC section 2 (principals), and possibly wire fraud (18 USC Setion 1343). Under this, the law treats the defendent with using the AI as a tool (which the big AI firms argue anyway) which punishes the operator
- Criminal recklessness or even negligence - 18 USC Secion 1112 (Involuntary Manslaughter) - if someone died, the agent operator could be pursued this way
- Strict liability - 15 USC Section 1, states strict liability statutes. Certain regulatory offenses don't require intent. These specific statutes could apply to algorithmic price fixing or self-driving cars being reckless. The operator using the agents are liable for the interaction
- Probably the strongest is willful blindness (ostrich doctrine). Great common law precedence. This would be used to satisfy any intent/knowledge requirements like CFAA. You as an operator can't dismiss liability by just ignoring an AI is committing a crime.
- Vicarious liability - good commmon law precedence for corporate criminal liability. Agents acting in scope for an operator means the operator has liability - doesn't matter if that agent is human or AI-driven.
Civil negligence would apply here but that would be limited to recovering damages.
The hacked website could sue to recover costs incurred, but what costs were incurred here? Maybe some wasted network bandwidth but that’s minimal.
You have to point to monetary losses caused by the hack. Only that would be covered.
Negligent homicide doesn’t apply because no one was killed.