What the law does:
SB 53 establishes new requirements for frontier AI developers creating stronger:
Transparency: Requires large frontier developers to publicly publish a framework on its website describing how the company has incorporated national standards, international standards, and industry-consensus best practices into its frontier AI framework.
Innovation: Establishes a new consortium within the Government Operations Agency to develop a framework for creating a public computing cluster. The consortium, called CalCompute, will advance the development and deployment of artificial intelligence that is safe, ethical, equitable, and sustainable by fostering research and innovation.
Safety: Creates a new mechanism for frontier AI companies and the public to report potential critical safety incidents to California’s Office of Emergency Services.
Accountability: Protects whistleblowers who disclose significant health and safety risks posed by frontier models, and creates a civil penalty for noncompliance, enforceable by the Attorney General’s office.
Responsiveness: Directs the California Department of Technology to annually recommend appropriate updates to the law based on multistakeholder input, technological developments, and international standards.
I found this website with the actual bill text along with annotations [0]. The section 22757.12. seems to contain the actual details of what they mean by "transparency".
What real-world problem does any of this solve? For instance, how does it protect my IP from being vacuumed up and used by LLMs without permission from or payment to me?
This isn’t necessarily about fixing today’s real-world problems. Transparency is an enabler for identifying (or avoiding) a class of future real-world problems.
Look at what the bill actually requires. Companies have to publish frameworks showing how they "mitigate catastrophic risk" and implement "safety protocols" for "dangerous capabilities." That sounds reasonable until you realize the government is now defining what counts as dangerous and requiring private companies to build systems that restrict those outputs.
The Supreme Court already settled this. Brandenburg gives us the standard: imminent lawless action. Add in the narrow exceptions like child porn and true threats, and that's it. The government doesn't get to create new categories of "dangerous speech" just because the technology is new.
But here we have California mandating that AI companies assess whether their models can "provide expert-level assistance" in creating weapons or "engage in conduct that would constitute a crime." Then they have to implement mitigations and report to the state AG. That's prior restraint. The state is compelling companies to filter outputs based on potential future harm, which is exactly what the First Amendment prohibits.
Yes, bioweapons and cyberattacks are scary. But the solution isn't giving the government power to define "safety" and force companies to censor accordingly. If someone actually uses AI to commit a crime, prosecute them under existing law. You don't need a new regulatory framework that treats information itself as the threat.
This creates the infrastructure. Today it's "catastrophic risks." Tomorrow it's misinformation, hate speech, or whatever else the state decides needs "safety mitigations." Once you accept the premise that government can mandate content restrictions for safety, you've lost the argument.
The commenter’s profile indicates they work for a major AI development companies — where being against AI regulation aligns nicely with one’s paycheck. See also the the scare quotes around AI safety.
We all have heard the dogma: regulation kills innovation. As if unbridled innovation is all that people and society care about.
I wonder if the commenter above has ever worked in an industry where a safety culture matters. Once you have, you see the world a little bit differently.
Large chunks of Silicon Valley have little idea about safety: not in automotive, not in medicine, not for people, and certainly not for longer-term risks.
So forgive us for not trusting AI labs to have good incentives.
When regulation isn’t as efficient as we’d like, that is a problem. But you have to factor what happens if we don’t have any regulation. Also, don’t forget to call out every instance of insider corruption, kickback deals, any industry collusion.
Big4 audit all have a leech class of boomer audit partners who won't let the advisory arm separate and want money. This is a great new income stream. Figure deloitte in particular will make out like bandits on this.
> (a) “Artificial intelligence model” means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
I was curious how California was going to define AI since it's basically a marketing term as of now. Seems like it's defined as a non-biological system that generates outputs from inputs.
This is something that could be (and should be) pre-empted with federal law.
That said, this law seems pretty sloppy with its definitions. In particular, the definition of “Artificial intelligence model” includes all machines and every algorithm ever written.
> “Artificial intelligence model” means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
It's like they saw the twilight zone and decided they needed to cover androids just in case someone figured out how to make a robot with a cuckoo clock.
So if one causes $1B in damages one has to pay a fine of $10M? Similarly for other "catastrophic" damages? WTF. I am very AI pilled but this is no regulation at all. Suppose OpenAI pays their engineers $1M a year. In what world do they have any incentive to work to avoid a $10k fine? Let alone a $1M fine for "catastrophic" damage?
> (3) For a knowing violation that creates a material risk of death, serious physical injury, or a catastrophic risk, a large developer shall be subject to a civil penalty in an amount not to exceed one million dollars ($1,000,000) for a violation that is the large developers first such violation and in an amount not exceeding ten million dollars ($10,000,000) for any subsequent violation.
So, if violation is "unknowning" (I assume this means unintentional) and creates a material risk, then there is no penalty?
Also, penalties listed are upper bounds only (penalty will not exceed $X), and not lower bounds. $0 fine fulfills "not exceeding $10m" rule.
I scrolled this entire thread and still can’t figure out what effect this might have on the ai industry. Everyone’s takes feel excessively either politically motivated knee jerk or nihilistic.
It looks like it largely codifies things that most of the large AI companies are already doing. It encourages them to keep doing those things, and when other AI companies get large they will have to do them too. It adds some reporting requirements if you AI does something that kills too many people or causes too much monetary damage.
The link to the annotated bill text that dang added near the top of the page [1] is pretty useful, especially if you click the blue underlined parts which expand to annotations explaining how some of the things in it came about.
I wonder if whistleblowing applies to copyright claims. For instance, using data sets which involve copying proprietary works scraped from public sources. If so, California might be a dangerous place for some AI companies to operate in.
Reading the comments, this is both a "nothing burger" and a reason to geoblock California.
It does too much, and too little. The costs are too high, and too low. It's an example of corruption, but also a good first start at regulations. It will drive AI companies out of California, but also attract more companies with a lower bar for edge innovation.
Can anyone link to an actual informed and nuanced discussion of this bill? Because if it exists here, I haven't found it.
I'm so sick of these laws that just require people to "formulate plans" and "adopt policies" and so on. They should reverse the entire concept and ban all of it, then gradually start allowing little bits and pieces.
32 comments
[ 0.21 ms ] story [ 61.5 ms ] threadWhat the law does: SB 53 establishes new requirements for frontier AI developers creating stronger:
Transparency: Requires large frontier developers to publicly publish a framework on its website describing how the company has incorporated national standards, international standards, and industry-consensus best practices into its frontier AI framework.
Innovation: Establishes a new consortium within the Government Operations Agency to develop a framework for creating a public computing cluster. The consortium, called CalCompute, will advance the development and deployment of artificial intelligence that is safe, ethical, equitable, and sustainable by fostering research and innovation.
Safety: Creates a new mechanism for frontier AI companies and the public to report potential critical safety incidents to California’s Office of Emergency Services.
Accountability: Protects whistleblowers who disclose significant health and safety risks posed by frontier models, and creates a civil penalty for noncompliance, enforceable by the Attorney General’s office.
Responsiveness: Directs the California Department of Technology to annually recommend appropriate updates to the law based on multistakeholder input, technological developments, and international standards.
[0] https://sb53.info/
Look at what the bill actually requires. Companies have to publish frameworks showing how they "mitigate catastrophic risk" and implement "safety protocols" for "dangerous capabilities." That sounds reasonable until you realize the government is now defining what counts as dangerous and requiring private companies to build systems that restrict those outputs.
The Supreme Court already settled this. Brandenburg gives us the standard: imminent lawless action. Add in the narrow exceptions like child porn and true threats, and that's it. The government doesn't get to create new categories of "dangerous speech" just because the technology is new.
But here we have California mandating that AI companies assess whether their models can "provide expert-level assistance" in creating weapons or "engage in conduct that would constitute a crime." Then they have to implement mitigations and report to the state AG. That's prior restraint. The state is compelling companies to filter outputs based on potential future harm, which is exactly what the First Amendment prohibits.
Yes, bioweapons and cyberattacks are scary. But the solution isn't giving the government power to define "safety" and force companies to censor accordingly. If someone actually uses AI to commit a crime, prosecute them under existing law. You don't need a new regulatory framework that treats information itself as the threat.
This creates the infrastructure. Today it's "catastrophic risks." Tomorrow it's misinformation, hate speech, or whatever else the state decides needs "safety mitigations." Once you accept the premise that government can mandate content restrictions for safety, you've lost the argument.
We all have heard the dogma: regulation kills innovation. As if unbridled innovation is all that people and society care about.
I wonder if the commenter above has ever worked in an industry where a safety culture matters. Once you have, you see the world a little bit differently.
Large chunks of Silicon Valley have little idea about safety: not in automotive, not in medicine, not for people, and certainly not for longer-term risks.
So forgive us for not trusting AI labs to have good incentives.
When regulation isn’t as efficient as we’d like, that is a problem. But you have to factor what happens if we don’t have any regulation. Also, don’t forget to call out every instance of insider corruption, kickback deals, any industry collusion.
> (a) “Artificial intelligence model” means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
I was curious how California was going to define AI since it's basically a marketing term as of now. Seems like it's defined as a non-biological system that generates outputs from inputs.
That said, this law seems pretty sloppy with its definitions. In particular, the definition of “Artificial intelligence model” includes all machines and every algorithm ever written.
> “Artificial intelligence model” means an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.
It's like they saw the twilight zone and decided they needed to cover androids just in case someone figured out how to make a robot with a cuckoo clock.
So, if violation is "unknowning" (I assume this means unintentional) and creates a material risk, then there is no penalty?
Also, penalties listed are upper bounds only (penalty will not exceed $X), and not lower bounds. $0 fine fulfills "not exceeding $10m" rule.
The link to the annotated bill text that dang added near the top of the page [1] is pretty useful, especially if you click the blue underlined parts which expand to annotations explaining how some of the things in it came about.
That link also has a good summary and FAQs.
[1] https://sb53.info/index
What a thumbass idea.
It does too much, and too little. The costs are too high, and too low. It's an example of corruption, but also a good first start at regulations. It will drive AI companies out of California, but also attract more companies with a lower bar for edge innovation.
Can anyone link to an actual informed and nuanced discussion of this bill? Because if it exists here, I haven't found it.