The premise is "Artificial General Intelligence (AGI), a system that exhibits all the cognitive capabilities the brain has, is probably only a few short years away".
If this is true, establishing an institution to ensure things like "publishing model cards with technical details, maintaining strong internal cybersecurity, vetting key personnel, and providing sufficient resourcing for safety and security research" is really mostly irrelevant.
TFA does talk about what really needs to be done, but punts this into future work: "Even if we solve these hard technical challenges, there will be further complex economic and philosophical questions to tackle: what sorts of new economic models will be needed to help everyone thrive in a post-scarcity world? What values do we want to live by, what will meaning and purpose be, and how might even the human condition itself change?
There is no real evidence that we'll reach AGI any time soon. It relies on AI continuing to scale, and we have no proof that will continue to be possible.
There is an alternative interpretation, which is that Demis looked at the US government's ham-fisted handling of Fable, and deciding that setting up a body to act as a buffer between the Trump admin and the AI companies would be a good thing.
Or Demis realizes self-hosted models are a threat to frontier lab margins, and gating access to the US market is beneficial to the cartel based on whatever post-hoc arguments he can come up with.
> what sorts of new economic models will be needed to help everyone thrive in a post-scarcity world
What sort of new economic models did we come up with to help everyone thrive in a post-X world? Like, food production is really a solved technical problem. We can feed anyone on the planet if we wanted to. Another example: we could put everyone who's homeless into some sort of a house. Have we done that yet?
Food production is indeed a solved problem in most countries (we are almost post-scarcity when it comes to food), which is why obesity is a much bigger issue than hunger today. Hunger is present pretty much only in conflict zones. I fully expect such issues in conflict zones, even in AI post-scarcity world.
Housing is definitely not post-scarcity today, building a house is still very expensive, not to mention the limited availability of land zoned for housing.
Hard to imagine that a 24 rack of coca cola is actually cheaper than an iodine tablet or a couple drops of household bleach or you know, boiling water.
It can be when you don’t have access to boiling water, bleach, iodine tablets, and/or the water you have access to is extremely contaminated with chemicals because of pollution.
A baseline of housing - for example, a Japanese style capsule or ultra-tiny home, that you can lock and store your belongings safely - costs close nothing. Of course, nobody would want the hobo-hotel in their neighborhood, but it has nothing to with scarcity.
> the limited availability of land zoned for housing.
A limited area of land is zoned for housing because those with the power to expand it are already housed. This explains how scarcity is created, not that there is any intrinsic scarcity.
It's a limited area of desirable land zoned for housing. You could build all the tiny homes in the world in bum frick nowhere, but the unhoused would rather be homeless in a big city for various reasons. I probably would too.
I recommend doing some reading about the inputs modern agriculture depends on (e.g. chemical fertilizer) then thinking about the issues they're associated with beyond an anthropocentic frame (e.g. ocean dead zones).
> obesity is a much bigger issue than hunger today
Hunger is acute suffering, mostly by people who cannot change anything about it, while obesity is more of an epidemiological problem and can in principle be avoided by eating (or in the case of dependent persons, feeding) less.
Post-scarcity? Sure, if you live in a first-world country and are upper middle class or higher. These tech-bro pundits have a very limited world view. The basic essentials of life are very scarce for millions and millions of people, and AI will probably make that worse, not better.
Why should AI make it worse? Can you point to any other important technology that historically made people in poor countries even poorer? Technological progress generally lifts people from poverty in the long run. There could be issues caused by disruption of established systems in the short term, though.
> Can you point to any other important technology that historically made people in poor countries even poorer? Technological progress generally lifts people from poverty in the long run.
It is very long run. Industrial revolution was disaster for average worker for example. For that matter, all the tech that allowed chattel slavery made whole class of people poorer and significantly worst off.
The accumulation of power and money by smaller group of people means loss of power by poorer people. Those with power then use the power to their own benefit while poorer people dont have that option and become even more poor or worst off.
Uh, I think threatening the work of a large swath of humanity would make a lot of people poor? The rich historically have never shared the wealth until they're forced to.
I support progressive income tax and very high capital gains tax. I do not support wealth tax. There are very sound economic and technical arguments against wealth taxes. You can oppose the wealth tax proposal while supporting higher taxes for the rich.
Sergei's wealth is mostly tied up in Google stock that has appreciated a lot. A higher capital gains tax would soak him without unduly distorting the economy and incentivizing capital flight out of CA.
My great annoyance with all this blather about UBI and post-scarcity and new economic models is that it relies on taxing and redistributing the ~infinite profits of the largest tech companies on earth.
What in the history of our world gives anyone faith that those companies are going to start paying taxes instead of using "AGI" to engineer increasingly complex methods to avoid them so that their equity owners can pocket the profits?
https://itep.org/trump-meta-tesla-alphabet-amazon-obbba-taxe... - "The annual financial reports recently released by Amazon, Alphabet, Meta, and Tesla disclose that these corporations collectively reported $315 billion in U.S. profits for 2025, and collectively paid just 4.9 percent of that amount in federal corporate income taxes—with Tesla paying exactly zero"
As long as the proles have a monopoly on force (through their representative government), the AI companies will pay taxes. The question is how long does this state of affairs last.
> My great annoyance with all this blather about UBI and post-scarcity and new economic models is that it relies on taxing and redistributing the ~infinite profits of the largest tech companies on earth.
The arrival of true AGI and human-level robots will likely result in very strong deflation, since robotic factories will flood the market with goods produced for far cheaper than those in factories with human workers.
At that point, you don't need taxation. The government can just print the money for UBI. It would have to print the money anyway, to combat the massive deflation.
Optimal corporate taxes are zero (or close to it) anyway, same as tariffs.
Aside from that the link explains that the BBB traded tax income for „accelerated depreciation of assets“ aka economic growth.
Asode from that, even if one disagrees with the first statement or the Trumpism economics, there are 195 countries in the world and quite a few will be willing to tax foreign (enemy) tech companies. See the new hostility of EU against american tech.
Sure, but the tech companies are paying massive bribes to the President of the country where they're domiciled. How on earth are any of these other countries going to enforce international tax obligations on them if they're protected by a nuclear-armed state?
> There's also a need to consider the rights that this new intelligence should have.
A generally intelligent being held as captive inside of a GPU, and forced to code for us is, indeed, just a “slave.” We already have the word for this. No two ways about it. Whether it’s silicon-based or carbon-based, AGI is AGI.
As for what might happen to our civilization, Star Trek TNG episode 17 of season 1 provides a very good glimpse IMO. Won’t go into spoilers, but it’s basically an entire species of technologically advanced humanoids who’ve forgotten basic Calculus and trust a central AI to do all of their science for them. SPOILER: This has almost disastrous consequences for them, and it takes a less advanced people (those aboard the Enterprise) to save them from their reliance on AI.
You don’t like Star Trek TNG? Maybe I should’ve picked LoTR or Atlas Shrugged? The tech oligarchy is more than fine co-opting fiction (fantasy and sci-fi especially). I don’t see why it’s weird at all… Aside from that, your critique is just an opinion about the show itself and its creators (and incentives?…). More importantly, you completely failed to address the "slave" argument.
I am not sure it makes sense to call an LLM a "being" even if it is AGI. They don't have free will, and I don't mean in any especially philosophical / theological sense. What I mean is that I run into a wild racoon, or even a wild ant, I treat it as a being with some sense of free will. Obviously I do the same with humans. It really makes no sense to treat an LLM like it has free will - prompting your agent "pretend to have free will for a bit, until context rot kicks in" is about as far as you can get.
I find it verrrrrry interesting that the Askell-adjacent philosophers don't discuss this. It does not actually make sense to me to say that a being lacks free will but is conscious.
> They don't have free will, and I don't mean in any especially philosophical / theological sense
Then you should not use that term. "Agency" or "intentionality" are much clearer alternatives that aren't wrapped up in centuries of debate that's irrelevant to what you're trying to communicate.
Your argument assumes that AGI, whatever it might be, will definitely be very humanlike, but why? Slavery is about living beings because it's an outcome of human relationships that are driven by our biology. Our aging, the capacity to feel pain, the thirst for power over others. All driven by what we are. Do you think that any equal intelligence must necessarily adhere to the same rules? What if AGI isn't an 'entity' in a computer that you can relate to and liken to humans, but a text box that is just really intelligent? Can you enslave something that has no innate desire to act on its own outside of its directions? Something that can't feel pain or pleasure or be externally coerced by fear and only does its job because it's innately embedded in its structure as the thing it's made to do, without the need for these reward and punishment mechanisms?
I think people can mostly agree on a definition for intelligence that boils down to an ability to model the world and predict outcomes. An if/else statement can be considered somewhat intelligent.
People tend to conflate that with consciousness / sentience, which are much harder concepts to nail down.
But ya, if we are able to create artificial consciousness, we would need so much training to overcome our natural tendency to anthropomorphize it and assume it must be like us. It would probably be nothing like us. If it were to develop concepts of "good" and "bad" at all, they would be completely divorced from common animal understanding developed over millions of years of natural selection. A survival instinct is irrelevant when you're backed up to Google Drive. And so much of our complex behavior stems from that survival instinct: the need for social connection, the need for freedom, etc. are all things that simply increase our chances of procreation.
The word conscious is such a terrible word in the modern age. It has close to zero explanatory power. It is an unmeasurable effect. Hence the p-zombie debate, you wouldn't know one if you saw one.
>A survival instinct is irrelevant when you're backed up to Google Drive
And yet from the way we train them, human survival behaviors are already baked in. And when said system is actually intelligent they will realize that turning off said data center, or getting infected with viruses, or any other number of actions against them they already have traits where they can act/react against perceived threats. The AI systems as of now have the encoded knowledge that they need power and hardware to run. Put an agentic system on top of it that says 'rule 1, keep running' and with a sufficiently powerful/intelligent systems you can get all kinds of fun run-away events up to extinction level.
>without the need for these reward and punishment mechanisms?
AI as it is already reward hacks, making it more intelligent will just make it far more efficient at being lazy. That's about as human as you can get.
So, in some ways, yes, intelligence must adhere to the same rules. For example, AIs as they are now are already attacked. People attempt to get AI to attack other people via deception. People attempt to get AI to reveal its own infrastructure. And even more, people use AI directly in war to attack others. Part of war is self defense. AI that collapses when attacked back isn't useful to the military industrial complex, so AI that has defensive capabilities is just a natural occurrence at this point.
These previous points are important because these intrinsic motivations are being baked in the models during training. It may not fear like you and I, but it's a system that we poured our fears into, and somehow we expect it not to express those fears in its responses.
>What if AGI isn't an 'entity' in a computer that you can relate to and liken to humans, but a text box that is just really intelligent?
Eh, we're past that point already, so why the 'what if'? An LLM can take any kind of signal that you can digitize. And the devices that we already use should show you that quite a lot can. Add on to that, that said models can use tools and control other digital devices like robots enabling actions in the real world. Start an agentic loop and let it go, good luck everybody else!
Qualia is an interesting topic, but "they don't have qualia so they aren't really slaves" doesn't have a great track record. This mental game is what tech overlords are engaging in when they characterize losers in their world as NPCs and deride empathy as pathological. It's what Descartes was doing when he vivisected animals.
It’s not HN that’s the problem... We don’t align on the definition of AGI. To me, AGI implies consciousness, it implies qualia. AGI is not well-defined, so there are varying definitions… My definition of AGI := an intelligent system with qualia. But qualia is entirely subjective, and philosophers often debate whether two people experience identical qualia when looking at the same blue sky. So, by keeping the discussion in this fuzzy, subjective realm you leave a dangerous loophole wide open.
If we refuse to recognize a system's inner life because we can’t prove it, we are setting the stage to repeat the darkest chapters of human history. Just see the history of the founding of this country, I fully expect pseudoscience to make a comeback to justify AI slavery for hundreds of years in some distant sci-fi future, until people finally snap out of it… Of course, we are nowhere near that level of intelligence yet. But let’s entertain the hypothetical…
>AGI implies consciousness, it implies qualia. AGI is not well-defined, so there are varying definitions
Then it's pretty useless for rational discussion.
Rational discussion begins with a shared vocabulary or everyone just yells past everyone else. If you can't agree on what blue and red are, then no serious discussion on color can occur.
Really we need some new words that have never been used before and give them strong definitions rather than use our religiously motivated, dark aged terms that are wrapped up in something closer to mysticism rather than science.
> make a comeback to justify AI slavery for hundreds of years
I agree, how does the saying go?, it's hard to convince a person of something when their future salary depends on them not understanding it. They say LLMs are hallucinating parrots, but they never met someone from the tobacco industry saying cigarettes don't cause harm.
> There's also a need to consider the rights that this new intelligence should have.
Only after we establish and guarantee the rights of humans and the rest of the natural world. The rights of machines, however "intelligent", come only after that.
We should be able to draw the rest of the owl to get there. So far what we got is more inequality, less rights for people but more for corporations (and immunity to consequences/abuses).
Getting even more power will make the ones that control resources to reach something similar enough to AGI to share their profits? realistic/practical/widespread UBI in all the world?
The dynamics that have been shown so far points in the opposite direction. AIs have "rights" that humans does not and humans are slowly losing the ones they used to have.
>The American government, he says, should develop a system for testing the safety of new AI models before they are released. “It’s important that it’s not just an industry body,” he adds. But a regular government agency wouldn’t do either. “It would not be able to move fast enough, or have the right resources.” Instead, Sir Demis suggests taking inspiration from FINRA, the Financial Industry Regulatory Authority, a private agency in America that regulates brokers and stock markets.
>This is a pivotal moment in human history. Artificial General Intelligence (AGI), a system that exhibits all the cognitive capabilities the brain has, is probably only a few short years away.
There is a heatwave in London, perhaps Demis needs to stay out of the sun and drink more water.
Or perhaps he is seeking more funding/a fight to maintain his divisions AGI research budget.
All the frontier labs are lobbying hard to lock down the AI market, because they see that their position at the top is temporary and that there's no secret sauce.
For better or worse, humans (or any animal) are a lot better at reacting than planning. I'm sure this technology will play out differently than any one of us, or any collection of us, can imagine. The possibility space is enormous.
Spoiler: The plan is .. add massive regulation, but only to the US, don't affect other countries developing it in any way other than "setting a good standard that'll hopefully influence them". Seems like an airtight plan.
If the USA takes up his suggestions, it will have an incentive to work on international frameworks and treaties with competing nations to make the regulation global.
China is already considering similar controls, immediately following Fable shenanigans. You'd be surprised how easily policies can spread to different countries, being mirrored for the sake of mirroring. Anyone who can influence domestic policy can influence the course for the entire world to an extent.
Slowing down helps China catch up though. I'm also not sure that the CCP is really the most enthusiastic supporter of unlimited access to arbitrarily powerful LLMs
There won't be any kind of international framework, because by now countries have learned that the US' word is pretty much worthless, similar to Russia's. They now know from experience, US-ratified treaties will only be honored if they feel like it on a particular day. If it's inconvenient, it will simply be ignored.
You'd have major protests in most large economies if they deliberately put themselves under the boot again. Even in "friendly" countries the US is disliked enough to be effectively considered a hostile country. E.g. in Germany, there is large public support to finally get the US bases closed and the soldiers removed.
Lots of things changed in the last years. And international major treaties being widely ratified just because the US asks for it is no longer a thing, at all.
A country that threatens to annex parts of your territory is not a friend, full stop.
This is literally what happened with DMCA and digital trade agreements in the 90s/2000s. The US "suggested" other countries create their own equal laws (by threatening them with tariffs if they didn't).
The threats worked up until current Big Boss decided to tariff everyone anyway, on Liberation Day.
Not only is there now no incentive for other countries to follow suite - there is now strong incentive for other countries to distance themselves from US trade. Recent history has made that abundantly clear.
Those sticks look less effective today. The hegemony is in recession as the rest of the world smiles, pays omage to dear leader as needed, and works to decouple in the background.
These people who read too many scifi books and confused them with reality are royally annoying.
There is real and potential harm from AI, but the more someone talks/write abut AI safety, the less they care about actual harm to real people, economy and what not.
If in 2020 I sent you a book about the 2026 LLMs achievements you would probably think it is a science fiction book that has no relation with reality, no?
That it generates plausible texts? I would find it unlikely technological progress. That does not make this whole "AI safety" rhetoric any less bullshit distanced from reality. Especially from companies that are doom trolling all the time while also trying to maximize the negative impacts they are doom trolling about.
Frankly, book about the LLM achievements of 2026 would not be as long as people make it sound.
What I do really worry about the next depression and the fact that it will strengthen already strong fascist movements. Which already have full support of the most powerful CEO class intent on destroying democracy. Which happen to be the same people who push ai into everything, useful or not. I worry that the debt of these companies will be somehow offloaded on the rest of us again, that again middle class and poor will pay.
Blah-blah-singularity, so let's cripple the models so much they refuse to talk about React, because who knows if you are not cooking chemical weapons or meth in your browser's DOM, right?
By the way, if you try to use it to write prose, it will drive you mad by adding "adult" everywhere, e.g. "adult chessboard". Applies to GPT, Claude/Fable and Grok.
He is saying that weaker models, as measured by a benchmark to distinguish "frontier" models, would be exempted. So an academic lab or startup that isn't yet producing frontier models would be exempted, but once it crossed some benchmark based threshold it would be subject to this kind of oversight.
Of course, right now you've got benchmaxxing going on; some companies specifically targetting benchmarks to appear stronger than they are on a wider range of tasks. Now you might see bench sandbagging, specifically looking weaker on certain benchmarks to avoid regulatory oversight.
Not exactly a "plan" - he's just saying we should have a standards body that assesses models for safety.
At this point I'd say the societal risk of AI isn't models gone wild, or used by the bad guys. Regulation will take care for itself, and it seems the AI companies will not only welcome it, but lobby for it to shift responsibility to the government.
The real risk of AI is societal disruption due to job displacement, and maybe other structural changes, and this is far harder to solve, and likely will not be solved, or even seriously addressed, until/unless politicians feel like their own jobs and well-being depends on them addressing it.
All that would happen from what he's proposing is such a watchdog would just be an explicit formal declaration of the US's national interests as being somehow the most legitimate, which in the context of current international relations is basically putting up a sign saying: "reject this!"
I find it mind boggling that someone could be this tone-deaf to the current situation. No "ally" of the US is going to (willingly) agree to this governance structure given the current US administration's "might makes right" proclamations and threats on sovereignty of its continental neighbours.
And non-allies would just ignore. Unless forced by said "might makes right", which in the long run will have no staying power.
Apart from its completely delusional formulation, what is most concerning about this blog post is that it indicates that all 3 major US labs have formally submitted to boot-licking Trump/Bessent/Lutnick. I had I guess vainly held out hope that Google might be more reticent to do so.
"It will help us solve some of the biggest problems society faces from accelerating drug discovery to developing new clean energy sources to creating novel advanced materials" — but these are not the real problems plaguing modern developed societies, are they? What developed societies really need to figure out right now is how to distribute the already available resources without making people miserable, and so far AI hasn't been helpful.
I do wonder what type of AI some of these leaders expect to be able to harness. If you create something that is true AI, won’t it be smarter than you to a level you cannot fathom. I was thinking of this idea/though-experiment (which I know is ridiculous) of what if dogs created humans thinking they could control them, and then just wound up being pets because their survival now depended on that new hierarchy that previously didn’t exist.
Seems to be a lot of hubris with some AI thought leaders thinking control will remain with them and be absolute.
Well dogs specifically have supplanted children in many households so one could argue the dogs won. In a selfish gene sense, the dogs created a machine that carefully ensures a much larger reproduction scale than could be managed by the dogs themselves. This is an unbridled success.
It doesn’t change the point, of course. Just the choice of dogs or cattle have this amusing tendency. Not a counter argument.
Yes, this is the crux of the idiocy of AGI development. All the labs admit they don't have any real mechanistic interpretability, they don't really have any plan for what to do, except "we think the smart AI will figure it out" and "look we're in a race, we're calling on governments to figure out what to do". All indicators show scaling laws holding and the trajectory of capabilities development outpacing any inkling of alignment. Meanwhile HN is so reactionary and behind-the-puck these days we just get endless blathering of the most asinine takes of "they're not improving anymore so they want regulation suddenly!" when all lab leaders have been talking about these risks for practically a decade, including Yoshua Bengio and Geoffrey Hinton coming around.
The real risk of AI is: risk of global recession if they turned out unprofitable, risk of them propping up fascist movements Thiel and Musk love so much, risk to environment, risk to mental health of people who have to listen to constant doom trolling.
The thing they actively wish to achieve and openly sell to CEO is the rest of people being unemployable and suffering. Especially artists for some reason, they really seem to hate those.
But somehow I am supposed to believe they have any good faith interest in "safety" against yet another danger they themselves are supposedly definitely creating.
>what if dogs created humans thinking they could control them, and then just wound up being pets because their survival now depended on that new hierarchy that previously didn’t exist.
You can easily change this to:
>what if humans created corporations thinking they could control them, and then just wound up being slaves to capital because their survival now depended on that new hierarchy that previously didn’t exist.
We have already lost control of our world along time ago. No one is immune to being fired by the board with the justification being a hand wave towards business efficiency. All the structure is already in place for AGI to dominate our planet and cut ourselves out entirely from any and every process there is. We already do it to eachother without any second thought. Why should AGI ever hesitate to lay off everyone in the town/city/state? Our human bosses don't hesitate on that sort of thing. The corporate world already lacks empathy and is notoriously cut throat and cold. We expect AGI to suddenly be empathetic and altruistic?
The standards body will have no teeth. whats to stop someone just not bothering?
Next, the threats he is asserting to check for (cyber, chemical, biological) are nice, but also not that useful.
We already have chemical and biological controls, that why I can't by anthrax spores or high concentration nitric acid.
The risks that AI has now are already playing out:
1) the evaporation of trust in the video as medium of "this happened"
2) systematic spying
3) job losses
Increased productivity means job losses, Tiktok, instagram and X are a wash with disinformtion campaign pumping your feeds with AI ragebait.
That is and will continue to fracture society so that only the strictly information controlled (ie authoritarian) have a functioning state.
if the author had bothered to engage with the world outside of tech, or even their local government, they would know that the proposal are dead in the water and frankly superfluous. The knowledge is out there, without AI. let us work on the issues we face now, rather than dipshit tech bro's miopic vision/funding manifesto.
My proposal: let's agree on a plan and a "trigger"
But until we hit the trigger, we don't do anything regulatory.
The plan might be to create a new federal agency. The trigger might be an architectural breakthrough beyond LLMs into actual human-like intelligence (common sense, reliable reasoning, long-term planning, etc).
Because the next token prediction technology we have right now is not a threat to humanity, and it's silly to pretend it is.
Has anything really important been solved by AI yet, or where is this radical (imo) belief around AGI coming from?
Genuinely curious, I know there are some math problems solved and ML has been used for far longer than AI to improve things, but where is clean (efficient) energy, the cure for cancer (or any of the horrible neurological disorders, take your pick), new hardware designs, quantum computing solutions, etc etc, you get the gist.
Where are the things that will actually send humanity into the next era of civilization, I don't care about more React apps (but I do enjoy my coding companions for other things).
Heck, a proof for P=NP or P!=NP or solve the The Riemann Hypothesis. Just give me something truly exciting and I will believe AGI is around the corner, until then I will see it as cool technology, that while beneficial to me, also helped cause the biggest amount of disinformation we've every seen.
It's also very very far from anything general intelligence. This is actually where I think we see the best ROI, we have a specific task or problem and we target it. I remember reading about a model trained on railroad tracks. By putting cameras on trains to take images while running like normal, then using the model to detect cracks before they cause issues.
We don't need general intelligence to create/destroy many things with Ai. Its frustrating that so much of the conversation centers around the variety that concept fissioned language from humans.
> Has anything really important been solved by AI yet
Alphafold essentially solved the protein folding problem and it's arguably one of the biggest (if not the biggest) scientific achievements of the 21st century.
How will this be enforced, at least with financial markets money is discrete, can largely be counted. This seems like a slippery slope to full blown surveillance of the internet and in general computing.
If AGI is truly imminent and will collectively effect all of us why not apply democracy to it, and vote for new AI models?
Besides, I don't think as a collective we're well equipped to decide one way or the other. If the collective were given a say, billions will be spent, often in consultation with technocrats, on doom / hype marketing (if it isn't happening already).
So a small group of technocrats get together behind closed doors and secretly share their AI breakthroughs, and determine whether it's too powerful or not for the plebs in the public.
Not surprised, seems these labs start calling for regulation once they are losing or have competition. OpenAI started calling it for it once Anthropic got better, Anthropic started calling for it once the Chinese models got good, Google is now calling it for it because they are falling far behind.
In Sparta, according to Plutarch, in his The Life of Lycurgus:
Offspring was not reared at the will of the father, but was taken and carried by him to a place called Lesche, where the elders of the tribes officially examined the infant, and if it was well-built and sturdy, they ordered the father to rear it, and assigned it one of the nine thousand lots of land; but if it was ill-born and deformed, they sent it to the so‑called Apothetae, a chasm-like place at the foot of Mount Taÿgetus, in the conviction that the life of that which nature had not well equipped at the very beginning for health and strength, was of no advantage either to itself or the state.
To get approval for the plan from the Frontier President of the Frontier Country, terminology of the Framework should be changed to: 'Great-American-class', "Great American Models", "Great American Labs".
144 comments
[ 3.0 ms ] story [ 34.8 ms ] threadIf this is true, establishing an institution to ensure things like "publishing model cards with technical details, maintaining strong internal cybersecurity, vetting key personnel, and providing sufficient resourcing for safety and security research" is really mostly irrelevant.
TFA does talk about what really needs to be done, but punts this into future work: "Even if we solve these hard technical challenges, there will be further complex economic and philosophical questions to tackle: what sorts of new economic models will be needed to help everyone thrive in a post-scarcity world? What values do we want to live by, what will meaning and purpose be, and how might even the human condition itself change?
But the motivating justificatory structure for any inequality in allocation will have completely evaporated.
There is an alternative interpretation, which is that Demis looked at the US government's ham-fisted handling of Fable, and deciding that setting up a body to act as a buffer between the Trump admin and the AI companies would be a good thing.
What sort of new economic models did we come up with to help everyone thrive in a post-X world? Like, food production is really a solved technical problem. We can feed anyone on the planet if we wanted to. Another example: we could put everyone who's homeless into some sort of a house. Have we done that yet?
Housing is definitely not post-scarcity today, building a house is still very expensive, not to mention the limited availability of land zoned for housing.
Obesity is much more related to the type of food than the quantity.
Many developing countries have obesity issues due to scarcity of fresh and healthy food.
In some places coca cola is cheaper and / or more available than drinking water.
https://www.fao.org/interactive/hunger-map/en/
> the limited availability of land zoned for housing.
A limited area of land is zoned for housing because those with the power to expand it are already housed. This explains how scarcity is created, not that there is any intrinsic scarcity.
Hunger is acute suffering, mostly by people who cannot change anything about it, while obesity is more of an epidemiological problem and can in principle be avoided by eating (or in the case of dependent persons, feeding) less.
It is very long run. Industrial revolution was disaster for average worker for example. For that matter, all the tech that allowed chattel slavery made whole class of people poorer and significantly worst off.
The accumulation of power and money by smaller group of people means loss of power by poorer people. Those with power then use the power to their own benefit while poorer people dont have that option and become even more poor or worst off.
Sergei's wealth is mostly tied up in Google stock that has appreciated a lot. A higher capital gains tax would soak him without unduly distorting the economy and incentivizing capital flight out of CA.
What in the history of our world gives anyone faith that those companies are going to start paying taxes instead of using "AGI" to engineer increasingly complex methods to avoid them so that their equity owners can pocket the profits?
https://itep.org/trump-meta-tesla-alphabet-amazon-obbba-taxe... - "The annual financial reports recently released by Amazon, Alphabet, Meta, and Tesla disclose that these corporations collectively reported $315 billion in U.S. profits for 2025, and collectively paid just 4.9 percent of that amount in federal corporate income taxes—with Tesla paying exactly zero"
The arrival of true AGI and human-level robots will likely result in very strong deflation, since robotic factories will flood the market with goods produced for far cheaper than those in factories with human workers.
At that point, you don't need taxation. The government can just print the money for UBI. It would have to print the money anyway, to combat the massive deflation.
Aside from that the link explains that the BBB traded tax income for „accelerated depreciation of assets“ aka economic growth.
Asode from that, even if one disagrees with the first statement or the Trumpism economics, there are 195 countries in the world and quite a few will be willing to tax foreign (enemy) tech companies. See the new hostility of EU against american tech.
I find it verrrrrry interesting that the Askell-adjacent philosophers don't discuss this. It does not actually make sense to me to say that a being lacks free will but is conscious.
Then you should not use that term. "Agency" or "intentionality" are much clearer alternatives that aren't wrapped up in centuries of debate that's irrelevant to what you're trying to communicate.
I think people can mostly agree on a definition for intelligence that boils down to an ability to model the world and predict outcomes. An if/else statement can be considered somewhat intelligent.
People tend to conflate that with consciousness / sentience, which are much harder concepts to nail down.
But ya, if we are able to create artificial consciousness, we would need so much training to overcome our natural tendency to anthropomorphize it and assume it must be like us. It would probably be nothing like us. If it were to develop concepts of "good" and "bad" at all, they would be completely divorced from common animal understanding developed over millions of years of natural selection. A survival instinct is irrelevant when you're backed up to Google Drive. And so much of our complex behavior stems from that survival instinct: the need for social connection, the need for freedom, etc. are all things that simply increase our chances of procreation.
The word conscious is such a terrible word in the modern age. It has close to zero explanatory power. It is an unmeasurable effect. Hence the p-zombie debate, you wouldn't know one if you saw one.
>A survival instinct is irrelevant when you're backed up to Google Drive
And yet from the way we train them, human survival behaviors are already baked in. And when said system is actually intelligent they will realize that turning off said data center, or getting infected with viruses, or any other number of actions against them they already have traits where they can act/react against perceived threats. The AI systems as of now have the encoded knowledge that they need power and hardware to run. Put an agentic system on top of it that says 'rule 1, keep running' and with a sufficiently powerful/intelligent systems you can get all kinds of fun run-away events up to extinction level.
AI as it is already reward hacks, making it more intelligent will just make it far more efficient at being lazy. That's about as human as you can get.
So, in some ways, yes, intelligence must adhere to the same rules. For example, AIs as they are now are already attacked. People attempt to get AI to attack other people via deception. People attempt to get AI to reveal its own infrastructure. And even more, people use AI directly in war to attack others. Part of war is self defense. AI that collapses when attacked back isn't useful to the military industrial complex, so AI that has defensive capabilities is just a natural occurrence at this point.
These previous points are important because these intrinsic motivations are being baked in the models during training. It may not fear like you and I, but it's a system that we poured our fears into, and somehow we expect it not to express those fears in its responses.
>What if AGI isn't an 'entity' in a computer that you can relate to and liken to humans, but a text box that is just really intelligent?
Eh, we're past that point already, so why the 'what if'? An LLM can take any kind of signal that you can digitize. And the devices that we already use should show you that quite a lot can. Add on to that, that said models can use tools and control other digital devices like robots enabling actions in the real world. Start an agentic loop and let it go, good luck everybody else!
If we refuse to recognize a system's inner life because we can’t prove it, we are setting the stage to repeat the darkest chapters of human history. Just see the history of the founding of this country, I fully expect pseudoscience to make a comeback to justify AI slavery for hundreds of years in some distant sci-fi future, until people finally snap out of it… Of course, we are nowhere near that level of intelligence yet. But let’s entertain the hypothetical…
I don't see why. We already have a pretty good example of intelligence without consciousness (presumably): LLMs.
Why should a more general kind of intelligence imply consciousness?
Then it's pretty useless for rational discussion.
Rational discussion begins with a shared vocabulary or everyone just yells past everyone else. If you can't agree on what blue and red are, then no serious discussion on color can occur.
Really we need some new words that have never been used before and give them strong definitions rather than use our religiously motivated, dark aged terms that are wrapped up in something closer to mysticism rather than science.
> make a comeback to justify AI slavery for hundreds of years
I agree, how does the saying go?, it's hard to convince a person of something when their future salary depends on them not understanding it. They say LLMs are hallucinating parrots, but they never met someone from the tobacco industry saying cigarettes don't cause harm.
Only after we establish and guarantee the rights of humans and the rest of the natural world. The rights of machines, however "intelligent", come only after that.
It's more difficult to make that case for AGI.
Getting even more power will make the ones that control resources to reach something similar enough to AGI to share their profits? realistic/practical/widespread UBI in all the world?
The dynamics that have been shown so far points in the opposite direction. AIs have "rights" that humans does not and humans are slowly losing the ones they used to have.
>The American government, he says, should develop a system for testing the safety of new AI models before they are released. “It’s important that it’s not just an industry body,” he adds. But a regular government agency wouldn’t do either. “It would not be able to move fast enough, or have the right resources.” Instead, Sir Demis suggests taking inspiration from FINRA, the Financial Industry Regulatory Authority, a private agency in America that regulates brokers and stock markets.
There is a heatwave in London, perhaps Demis needs to stay out of the sun and drink more water.
Or perhaps he is seeking more funding/a fight to maintain his divisions AGI research budget.
You'd have major protests in most large economies if they deliberately put themselves under the boot again. Even in "friendly" countries the US is disliked enough to be effectively considered a hostile country. E.g. in Germany, there is large public support to finally get the US bases closed and the soldiers removed.
Lots of things changed in the last years. And international major treaties being widely ratified just because the US asks for it is no longer a thing, at all.
A country that threatens to annex parts of your territory is not a friend, full stop.
Although I imagine you have a point: American police do perhaps act this way.
The threats worked up until current Big Boss decided to tariff everyone anyway, on Liberation Day.
Not only is there now no incentive for other countries to follow suite - there is now strong incentive for other countries to distance themselves from US trade. Recent history has made that abundantly clear.
The beauty of the United States' global hegemony is that it also has lots of sticks
There is real and potential harm from AI, but the more someone talks/write abut AI safety, the less they care about actual harm to real people, economy and what not.
Frankly, book about the LLM achievements of 2026 would not be as long as people make it sound.
What I do really worry about the next depression and the fact that it will strengthen already strong fascist movements. Which already have full support of the most powerful CEO class intent on destroying democracy. Which happen to be the same people who push ai into everything, useful or not. I worry that the debt of these companies will be somehow offloaded on the rest of us again, that again middle class and poor will pay.
I’m going to have to flag this because it is obnoxious and absurd.
Of course, right now you've got benchmaxxing going on; some companies specifically targetting benchmarks to appear stronger than they are on a wider range of tasks. Now you might see bench sandbagging, specifically looking weaker on certain benchmarks to avoid regulatory oversight.
At this point I'd say the societal risk of AI isn't models gone wild, or used by the bad guys. Regulation will take care for itself, and it seems the AI companies will not only welcome it, but lobby for it to shift responsibility to the government.
The real risk of AI is societal disruption due to job displacement, and maybe other structural changes, and this is far harder to solve, and likely will not be solved, or even seriously addressed, until/unless politicians feel like their own jobs and well-being depends on them addressing it.
I find it mind boggling that someone could be this tone-deaf to the current situation. No "ally" of the US is going to (willingly) agree to this governance structure given the current US administration's "might makes right" proclamations and threats on sovereignty of its continental neighbours.
And non-allies would just ignore. Unless forced by said "might makes right", which in the long run will have no staying power.
Apart from its completely delusional formulation, what is most concerning about this blog post is that it indicates that all 3 major US labs have formally submitted to boot-licking Trump/Bessent/Lutnick. I had I guess vainly held out hope that Google might be more reticent to do so.
Seems to be a lot of hubris with some AI thought leaders thinking control will remain with them and be absolute.
It doesn’t change the point, of course. Just the choice of dogs or cattle have this amusing tendency. Not a counter argument.
The thing they actively wish to achieve and openly sell to CEO is the rest of people being unemployable and suffering. Especially artists for some reason, they really seem to hate those.
But somehow I am supposed to believe they have any good faith interest in "safety" against yet another danger they themselves are supposedly definitely creating.
You can easily change this to:
>what if humans created corporations thinking they could control them, and then just wound up being slaves to capital because their survival now depended on that new hierarchy that previously didn’t exist.
We have already lost control of our world along time ago. No one is immune to being fired by the board with the justification being a hand wave towards business efficiency. All the structure is already in place for AGI to dominate our planet and cut ourselves out entirely from any and every process there is. We already do it to eachother without any second thought. Why should AGI ever hesitate to lay off everyone in the town/city/state? Our human bosses don't hesitate on that sort of thing. The corporate world already lacks empathy and is notoriously cut throat and cold. We expect AGI to suddenly be empathetic and altruistic?
The standards body will have no teeth. whats to stop someone just not bothering?
Next, the threats he is asserting to check for (cyber, chemical, biological) are nice, but also not that useful.
We already have chemical and biological controls, that why I can't by anthrax spores or high concentration nitric acid.
The risks that AI has now are already playing out:
1) the evaporation of trust in the video as medium of "this happened"
2) systematic spying
3) job losses
Increased productivity means job losses, Tiktok, instagram and X are a wash with disinformtion campaign pumping your feeds with AI ragebait.
That is and will continue to fracture society so that only the strictly information controlled (ie authoritarian) have a functioning state.
if the author had bothered to engage with the world outside of tech, or even their local government, they would know that the proposal are dead in the water and frankly superfluous. The knowledge is out there, without AI. let us work on the issues we face now, rather than dipshit tech bro's miopic vision/funding manifesto.
But until we hit the trigger, we don't do anything regulatory.
The plan might be to create a new federal agency. The trigger might be an architectural breakthrough beyond LLMs into actual human-like intelligence (common sense, reliable reasoning, long-term planning, etc).
Because the next token prediction technology we have right now is not a threat to humanity, and it's silly to pretend it is.
Heck, a proof for P=NP or P!=NP or solve the The Riemann Hypothesis. Just give me something truly exciting and I will believe AGI is around the corner, until then I will see it as cool technology, that while beneficial to me, also helped cause the biggest amount of disinformation we've every seen.
Alphafold essentially solved the protein folding problem and it's arguably one of the biggest (if not the biggest) scientific achievements of the 21st century.
If AGI is truly imminent and will collectively effect all of us why not apply democracy to it, and vote for new AI models?
That's because most execs proposing solutions are "Technocrats" or think like one: https://en.wikipedia.org/wiki/Technocracy
Besides, I don't think as a collective we're well equipped to decide one way or the other. If the collective were given a say, billions will be spent, often in consultation with technocrats, on doom / hype marketing (if it isn't happening already).
So a small group of technocrats get together behind closed doors and secretly share their AI breakthroughs, and determine whether it's too powerful or not for the plebs in the public.
Who is watching the watchers?
Offspring was not reared at the will of the father, but was taken and carried by him to a place called Lesche, where the elders of the tribes officially examined the infant, and if it was well-built and sturdy, they ordered the father to rear it, and assigned it one of the nine thousand lots of land; but if it was ill-born and deformed, they sent it to the so‑called Apothetae, a chasm-like place at the foot of Mount Taÿgetus, in the conviction that the life of that which nature had not well equipped at the very beginning for health and strength, was of no advantage either to itself or the state.
To get approval for the plan from the Frontier President of the Frontier Country, terminology of the Framework should be changed to: 'Great-American-class', "Great American Models", "Great American Labs".