I would agree with this, except it sits on the thesis that one company really has that much of an impact. I don't think it actually does. AI is here and it will continue to progress. Norway controlling OpenAI is not going to change it's trajectory -- the cat is already out of the bag.
That's a really unfortunate framing. The cat isn't out of the bag; otherwise, there wouldn't be efforts to scale up data centers to support forecasted demand.
The inevitability narrative is a self-fulfilling prophecy, and it ends if we decide not to keep telling it.
"The cat is out of the bag" doesn't mean the cat is gone, it means the cat has freedom. I read the GP as meaning "no single company can currently control the trajectory of AI." That's not unfortunate -- that's great!
Thanks for the correction. In my own words, to commit to memory: once "the cat is out of the bag", the point is not that it's difficult to put the cat back in, the point is that we now know it's a cat and putting it back is not relevant.
I've seen two expressions referring to "it's difficult to put the (thing) back in" that are in more-or-less common use in English. The slightly more common one is "You can't put the genie back into the bottle", and the slightly less common one (in my experience, at least) is "You can't put the toothpaste back into the tube." (Which actually inspired me to try, and it is actually possible to put the toothpaste back into the tube with careful application of suction, by squeezing the tube gently, touching the end to the excess toothpaste, then releasing the tube and letting some of the excess get sucked back in. But I digress.)
I've actually seen the "genie back into the bottle" phrase used in contexts that refer to secrets, e.g. where "the cat is out of the bag" would actually be more appropriate, so there's definitely some overlap there. For example, I've seen people talking about nuclear proliferation say "you can't put the nuclear genie back into the bottle", meaning that the secret of how to build a nuclear weapon is well-known, and it's not really possible to make it secret again. But I've also seen people use the same "you can't put the nuclear genie back into the bottle" expression referring to a nuclear weapon exploding: once the radioactive material in the bomb has gone critical, there's nothing anybody can do to stop the explosion that's about to happen.
Personally, I like the "genie back into the bottle" phrase better. But some people like more grounded-in-reality phrases rather than fantasy-based phrases, and they prefer "toothpaste back into the tube". Either one works well, though.
I think I have to commend you for trying to put the toothpaste back in the tube. These expressions are more or less some form of fatalism and when used in political context it is very annoying. Especially when some politicians are working there asses off to make something claimed inevitable happen.
In the history of the world, one company can absolutely have world changing impact. From the British East India company to Kodak to Google, one company can have that much of an impact. Whether OpenAI is such a company is a different question for future history books to decide, but that part of their thesis at least is borne out by history.
I'm not against Norway controlling OpenAI, I think it would actually be pretty cool and prevent some of the most obvious vectors (mining personal data etc.)
I just think AI is so much bigger than one company. We already have 3 major frontier labs + Chinese labs releasing at near-frontier levels.
My thesis is that someone will solve the high inference hardware cost problem and another paradigm shift will come where AI becomes even more decentralized and democratized. Obviously that is a hunch but that is what my gut tells me.
Yudkowsky's solution of detecting and physically shutting down non complying datacenter, including bombing if the country doesn't cooperate, isn't too much out of the question.
We would have done the same if it is about virus or nuclear weapons, and AI could become serious existential threat in the next 5 years.
Do you think it's possible to bomb China's AI datacenters? They are already here, and they are progressing AI at their will. What will others do about it?
It is possible for US to bomb anything. Just the retaliation could be huge.
Would you had said the same thing if a country like was developing and testing deadly virus on live population. I think in 5 years, it is possible that the AI will become that topic(if we assume AI becomes good and start hacking and controlling everything).
More or less of an existential threat than a conflict between nuclear powers because one of them decided to launch a unilateral strike against a datacentre located outside one of their cities just in case it hosted a particularly dangerous chatbot?
Trolley problem. What's worse than all of humanity dead from ASI?
Sam Altman: "Development of superhuman machine intelligence (SMI) is probably the greatest threat to the continued existence of humanity. There are other threats that I think are more certain to happen (for example, an engineered virus with a long incubation period and a high mortality rate) but are unlikely to destroy every human in the universe in the way that SMI could."
most of humanity dead because some morons decided to try to kill everyone just in case the server farm evolved the inclination and the capacity to try to kill everyone...
Trolley problem: on the left track is a class of schoolchildren. On the right track is a a computer which a Harry Potter fanfic author argues has magic powers and the marketing department of Apple kind of agrees. Do you let Eliezer kill the kids?
no one dead because we're willing to use our means to enforce a restriction on a scenario that is likely on our trajectory to kill everyone. We also use serious threats to prevent nuclear proliferation that endangers everyone
Serious threats which have completely failed to persuade North Korea et al to stop pursuing nukes. The closest we've actually come to a nuclear exchange was "using serious threats" during the Cuban missile crisis, and the US is currently losing a war nominally to prevent nuclear proliferation to a state that fortunately doesn't actually have the capacity to retaliate except at very short range. You might remember a few other nominally intelligence-lead claims about weapons of mass destruction too not amounting to very much either...
And in this case we're not talking about bombing sites if they appear to be used for uranium enrichment which isn't exactly an everyday need, we're talking about bombing them if they appear to be optimised to contain lots of computers. You don't even have to be an AI sceptic to imagine a policy of threatening to launch missile strikes on Chinese businesses if they build out capacity in a way distant observers don't like might be more likely to result in millions of deaths than the whatever runs on the Chinese business' computers...
OpenAI may be what IBM was to PC or Motorola was to mobile phones. PC and mobile phones certainly changed the world but IBM nor Motorola controlled that change.
Pretty sure this is satire. Maybe it alludes to recent concerns that Norway’s sovereign wealth fund could be exposed to an AI-driven investment bubble or valuation correction.
100% the US government would maintain that AI is the key to the success of the US economy over China, and that it can only happen as long as it's run by profiteering billionaires aiming to extract money from society at the fastest rate possible. Only by accelerating ecological collapse for profit can private US capitalism maintain its edge over freely shared Chinese open-weight models.
Wouldn't be needlessly expensive and risky before the bubble pops? Just buy the scraps after it pops and If it's not bubble, they can alternatively just buy a fab dedicate its output to build ridiculously large infrastructure in Norway and power it with their wast hydro and wind sources, then bring the talent by providing freedom and resource access.
Why on earth would Norway do a charity to the world? The oil fund while it has some moral guidelines, profit making is still the primary goal. There is no guarantee OpenAI will be profitable ever.
Yes, and I think you would get much more effect by going for much smaller and cheaper firms.
Cerebras is only 700 people. I wouldn't buy it, but since it's only 700 people, you can probably build something like it with 1400 people in a couple of years if you set the goals right and hire the right people to lead it.
Euclyd, VSORA etc. exist. If they were buying American, why not Groq.
Then once one has good hardware without an enormous markup, then one can start thinking about building LLMs with it.
according to my understanding, this would (necessarily) include relocation of all OAI stuff to .. somewhere in Europa?
(regardless if this "somewhere" is FR or UK or IT or DE or CH))
OpenAI will likely not IPO this year because their numbers are so poor compared to those of Anthropic's. However if they don't IPO, they cannot pay for all those datacenters they ordered.
Meanwhile, Chinese companies are creating SOTA models and make them available for free, strongly limiting the amount of money that OpenAI and Anthropic can demand for their similar models via API. Their sky high valuations are likely to suffer as chinese models are considered "good enough" and safe to use when hosted in the West. They even have fewer safeguards and are thus more suitable for cybersecurity tasks.
Fastest way to burn $800 billion. Even the Chinese government isn't nationalizing its AI companies, because there's no chance a cutting edge technology company can remain competitive when run by bureaucrats.
OpenAI has a very limited amount of time before their value collapses in the face of cheap Chinese models. The price war is just getting started. It won't be pretty.
OpenAI is worth $800B because that’s what the last funding round valued it at. It does not mean that >50% of existing shareholders will agree to sell for that amount. If a serious buyer does emerge they will likely hold out for 2x that, maybe more.
And everyone should just get along. And software should just work perfectly. And everything should be free.
What's the point of writing something like this? Surely the writer isn't delusional enough to actually expect this to affect state leadership decision-making?
People on the internet have some weird fantasy. News at 6.
Seriously why would Norway do that? With a fraction of that amount of money they can build energy and computing facilities. They can distill OpenAI models. Why would that bet all on one single company?
The US won't let Norway do that anyway. Biden blocked Japan's Nippon Steel from buying US Steel[0]:
Great technology, but $20-$200 a month technology, with a lot of people going for free options. Like the telecom industry. Average is probably going to be like a phone bill. Basically: you can become Verizon or AT&T or T-Mobile. $450 billion a year in revenue. A superb business, but merely a superb business.
Would Norway be prepared to commit to huge future capex spending? Like the pitch that OpenAI is going to achieve AGI seems to rely on vast investments in more compute over the coming years. If you just pay the $800B and then take your foot off the gas, do you still have a frontier lab or have you just paid a lot of money to remove a competitor from the market?
I don't think it works that way for AI. Comms networks are extensive physical infrastructure that, by definition, has to cover ground. AIs cover the ground by riding the existing comms lines.
In telecomms, the late mover has the advantage of not having legacy networks to maintain, and having better technologies available at rollout time. What is the "late mover advantage" in AI? Being able to distill from every bleeding edge frontier lab? That gets you near parity at best.
That assumes they release their models publicly. The future is leaning towards these labs air gapping their best stuff (Model 2, etc) and using it internally to snipe their competitors and charge insane amounts for monitored use in consulting environments.
You can't distill or catch up if you can't access the models. You'll basically have a situation where nation-states will need to try and steal the models Oceans 11 style.
Were you not paying attention? Modern LLMs are incredibly general AIs, and nearing human or superhuman performance in many areas. And the tech just keeps advancing.
If you don't see "AGI is possible", you aren't looking.
Modern LLMs show empirically that the "difference" is fuzzy at best. You can try to draw a distinction. You can try to draw the lines in the sand and define "true intelligence" in a way that would include humans but exclude "false intelligence" of Mythos 5. An exercise in vanity, in my eyes.
Or you can go with "any sufficiently advanced language generation is indistinguishable from intelligence" and drop the matter. My advice is to do exactly that.
We used to think that if a computer could play chess, it would be intelligent. Maybe back then they were also people saying "stop trying to draw lines, just admit it's intelligent!!" Good thing we didn't listen to them.
The act of distinguishing between human intelligence and LLMs is what allows us to figure out how to make it better.
Right. It's actually amazing that large language models can be as effective as they are, given that at their core they are simply matching up word-frequency patterns. But with a large enough context and enough parameters, those word-frequency patterns actually do a decent job of simulating intelligence: I'm able to give rather ambiguous instructions to Claude Code (like "go back to the suggestion you made a while back about (foo) and explain in more detail what the benefits and drawbacks of that approach would be"), and it is able to look through its context, find the part where it suggested (foo), and expand on its suggestion. This is a qualitative difference in human-computer interaction: I can type instructions that are very similar to what I would say to another human being, rather than having to be utterly unambiguous the way you have to be in writing code. It's also good at synthesizing information faster than I could: these days instead of searching MSDN for some obscure API method, I ask Claude "what's the syntax to create a foo from a bar?" and it finds me the MakeBarIntoFoo method faster than I would have (especially because I would have started with CreateFooFromBar and not found it).
But I never forget that it's a simulation of intelligence. I use it for the things it's trained on (generating code) and I don't expect the model to be good at writing poetry, or fiction. Nor do I expect it to have any actual understanding of the things it is actually trained on. Modern models are pretty good at simulating understanding, but even so they will still produce things that a human being would immediately know is wrong, e.g. image-generation models producing hands with the wrong number of fingers, or a person with three arms, or whatever. Those happen less and less often as models have been better trained (and I bet that verification steps are happening behind the scenes to catch and discard some of the classic mistakes), but they still happen.
It's the dancing bear, except this bear is actually managing some really spectacular dance moves. Some of the time. Other times it falls flat on its face. But it's really, really impressive that the bear is actually managing to dance so well.
When I think of Norway, the first thing that comes to mind is its willingness to commit to huge future capex spending. Not to mention, selling more than half its equities in order to buy one risky asset totally aligns with its investment strategy and, you know, having to pay pensions, other minor concerns like that.
In all seriousness though it is pretty interesting to consider if it really happened, a sovereign wealth fund pivots to the crazy high risk strategy, writes a blank check and tries to actually win. If it works does that country just become the rulers of the world?
Norway's fund would first have to definitively find out what it takes to actually run all of OpenAI's compute. Monetarily wise. We still don't know that number.
165 comments
[ 0.21 ms ] story [ 5.3 ms ] threadThe inevitability narrative is a self-fulfilling prophecy, and it ends if we decide not to keep telling it.
People would sell piglets and would try to sell "a pig in a poke" (where a poke is a bag).
That's why you don't buy a pig in a poke because you can't see what you're buying.
Letting the cat out of the bag is because you've revealed that it was a cat rather than a piglet. You've revealed the deception.
I've actually seen the "genie back into the bottle" phrase used in contexts that refer to secrets, e.g. where "the cat is out of the bag" would actually be more appropriate, so there's definitely some overlap there. For example, I've seen people talking about nuclear proliferation say "you can't put the nuclear genie back into the bottle", meaning that the secret of how to build a nuclear weapon is well-known, and it's not really possible to make it secret again. But I've also seen people use the same "you can't put the nuclear genie back into the bottle" expression referring to a nuclear weapon exploding: once the radioactive material in the bomb has gone critical, there's nothing anybody can do to stop the explosion that's about to happen.
Personally, I like the "genie back into the bottle" phrase better. But some people like more grounded-in-reality phrases rather than fantasy-based phrases, and they prefer "toothpaste back into the tube". Either one works well, though.
The saying means that you can't un-tell a secret... once a secret is known to the public, you can't make it a secret again.
I think the more appropriate saying would be a similar, but slightly different, one: "That ship has already sailed"
That one is more about "it is too late to do anything about this" and less about information.
I just think AI is so much bigger than one company. We already have 3 major frontier labs + Chinese labs releasing at near-frontier levels.
My thesis is that someone will solve the high inference hardware cost problem and another paradigm shift will come where AI becomes even more decentralized and democratized. Obviously that is a hunch but that is what my gut tells me.
We would have done the same if it is about virus or nuclear weapons, and AI could become serious existential threat in the next 5 years.
Would you had said the same thing if a country like was developing and testing deadly virus on live population. I think in 5 years, it is possible that the AI will become that topic(if we assume AI becomes good and start hacking and controlling everything).
Sam Altman: "Development of superhuman machine intelligence (SMI) is probably the greatest threat to the continued existence of humanity. There are other threats that I think are more certain to happen (for example, an engineered virus with a long incubation period and a high mortality rate) but are unlikely to destroy every human in the universe in the way that SMI could."
Trolley problem: on the left track is a class of schoolchildren. On the right track is a a computer which a Harry Potter fanfic author argues has magic powers and the marketing department of Apple kind of agrees. Do you let Eliezer kill the kids?
And in this case we're not talking about bombing sites if they appear to be used for uranium enrichment which isn't exactly an everyday need, we're talking about bombing them if they appear to be optimised to contain lots of computers. You don't even have to be an AI sceptic to imagine a policy of threatening to launch missile strikes on Chinese businesses if they build out capacity in a way distant observers don't like might be more likely to result in millions of deaths than the whatever runs on the Chinese business' computers...
Cerebras is only 700 people. I wouldn't buy it, but since it's only 700 people, you can probably build something like it with 1400 people in a couple of years if you set the goals right and hire the right people to lead it.
Euclyd, VSORA etc. exist. If they were buying American, why not Groq.
Then once one has good hardware without an enormous markup, then one can start thinking about building LLMs with it.
Because they're too stupid and stubborn to rename themselves.
I do not see any chance that this may happen
The entire EU and Norway could simply block all clankers and US propaganda and experience an intellectual renaissance. For $0.
Meanwhile, Chinese companies are creating SOTA models and make them available for free, strongly limiting the amount of money that OpenAI and Anthropic can demand for their similar models via API. Their sky high valuations are likely to suffer as chinese models are considered "good enough" and safe to use when hosted in the West. They even have fewer safeguards and are thus more suitable for cybersecurity tasks.
tl;dr: They're fucked. Don't buy them.
“Invade Greenland and we switch it off!”
What's the point of writing something like this? Surely the writer isn't delusional enough to actually expect this to affect state leadership decision-making?
Seriously why would Norway do that? With a fraction of that amount of money they can build energy and computing facilities. They can distill OpenAI models. Why would that bet all on one single company?
The US won't let Norway do that anyway. Biden blocked Japan's Nippon Steel from buying US Steel[0]:
[0]: https://www.bbc.com/news/articles/cx2vz83pg9eo
Great technology, but $20-$200 a month technology, with a lot of people going for free options. Like the telecom industry. Average is probably going to be like a phone bill. Basically: you can become Verizon or AT&T or T-Mobile. $450 billion a year in revenue. A superb business, but merely a superb business.
So all they have to do is wait a bit and build a frontier model of their own.
In telecomms, the late mover has the advantage of not having legacy networks to maintain, and having better technologies available at rollout time. What is the "late mover advantage" in AI? Being able to distill from every bleeding edge frontier lab? That gets you near parity at best.
So they don't spend 20 trillion dollars to achieve AGI, and in the end they still end up at parity for a tiny fraction of the cost.
How can you claim that this isn't a win?
You can't distill or catch up if you can't access the models. You'll basically have a situation where nation-states will need to try and steal the models Oceans 11 style.
They might as well invest in cold fusion, warm superconductivity, curing cancer or whatever sci-fi concept you have.
If you don't see "AGI is possible", you aren't looking.
"The ability to speak does not make you intelligent." - Qui Gon Jinn
Modern LLMs show empirically that the "difference" is fuzzy at best. You can try to draw a distinction. You can try to draw the lines in the sand and define "true intelligence" in a way that would include humans but exclude "false intelligence" of Mythos 5. An exercise in vanity, in my eyes.
Or you can go with "any sufficiently advanced language generation is indistinguishable from intelligence" and drop the matter. My advice is to do exactly that.
The act of distinguishing between human intelligence and LLMs is what allows us to figure out how to make it better.
But I never forget that it's a simulation of intelligence. I use it for the things it's trained on (generating code) and I don't expect the model to be good at writing poetry, or fiction. Nor do I expect it to have any actual understanding of the things it is actually trained on. Modern models are pretty good at simulating understanding, but even so they will still produce things that a human being would immediately know is wrong, e.g. image-generation models producing hands with the wrong number of fingers, or a person with three arms, or whatever. Those happen less and less often as models have been better trained (and I bet that verification steps are happening behind the scenes to catch and discard some of the classic mistakes), but they still happen.
It's the dancing bear, except this bear is actually managing some really spectacular dance moves. Some of the time. Other times it falls flat on its face. But it's really, really impressive that the bear is actually managing to dance so well.
In all seriousness though it is pretty interesting to consider if it really happened, a sovereign wealth fund pivots to the crazy high risk strategy, writes a blank check and tries to actually win. If it works does that country just become the rulers of the world?
Isn't OpenAI running on investor money right now?
j/k. I love the outside of the box thinking here, but that doesn't seem like what the annuity was setup for. Of course thats up to Norway.