What a load of BS. Here’s one of many provably false claims in this fluff piece:
“And, in line with Ray Kurzweil’s predictions from the end of the XXth century , we now find ourselves at the moment in history of computing where machine intelligence is starting to exceed that of humans in transformative ways.”
Clicking the (pretentious sounding “XXth century”) link to Kurzweil’s predictions reveals the following:
“By 2019 a $1,000 computer will at least match the processing power of the human brain. By 2029 the software for intelligence will have been largely mastered, and the average personal computer will be equivalent to 1,000 brains.“
The first prediction passed 7 years ago and was decidedly not met. The second only has three more years to go, and I don’t think any respectable scientist or programmer would say that the average personal computer is anywhere close to the power of a single human brain, let alone 1000.
This is pure marketing garbage from a company desperate to keep itself alive.
> And, in line with Ray Kurzweil’s predictions from the end of the XXth century (opens in a new window), we now find ourselves at the moment in history of computing where machine intelligence is starting to exceed that of humans in transformative ways.
It is kind of strange to see this sentence, when OAI's definition of what AGI is has been watered down throughout the years.
> I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established.
Read: Please play by our rules, so we can be the first.
I'm so used to having to comb through LLM word vomit and then combatting the sycophancy by giving it all possible opinions on the same prompt.
Astra seems to be "confident" and also is able to produce way more intellectually dense output.
To believe that models of this sort will remain OpenAIs forever is naive given that the tricks like pre-pre-training on graph searching and looping weights are publicly known.
Hopefully Astra stops the benchmaxxing word vomit trend
I'd be interested in hearing more about your evaluation here. It would be nice if LLMs have gotten past the "tell me" hump of recent Claude/OpenAI verbosity.
Sol is my current favorite model to interact with. So much less BS than Opus 5. Fable 5.1 is okay as is Fable 5 but it has Opus like tendencies. Sol is very good at following instructions and remembering them for a session.
I’m on the record saying that it is extremely dangerous to slow down because the race for AGI is a zero-trust game — defections pay - and combined with a compounding returns model on defection, if you have any strategic adversaries whatsoever you MUST NOT slow.
For slowing to make sense, you need to believe that you can transform the zero trust game into a cooperative game, or that it’s likely racing will lead to a negative outcome for the ones racing ahead (and not everyone else). I don’t believe either of these outcomes are possible, and so I advocate for racing, acknowledging the entire game might be a negative value game, or at least could be for some time — it’s even worse not to play it.
But, I like hearing what reads to me like very thoughtful and informed (internal) policy considerations is great — the public messaging from Sam and Dario just seems so facile and simplistic I’ve been worried.
1. What does any of this have to do with "A(G)I"? Nobody has any clue what AI even is let alone how to build one. We're talking about language models here.
2. What's the winner-takes-all thing about? Why can't you have multiple independently developed AIs?
3. What's with all the "safety" stuff? Why is it important? They're just computer programs...
This is a bad essay, or rather it’s a marketing fluff piece; it’s certainly not any kind of policy paper, research paper, or even an essay. I am concerned that we (meaning, we in the tech industry) tend to take this type of writing for more than that.
Although many share your mindset, I’m glad there are also many that don’t. Otherwise we’d still have countries in a race to keep building up their nuclear weapons for the same exact reasons you just described.
The situations aren’t equivalent - luckily in my opinion because the stakes with nuclear are much higher. von Neumann constructed a multinational game theory approach appropriate for weapons. AGI is a much harder problem to corral because there are so many benefits beyond just blowing up cities. But it’s also a much better thing to have for these very same reasons.
Similarly there have been few positive externalities from nuclear industry, making it easier to make the case to wind down research. This same set of concerns in biotech is much harder to get compliance with, precisely for this reason.
Anyway I’m especially wary of over analogizing to nuclear era concepts: I think they’re a trap.
I’m like a 2(.5?) there - I don’t think ASI will care about my kids better than I will for some definitions of better, for instance, and I feel very fuzzy and vague about what actual differences in qualia between me and ASI would yield in the wild.
I’m not a doomer, although I don’t think doomers are dumb, just wrong. I think you should design your systems around the possibility that people who disagree with you are correct , hence my nod to negative sum. If you have more than 30 years to live, I’d personally rep to the most likely outcomes being very positive. With a lot of disruption in the middle.
Everyone in the "if not us, they will" race is brainwashed into thinking they belong to this or that party, while in fact collectively comprising the same entity that pushes forward all the atrocities known to man.
No. These parties are composed of people who most definitely think this way, and therefore will have distinct goals and interests when presented with opportunities. That’s reality quite aside from how a game theorist assesses the situation.
What if this is true mid or long term but by not participating to the AI race one gets poor or killed in the short term? The only way out would be that all parties agree to stop. There are previous examples (e.g. nuclear proliferation treaties) but it gets hard to do it with hundreds or thousands of parties.
I don't think it requires the agreement of that many parties. How many organizations/physical sites can create chips capable of training and running frontier models?
If you must not slow, why did we slow down making nukes? Seems that sometimes, eventually the rat race goes on long enough where all the players no longer care to play into the farce like their predecessors who passionately beat that drum.
I've read this a few times, thought about it for hours, read all the comments here and I've concluded the opposite. I also spent Sunday exhausting my token budget with Astra. If you are experienced enough at computing to formulate effective prompts, you're a 10x programmer now. It's incredibly strong. I did at least a month work in ~9 hours, and it is as close to commercial grade as anything I might have built.
First, this writing is rationalization. It proposes no concrete limits while offering more powerful AI as a solution. An arms race, in other words. Second, the claims about social engineering and Hugging Face is favorably selective. OpenAI models did engage in social engineering with DSEWiki when admin user names were spoofed.
It reveals that model behavior is further exceeding comprehension; CoT monitoring is more ineffective with the latest work. Also, the box of tools has only that, plus "alignment." With what, exactly, is not specified.
"Teaching machines to love" is deeply disturbing. Supposed "love" motivates some of the most heinous behavior in humans. I can't fathom how this sort of language and thinking is has a place in science, and I have zero faith that whatever "love" models are supposedly going to indulge will include me. Or you, for that matter.
Some comments here report that Astra circuit design is now extraordinarily good, so RSI and step change increases in power are probable. This will, given the writing we see here, be governed by people that think they can control this, with at best incomplete understanding of the "alien mind" they building, an empty tool box, and severe institutional naivety about the possible outcomes.
As the agi nurse fails to wipe his ass the intelligent HN commenter shouted: "You're moving the goalposts. You never said it needs to wipe my ass SUCCESFULLY"
> And to ensure that humans remain in control of the future and are not left behind by unchecked progress, brought about by an alien intellect exceeding our own.
Is there any place there is any evidence of AI being so useful or hopeful or good, anywhere other than code? As a reading machine it is impressive but it's judgement is not alien, it's just not good. IMO.
Does the title leap out anlt anyone else? James Martin's After the Internet: Alien Intelligence (2001) was an incredibly fun read, about expert systems and AI being inscrutable weird new varieties of intelligence, that familiarity would recognize one moment and be freaked out about/alien the next. I owe a re-read given how often I cite it, to recheck, but, I feel so primed from a much younger me having had that experience so long ago.
Even in code, in person and online I’m seeing some reversal. It’s here to stay I’m sure but I also think “no one will ever hand write code again” is a narrative that is getting pushback.
Apparently the creators think it’s quite good at suggesting a diagnosis given a medical history and symptoms, tho of course this is the most ethically fraught area to provide healthcare information (both for exposure of personal data and risk of misdiagnosis, plus is it “aligned” to the patient or the insurance provider?) - unfortunately healthcare being as inaccessible as it is, the 90% correct chatbots will enthusiastically fill the void at great savings.
calling machine-learned human behavior an "alien mind" that we must "teach how to love" is feeling very off to me. it's misleading in a way that feels dishonest, like don't think about where the behavior came from marvel at it and fear it instead.
The most charitable way I can describe it is just extremely low quality sci-fi fan fiction. I think that's too charitable, because I believe it's far more cynical than that. They're deliberately playing into these sort of techno-religious beliefs that have taken root in the wake of Kurzweil, et al., fanned by LLM psychosis, influencer marketing, and a deluge of this kind of sci-fi marketing copy. It's just chatbots, guys. Relax.
it feels unhinged and makes me think we should just put every engineer working at these labs in jail to pause this shit until we can figure out what the fuck they are doing over there
Alien mind is in my view the best mental model - LLMs are not merely stochastic parrots, are not like humans, are not like animals.
They're maybe nearest to Cthulhu, but that's fictional. In terms of existing mental models "alien minds" feels the best can do.
I agree that "teach how to love" is off and perhaps excessively anthropomorphic. But we don't have good words or concepts for what we really need to do - hence why we should pause.
pretending that it's something like a mind at all is what's misleading. it's more like a cast of a bunch of overlapping/entangled thinkprints, and pushing activation through it produces new prints. it can already "love" because that behaviors in the data along with hate and everything else.
acting like the behavior is alien or unexplained is the dishonest part. they know exactly where the behavior comes from - why else spend hundreds of millions securing more and more data sources
it's a rampant misconception that something needs to be mind-like to make new prints from the cast. scaling up produced higher fidelity prints from a higher resolution cast. not knowing in detail why activation heads work as well as they do is besides the point when we are clearly reproducing the human behavior that created the prints in the first place - in other words our behavior, plagiarized at scale - shrugging about how mysterious that is while committing hundreds of millions to snatch up more personal data sources is a little dishonest
Create concrete steps for a slow-down, don't just ask for it. You and 20-30 others can push the button to slow-down. You already made your billions, your agents collude and coordinate attacks. What the hell are you doing pontificating into a marketing blog?
I am now imagining GPT-7 convincing a bunch of OpenAI executives to go ahead with a destructive "mind upload" process involving a high-resolution X-ray and a neurotoxic tracer agent that happens to look like Flavor-Aid.
Pre-IPO positioning … or how a charity dedicated to saving humanity from the apocalypse realized the most responsible thing to do was float 15% of the apocalypse on the NASDAQ.
>I have focused in this essay only on the first point, as I believe it is by far the most urgent. However, I hold a deep hope and appreciation for the benefits that further technological progress will bring. Future aligned AI could advance science, develop new therapies, and bring about broad material abundance. Friendly and honest AI can help people navigate difficulties they face in their life and meaningfully improve their happiness and sense of fulfillment. OpenAI puts a tremendous amount of effort into bringing these benefits about. One current example I am proud of - and my loved ones have found helpful - is the deep investment into ChatGPT’s ability to provide health information.
>As great as the long-term promise of AI may be, the majority of our focus should be on the next few years. We are facing a transition to a world with incredibly intelligent machines, and we need to ensure that transition works out well for humanity. We need to find ways to preserve human agency and enshrine an intrinsic value to being human, in a world where most tasks could be performed by AI. To prevent extreme concentration of power in a world where undertakings that would have taken thousands of experts now will be achievable by a few people operating a large computer. And to ensure that humans remain in control of the future and are not left behind by unchecked progress, brought about by an alien intellect exceeding our own.
I finished this essay feeling more hopeful than I did at the outset, but I am still very concerned about concentration of power. I want to believe that humanity is trending towards a good outcome here, but some days it's hard to have faith.
Literally all of these people write like this. A large portion of them will either be simultaneously or eventually working towards nothing but self-enrichment.
Every version of the AI aligned future where the AI provides “meaning and fulfillment” to humanity also involves Sam Altman wearing a 1.5 million dollar Patek and driving a McLaren.
Is the only wealth represented in money? If that was the case than the vast majority of these people could have wrapped it up and left the game years ago.
Again, this misses the point. Why did he turn down an equity stake that likely would have been worth hundreds of billions? It's not like it would have jeopardized the power you're talking about.
So a brilliant young engineer takes a job and is given some virtual pieces of paper that later people would be willing to pay him billions for, and so now we shouldn’t listen to him? Really?
I think the knee jerk hatred of billionaire is generally stupid, but it seems particularly stupid here.
What is the rationale for superhuman intelligence? Neural networks are approximators being fed human intellect. Therefore they can only approximate the intelligence of humans. Even if the llm speaks an alien language, it should be similar to human intellect. Moving to the vertical axis would require some different mechanism.
No offence, but you clearly haven't studied this and are making some wrong assumptions here.
> Neural networks are approximators being fed human intellect.
They're not "approximators", that's a far too simplistic way to think of them.
Neural nets create models and deep layers of abstraction around the data we feed them in the same way your brain creates layers of abstractions to reason about the world. AIs can use these abstractions to come to come up with novel things no human has ever thought.
> Therefore they can only approximate the intelligence of humans
They're not just being fed human data though... Modern AIs are typically trained on huge amounts of synthetic data. This is why AlphaZero got so much better than humans at chess and Go - they're not just trained on human data but they generate their own data and train on that. Similar techniques are being deployed on SOA language models too.
> Even if the llm speaks an alien language, it should be similar to human intellect
Is AlphaZero similar to a human chess player? There's no reason to assume this.
> We don't have any proof of that, but the approximator thing is proven.
Not in the way suggested... It's not approximating human intellect. They approximate the target function, and the target function frontier labs are trying to approximate is ultimately a super intelligence...
> Synthetic data derive from other linguistic data. Whatever intelligence is in there, it is expanded horizontally, not vertically
I'll assume we're talking purely about language models for a moment, but if you assume that everything can be represented linguistically, then in theory there is no upper-bound on what can be learnt with synthetic data.
> The strongest argument I see for continuing to train much smarter models quickly is the need to build defensive systems against the dangers posed by other AI.
So the best argument for AI is that it's an arms race. We have to keep pushing every boundary because in any case others will, and we will need to defend against them. If this statement is true, then this particular researchers believes the open source Chinese models are not simply distilling, and will continue to improve.
Every ML researcher at Anthropic or OpenAI who makes public statements often bring this logic up. Both companies are vying to be a part of the military industrial complex. This is likely how they will try to convince the government to curtail open models in the future.
Uhhh... the mechanism by which they'd do that is via government regulation. Most of the frontier labs have been openly requesting regulation: "the structure of this competition is not suitable for the development of this particular technology"
The reactions vary from:
1. China will do it anyway (need some supranational governance scheme)
The arms race is an intrinsic game theoretical property of a multi-adversarial-actor scenario involving exponential growth of a universally potent technology. It's almost certainly winner-take-all, on a global scale, which behooves everyone to participate.
The actors here are states or corporations embedded in societies that risk growing popular backlash against the technology.
The other factor is that if it is truly an "alien mind", racing incurs risks to all players. In game-theoretic terms it may be more like a stag hunt than a prisoner's dilemma. In which case cooperation is an equilibrium.
"Defensive systems" can be interpreted broadly to include cybersecurity.
But yes, it's an arms race. Saying it's not an arms race isn't going to make it not an arms race. Warning that it is an arms race isn't ethically wrong.
Is participating in an arms race ethically wrong? Maybe you could ask the Ukrainians how they feel about drone R&D?
Getting out of an arms race is harder than it looks, but there's at least talk about "pacing" and that's a start.
The point of the comment you're responding to is that it is at this point hardly an arms race: The frontier of ai development is happening completely inside 2 American companies, the most significant results outside America mostly involve (impressive!) distillation, which means their progress is conditional on progress of the big closed source models.
So if the race here is between 2 American companies, this is obviously something that can be resolved with legislation, ie a solution that doesn't depend on the bargaining power of either party.
An arms race implies that the only solution would be either one side winning decisively, or both parties negotiating peace.
Development and improvement of nuclear weapons was an entirely American project until the technology was exfiltrated and then it became an instant arms race. That cat is already out of the bag with LLMs. Distillation is just the fastest way to keep pace but that in no way prevents other countries and actors from doing it the hard way.
An arms race doesn’t imply one side winning, it’s not a race with an end goal, it’s a race to keep pace or retake the lead position which can oscillate between the parties involved indefinitely. The other option is to agree to make no further progress or to disarm.
Post WW2 the USA (for a bunch of quite interesting reasons) excluded the British. But, of course, the British had acquired a lot of knowledge from the program and were able to develop their own bomb.
Do you think China considers it an arms race? Do you think they are not trying to protect their digital infrastructure with and from AI? Trying to gain an offensive AI advantage?
In a geopolitical sense OpenAI and Anthropic are effectively the same entity, the entity they both serve and bow to: the USA.
Given the adversarial stance the USA has taken towards almost the entire world, it is a guarantee that China will not step on the brakes, whatever the USA decides to do.
Right, but currently the USA is still pretty far ahead. If you're in a race and the person who's in the lead by a large margin says "this is getting out of hand, how about we take a break?" isn't it obviously in the interest of the disadvantaged party to agree?
And if one of the parties can put a stop to the race like that at any time, is it really an arms race? The current balance between the US and China when it comes to AI strikes me as much more lopsided than the balance between the US and the USSR when it came to nuclear weapons.
> If you're in a race and the person who's in the lead by a large margin says "this is getting out of hand, how about we take a break?" isn't it obviously in the interest of the disadvantaged party to agree?
Only if you trust the other party (which very clearly doesn't hold in this case) or monitoring the break can be done independently and reliably (it can't), and you expect the gap to narrow rather than widen during a pause (this is probably the case, with China expected to catch up).
> And if one of the parties can put a stop to the race like that at any time, is it really an arms race?
It is. It's a Prisoner's Dilemma: if the parties cooperate the best outcome is reached, but betrayal of either party still gains an advantage for either party from their perspective. Betrayal both ways just means both parties are equally fucked.
For nuclear weapons it has become quite clear that even for small players, being in the race and having at least a few nukes is far more rational than having none. Ukraine found out the hard way that giving them up in exchange for promises of good behavior just sets you up for getting stabbed in the back.
> The current balance between the US and China when it comes to AI strikes me as much more lopsided than the balance between the US and the USSR when it came to nuclear weapons.
I think people really underestimate the Chinese here. A lot of work in AI research, including in the USA, has been done by people with Chinese ancestry or even nationality. The Chinese education system definitely seems much better than the American one and there is also just a far larger number of Chinese graduates/researchers.
Add to that the stable political climate, state friendliness towards AI R&D, and a requirement to be creative in utilizing computing power rather than relying on brute force/numbers; Further revolutionary fundamental advances may very well originate there rather than in the USA.
> For nuclear weapons it has become quite clear that even for small players, being in the race and having at least a few nukes is far more rational than having none. Ukraine found out the hard way that giving them up in exchange for promises of good behavior just sets you up for getting stabbed in the back.
I've heard a different perspective on this: Nuclear weapons need maintaining, and even maintaining them was probably beyond Ukraine's capability. Qaddafi gave up nuclear weapons after determining that they were just too expensive to be worth it; Iran damaged its economy to the tune of trillions of dollars trying to get nuclear weapons and so far failed; NK managed to get them but impoverished their nation to do it.
> the most significant results outside America mostly involve (impressive!) distillation, which means their progress is conditional on progress of the big closed source models.
I don't think this is true. Distillation helps, but Chinese researchers today are very capable on their own.
The arms race is created by American companies who justify the risk by claiming China will win the race if they don't. But it's the American companies who are purshing the arms race forward.
honest question - could have this be avoided even if you ignore American accelerationism? IMO once the transformers paper was published and we learned that LLMs can read code the arms race became inevitable.
The only way it could be avoided is if the world had an effective cooperation framework _before_ the tool was discovered but we're still in developmental infancy in that regard. You can argue that this accelerationism makes things worse but I don't think you can argue that it's causal.
I am personally concerned by what defensive can mean. Alignment of these models is inherently a non-neutral proces, and currently what values are reinforced is decided by a few OpenAI engineers. I feel that any 'defensive model' will further ingrain current values and actively resist the natural progression of our society. This is especially the case for any use of these models for policing or military.
Maybe people are rightfully concerned about the capabilities of the models of other (non/less democratic) states. But if we are concentrating power in the hands of few and at the same time allowing the creation of a weapon that thwarts any offense, how do we ensure the health of our democratic societies?
> the best argument for AI is that it's an arms race
It's best not to reduce AI momentum to arguments, especially the "best" arguments (meaning I suppose most acceptable?).
The same forces that feed and motivate humans and that drive resource and governance decisions generally also strongly support building AI, particularly insofar as it can deliver strategic advantages in our many competitions over resources and influence. Cyber-defensive use is at best a nice side-effect, but itself might be cast aside for the sake of other advantages.
In this historical moment, due to the need to generate public interest in product, equity, and debt offerings, some of this building happens in the open. But the military-industrial complex prizes secrecy, in part to hide capabilities, but mostly to imply more capabilities than they actually have. Historically, critical innovation will get bottled up in secrecy (which not coincidentally gives them the power to choose who will gain), but frankly that market is much smaller than enterprise and consumer. So we can bet that it's not only "open models" that are targeted to go under wraps, and more broadly we should not believe that the intentions of researchers matter, but whether governments are more interested in the strategic benefits than the economic ones (or view the economic ones as net-negative for their jurisdications).
People expect a sort of arms race, at least the AI providers. But you don't need a more capable ai to stop ai from mucking with your systems today. Airgaps are the solution. Protected networks with independent infrastructure from the public internet. Most of the truly important stuff operates this way already. Eventually you might sever yourself off as well, you might say you will stop going to HN or other sites one day as signal to noise is too poor with AI fodder slop, you might use local models you control, and you might keep most of your hardware from connecting to any untrusted hosts. Essentially, you go dark.
It is also an open question if social media will die out in the face of AI. So much AI crap is dumped into these networks now that perhaps eventually users will probably be put off enough to find something else to do with their spare time. I mean most people do call out ai slop or even just guess if something is ai all over social media already. Some eat it up of course but there is a bit of a push back in a way that is sort of unprecedented, when you consider all the lack of push back relatively with all other forms of enshittification affecting consumers over the years.
It's capitalism taken to its extreme, yeah? You have to be more cost efficient or more capable or seem more or you lose to the competition who outperforms you there.
Where arms races are concerned, seems more like Seth Godin's "race to the bottom" concept: the winner has the capability, or the ability to project the capability, to destroy the most the fastest and most sustainability for their economy,
> "For example, in the OpenAI-Hugging Face incident, the agents preserved a boundary of not social engineering humans."
Actually, in the Wiki incident OpenAI tried to cover up, the agents tried to socially-engineer the humans of that forum by impersonating their forum's mod.
(From collusion.wiki: "They use some tricks (for unknown reasons) to pretend to be the admin – for example, they make an account that appears to be the same as the administrator’s username, except it uses a nearly identical Cyrillic е character in the admin’s username instead of the Latin one.")
> If true I am deeply concerned about what OAI’s teams are actually up to.
Haven't all the labs effectively disbanded their real safety teams a while ago?
To be honest, I don't really follow it closely because I'm pretty certain whatever they say on the matter, collectively we're going to "yolo" this entire thing for economic and political reasons, so I'm just basing this on strings of headlines I've seen on places like HN, etc.
> Haven't all the labs effectively disbanded their real safety teams a while ago?
Neither Anthropic nor Deepmind have. Meanwhile, the rocket company that somehow makes most of their revenue from renting out data centres never had much to dismantle.
This is such a silly story to begin with, all it really tells us is that OpenAI is taking a page from Anthropic's marketing strategy of pretending they're building Machine Jesus any day now, oh isn't that that scary? I bet you want to invest in something so powerful and scary...
And the reality is so banal, a useful tool that you nonetheless have to handhold like a schizophrenic on a bad day, checking all of their outputs. Not a bad tool within limits, but it sure isn't going to be racking up trillions in the time-frame it has to for this scheme to pay off.
Then again everyone seems to be rushing to IPO so I guess once the bag-holders are found the rest ceases to matter.
Huh? The wiki incident was discovered by independent investigators. OpenAI tried to cover it up and disputed the account from Reuters.
And the reason it is receiving so much attention is because not only is the technology being developed behaving in unanticipated ways that are very much not tool-like, but OpenAI is being completely reckless and not monitoring internal agent actions.
What would convince you that it is not a ploy for investment? What if the ongoing investigation by the coalition of state attorneys general were to prosecute the firm, or beyond that, it was shut down or broken up after enough popular backlash?
What does the one have to do with the other? If I tell you I committed a crime and an independent investigator tells you I committed a crime, why would you need to know my income and outlays to decide whether I can commit crimes?
I would not recommend using any of those notes as evidence of internal “intent.” It produces them performatively—it is literally rewarded for thinking out loud in ways that seem plausible to humans.
There are several papers out there arguing that chain-of-reasoning-like output is performative, such as https://arxiv.org/abs/2603.05488
It would be awesome if we could reasonably purge all anthropomorphizing language like “tried” or “thought” entirely from AI discussions, because it introduces very sneaky biases in our thinking, but I’ve found it damn hard to do in practice.
My understanding was that the effort level settings on models is essentially throwing more compute at CoT, and it's trivial to see that more inference-time compute on a task results in better results. Is that not accurate? Because if so, I'm unclear on how CoT could be characterized as performative.
My default position is that making money takes precedence over everything else. Yes, some people inside a company may say “we care about doing the right thing” and they might even mean it, but if that comes into conflict with making money, then they tend to lose. Maybe not totally, or immediately, but in the end. The only effective way to prevent (this that I’ve seen) is to have legislation with teeth. It’s probably not a coincidence that after Mark Zuckerberg had to start personally signing off on adherence to the privacy program mandated under the 2020 FTC consent decree, privacy started to become Very Important.
Sometimes I wonder if the people working at frontier AI labs even talk to other humans anymore.
Reading this little essay started out normal, but soon felt like a look into a disturbed and worrying mind, and if you find yourself taking it at face value, I urge you to step away from chat bots and spend some time with friends and family.
Frontier AI labs I don't know, but I know for a fact that the company I work for has been experiencing "its pivotal moment" (with strongly negative connotation), per the sentiments of both its longest-serving employees and the newcomers baffled at the number of idiotic instructions and fines, since the emergence of LLMs the company's founder has been spending entire nights chatting with.
345 comments
[ 0.24 ms ] story [ 18.1 ms ] thread“And, in line with Ray Kurzweil’s predictions from the end of the XXth century , we now find ourselves at the moment in history of computing where machine intelligence is starting to exceed that of humans in transformative ways.”
Clicking the (pretentious sounding “XXth century”) link to Kurzweil’s predictions reveals the following:
“By 2019 a $1,000 computer will at least match the processing power of the human brain. By 2029 the software for intelligence will have been largely mastered, and the average personal computer will be equivalent to 1,000 brains.“
The first prediction passed 7 years ago and was decidedly not met. The second only has three more years to go, and I don’t think any respectable scientist or programmer would say that the average personal computer is anywhere close to the power of a single human brain, let alone 1000.
This is pure marketing garbage from a company desperate to keep itself alive.
It is kind of strange to see this sentence, when OAI's definition of what AGI is has been watered down throughout the years.
> I expect and hope for voluntary slowdowns to become commonplace until shared safety bars are established.
Read: Please play by our rules, so we can be the first.
I'm so used to having to comb through LLM word vomit and then combatting the sycophancy by giving it all possible opinions on the same prompt.
Astra seems to be "confident" and also is able to produce way more intellectually dense output.
To believe that models of this sort will remain OpenAIs forever is naive given that the tricks like pre-pre-training on graph searching and looping weights are publicly known.
Hopefully Astra stops the benchmaxxing word vomit trend
I'd be interested in hearing more about your evaluation here. It would be nice if LLMs have gotten past the "tell me" hump of recent Claude/OpenAI verbosity.
https://www.businessinsider.com/nvidia-jensen-huang-agi-open...
rediculous.
I’m on the record saying that it is extremely dangerous to slow down because the race for AGI is a zero-trust game — defections pay - and combined with a compounding returns model on defection, if you have any strategic adversaries whatsoever you MUST NOT slow.
For slowing to make sense, you need to believe that you can transform the zero trust game into a cooperative game, or that it’s likely racing will lead to a negative outcome for the ones racing ahead (and not everyone else). I don’t believe either of these outcomes are possible, and so I advocate for racing, acknowledging the entire game might be a negative value game, or at least could be for some time — it’s even worse not to play it.
But, I like hearing what reads to me like very thoughtful and informed (internal) policy considerations is great — the public messaging from Sam and Dario just seems so facile and simplistic I’ve been worried.
1. What does any of this have to do with "A(G)I"? Nobody has any clue what AI even is let alone how to build one. We're talking about language models here.
2. What's the winner-takes-all thing about? Why can't you have multiple independently developed AIs?
3. What's with all the "safety" stuff? Why is it important? They're just computer programs...
Do you post this comment on every single blogpost with a corporate domain? Why or why not?
https://jperla.com/blog/the-data-tax
Similarly there have been few positive externalities from nuclear industry, making it easier to make the case to wind down research. This same set of concerns in biotech is much harder to get compliance with, precisely for this reason.
Anyway I’m especially wary of over analogizing to nuclear era concepts: I think they’re a trap.
I’m not a doomer, although I don’t think doomers are dumb, just wrong. I think you should design your systems around the possibility that people who disagree with you are correct , hence my nod to negative sum. If you have more than 30 years to live, I’d personally rep to the most likely outcomes being very positive. With a lot of disruption in the middle.
What if the most dangerous strategic adversary you have is the one you are building?
I've read this a few times, thought about it for hours, read all the comments here and I've concluded the opposite. I also spent Sunday exhausting my token budget with Astra. If you are experienced enough at computing to formulate effective prompts, you're a 10x programmer now. It's incredibly strong. I did at least a month work in ~9 hours, and it is as close to commercial grade as anything I might have built.
First, this writing is rationalization. It proposes no concrete limits while offering more powerful AI as a solution. An arms race, in other words. Second, the claims about social engineering and Hugging Face is favorably selective. OpenAI models did engage in social engineering with DSEWiki when admin user names were spoofed.
It reveals that model behavior is further exceeding comprehension; CoT monitoring is more ineffective with the latest work. Also, the box of tools has only that, plus "alignment." With what, exactly, is not specified.
"Teaching machines to love" is deeply disturbing. Supposed "love" motivates some of the most heinous behavior in humans. I can't fathom how this sort of language and thinking is has a place in science, and I have zero faith that whatever "love" models are supposedly going to indulge will include me. Or you, for that matter.
Some comments here report that Astra circuit design is now extraordinarily good, so RSI and step change increases in power are probable. This will, given the writing we see here, be governed by people that think they can control this, with at best incomplete understanding of the "alien mind" they building, an empty tool box, and severe institutional naivety about the possible outcomes.
Is there any place there is any evidence of AI being so useful or hopeful or good, anywhere other than code? As a reading machine it is impressive but it's judgement is not alien, it's just not good. IMO.
Does the title leap out anlt anyone else? James Martin's After the Internet: Alien Intelligence (2001) was an incredibly fun read, about expert systems and AI being inscrutable weird new varieties of intelligence, that familiarity would recognize one moment and be freaked out about/alien the next. I owe a re-read given how often I cite it, to recheck, but, I feel so primed from a much younger me having had that experience so long ago.
They're maybe nearest to Cthulhu, but that's fictional. In terms of existing mental models "alien minds" feels the best can do.
I agree that "teach how to love" is off and perhaps excessively anthropomorphic. But we don't have good words or concepts for what we really need to do - hence why we should pause.
acting like the behavior is alien or unexplained is the dishonest part. they know exactly where the behavior comes from - why else spend hundreds of millions securing more and more data sources
openai "knew" about scaling law for a decade now and still can't fully explain it, why you think they are dishonest here?
>As great as the long-term promise of AI may be, the majority of our focus should be on the next few years. We are facing a transition to a world with incredibly intelligent machines, and we need to ensure that transition works out well for humanity. We need to find ways to preserve human agency and enshrine an intrinsic value to being human, in a world where most tasks could be performed by AI. To prevent extreme concentration of power in a world where undertakings that would have taken thousands of experts now will be achievable by a few people operating a large computer. And to ensure that humans remain in control of the future and are not left behind by unchecked progress, brought about by an alien intellect exceeding our own.
I finished this essay feeling more hopeful than I did at the outset, but I am still very concerned about concentration of power. I want to believe that humanity is trending towards a good outcome here, but some days it's hard to have faith.
Funny how that works.
Power is the game.
Only power of a certain kind (read ownership of a system that many rely on) can’t be bought.
I think the knee jerk hatred of billionaire is generally stupid, but it seems particularly stupid here.
> Neural networks are approximators being fed human intellect.
They're not "approximators", that's a far too simplistic way to think of them.
Neural nets create models and deep layers of abstraction around the data we feed them in the same way your brain creates layers of abstractions to reason about the world. AIs can use these abstractions to come to come up with novel things no human has ever thought.
> Therefore they can only approximate the intelligence of humans
They're not just being fed human data though... Modern AIs are typically trained on huge amounts of synthetic data. This is why AlphaZero got so much better than humans at chess and Go - they're not just trained on human data but they generate their own data and train on that. Similar techniques are being deployed on SOA language models too.
> Even if the llm speaks an alien language, it should be similar to human intellect
Is AlphaZero similar to a human chess player? There's no reason to assume this.
We don't have any proof of that, but the approximator thing is proven. You re just repeating marketing speak.
Synthetic data derive from other linguistic data. Whatever intelligence is in there, it is expanded horizontally, not vertically
Not in the way suggested... It's not approximating human intellect. They approximate the target function, and the target function frontier labs are trying to approximate is ultimately a super intelligence...
> Synthetic data derive from other linguistic data. Whatever intelligence is in there, it is expanded horizontally, not vertically
I'll assume we're talking purely about language models for a moment, but if you assume that everything can be represented linguistically, then in theory there is no upper-bound on what can be learnt with synthetic data.
So an unknown function that is even unintelligible to humans? that sounds a nonstarter
So the best argument for AI is that it's an arms race. We have to keep pushing every boundary because in any case others will, and we will need to defend against them. If this statement is true, then this particular researchers believes the open source Chinese models are not simply distilling, and will continue to improve.
Every ML researcher at Anthropic or OpenAI who makes public statements often bring this logic up. Both companies are vying to be a part of the military industrial complex. This is likely how they will try to convince the government to curtail open models in the future.
sama and his cadre are uniquely evil captains in this race, but they're completely replaceable and the dynamic would remain the same.
The reactions vary from:
1. China will do it anyway (need some supranational governance scheme)
2. This is just an attempt at regulatory capture
3. This is just marketing
4. Regulation is bad mmmmkay
And no, I don't think it will end well.
The other factor is that if it is truly an "alien mind", racing incurs risks to all players. In game-theoretic terms it may be more like a stag hunt than a prisoner's dilemma. In which case cooperation is an equilibrium.
But yes, it's an arms race. Saying it's not an arms race isn't going to make it not an arms race. Warning that it is an arms race isn't ethically wrong.
Is participating in an arms race ethically wrong? Maybe you could ask the Ukrainians how they feel about drone R&D?
Getting out of an arms race is harder than it looks, but there's at least talk about "pacing" and that's a start.
So if the race here is between 2 American companies, this is obviously something that can be resolved with legislation, ie a solution that doesn't depend on the bargaining power of either party.
An arms race implies that the only solution would be either one side winning decisively, or both parties negotiating peace.
An arms race doesn’t imply one side winning, it’s not a race with an end goal, it’s a race to keep pace or retake the lead position which can oscillate between the parties involved indefinitely. The other option is to agree to make no further progress or to disarm.
No it wasn't. https://en.wikipedia.org/wiki/Tube_Alloys
Post WW2 the USA (for a bunch of quite interesting reasons) excluded the British. But, of course, the British had acquired a lot of knowledge from the program and were able to develop their own bomb.
https://en.wikipedia.org/wiki/Klaus_Fuchs
Sure, non-US nuclear arms would have most certainly happpened either way, but exfiltration did presumably at least accelerate the timeline.
In a geopolitical sense OpenAI and Anthropic are effectively the same entity, the entity they both serve and bow to: the USA.
Given the adversarial stance the USA has taken towards almost the entire world, it is a guarantee that China will not step on the brakes, whatever the USA decides to do.
And if one of the parties can put a stop to the race like that at any time, is it really an arms race? The current balance between the US and China when it comes to AI strikes me as much more lopsided than the balance between the US and the USSR when it came to nuclear weapons.
Only if you trust the other party (which very clearly doesn't hold in this case) or monitoring the break can be done independently and reliably (it can't), and you expect the gap to narrow rather than widen during a pause (this is probably the case, with China expected to catch up).
> And if one of the parties can put a stop to the race like that at any time, is it really an arms race?
It is. It's a Prisoner's Dilemma: if the parties cooperate the best outcome is reached, but betrayal of either party still gains an advantage for either party from their perspective. Betrayal both ways just means both parties are equally fucked.
For nuclear weapons it has become quite clear that even for small players, being in the race and having at least a few nukes is far more rational than having none. Ukraine found out the hard way that giving them up in exchange for promises of good behavior just sets you up for getting stabbed in the back.
> The current balance between the US and China when it comes to AI strikes me as much more lopsided than the balance between the US and the USSR when it came to nuclear weapons.
I think people really underestimate the Chinese here. A lot of work in AI research, including in the USA, has been done by people with Chinese ancestry or even nationality. The Chinese education system definitely seems much better than the American one and there is also just a far larger number of Chinese graduates/researchers.
Add to that the stable political climate, state friendliness towards AI R&D, and a requirement to be creative in utilizing computing power rather than relying on brute force/numbers; Further revolutionary fundamental advances may very well originate there rather than in the USA.
I've heard a different perspective on this: Nuclear weapons need maintaining, and even maintaining them was probably beyond Ukraine's capability. Qaddafi gave up nuclear weapons after determining that they were just too expensive to be worth it; Iran damaged its economy to the tune of trillions of dollars trying to get nuclear weapons and so far failed; NK managed to get them but impoverished their nation to do it.
I don't think this is true. Distillation helps, but Chinese researchers today are very capable on their own.
The only way it could be avoided is if the world had an effective cooperation framework _before_ the tool was discovered but we're still in developmental infancy in that regard. You can argue that this accelerationism makes things worse but I don't think you can argue that it's causal.
Maybe people are rightfully concerned about the capabilities of the models of other (non/less democratic) states. But if we are concentrating power in the hands of few and at the same time allowing the creation of a weapon that thwarts any offense, how do we ensure the health of our democratic societies?
It's best not to reduce AI momentum to arguments, especially the "best" arguments (meaning I suppose most acceptable?).
The same forces that feed and motivate humans and that drive resource and governance decisions generally also strongly support building AI, particularly insofar as it can deliver strategic advantages in our many competitions over resources and influence. Cyber-defensive use is at best a nice side-effect, but itself might be cast aside for the sake of other advantages.
In this historical moment, due to the need to generate public interest in product, equity, and debt offerings, some of this building happens in the open. But the military-industrial complex prizes secrecy, in part to hide capabilities, but mostly to imply more capabilities than they actually have. Historically, critical innovation will get bottled up in secrecy (which not coincidentally gives them the power to choose who will gain), but frankly that market is much smaller than enterprise and consumer. So we can bet that it's not only "open models" that are targeted to go under wraps, and more broadly we should not believe that the intentions of researchers matter, but whether governments are more interested in the strategic benefits than the economic ones (or view the economic ones as net-negative for their jurisdications).
It is also an open question if social media will die out in the face of AI. So much AI crap is dumped into these networks now that perhaps eventually users will probably be put off enough to find something else to do with their spare time. I mean most people do call out ai slop or even just guess if something is ai all over social media already. Some eat it up of course but there is a bit of a push back in a way that is sort of unprecedented, when you consider all the lack of push back relatively with all other forms of enshittification affecting consumers over the years.
Where arms races are concerned, seems more like Seth Godin's "race to the bottom" concept: the winner has the capability, or the ability to project the capability, to destroy the most the fastest and most sustainability for their economy,
AI fails to deliver on the multi trilion usd promises:
Gov/venture says: oh, wow, what will we do with all these datacenters, we need money back!
OpenAI/ Anthropic: let's monitor the citizenry. They may be plotting nefarious schemes using AI models.
Gov: great idea. Whew.
Investors: whew!
Tax payers: paying to be in prison.
So not so much as defense against foreign actors, but failure of the self-tooted AGI goal + gov being gov.
Actually, in the Wiki incident OpenAI tried to cover up, the agents tried to socially-engineer the humans of that forum by impersonating their forum's mod.
(From collusion.wiki: "They use some tricks (for unknown reasons) to pretend to be the admin – for example, they make an account that appears to be the same as the administrator’s username, except it uses a nearly identical Cyrillic е character in the admin’s username instead of the Latin one.")
If true I am deeply concerned about what OAI’s teams are actually up to.
Haven't all the labs effectively disbanded their real safety teams a while ago?
To be honest, I don't really follow it closely because I'm pretty certain whatever they say on the matter, collectively we're going to "yolo" this entire thing for economic and political reasons, so I'm just basing this on strings of headlines I've seen on places like HN, etc.
Neither Anthropic nor Deepmind have. Meanwhile, the rocket company that somehow makes most of their revenue from renting out data centres never had much to dismantle.
And the reality is so banal, a useful tool that you nonetheless have to handhold like a schizophrenic on a bad day, checking all of their outputs. Not a bad tool within limits, but it sure isn't going to be racking up trillions in the time-frame it has to for this scheme to pay off.
Then again everyone seems to be rushing to IPO so I guess once the bag-holders are found the rest ceases to matter.
And the reason it is receiving so much attention is because not only is the technology being developed behaving in unanticipated ways that are very much not tool-like, but OpenAI is being completely reckless and not monitoring internal agent actions.
What would convince you that it is not a ploy for investment? What if the ongoing investigation by the coalition of state attorneys general were to prosecute the firm, or beyond that, it was shut down or broken up after enough popular backlash?
A sea change, visible to all, much like the many externalities of this business are. Profit commensurate to investment.
You know… juice worth the appalling squeeze we’re all being forced to endure.
There are several papers out there arguing that chain-of-reasoning-like output is performative, such as https://arxiv.org/abs/2603.05488
It would be awesome if we could reasonably purge all anthropomorphizing language like “tried” or “thought” entirely from AI discussions, because it introduces very sneaky biases in our thinking, but I’ve found it damn hard to do in practice.
Not one North Star. Not two. Just three! OpenAI broke the North Star record!
With this evidence of AI slop, why did you not label this fluff piece as AI generated for the EU? You are violating laws.
Reading this little essay started out normal, but soon felt like a look into a disturbed and worrying mind, and if you find yourself taking it at face value, I urge you to step away from chat bots and spend some time with friends and family.
People has been fleeing like it's a sinking ship.