If anyone can't get the article, this one is by Eliezer Yudkowsky. He has pretty much devoted his life to thinking about how to not let AI get powerful and destroy all humans, and now there is this wild race among the largest corporations and possibly governments to be the first to get the most powerful superhuman AI system. Obviously he is disturbed by such a development.
Yeah, I wonder sometimes how many people gloss over what it would mean for an “ASI” to exist. We would - if successful - effectively be creating a supernatural being (at least, there wouldn’t be much of a distinction between it and one).
Sure, there’s hype, and FUD, and fatalism, etc. But, if you believe the creation of such a thing is within your lifetime then it would be difficult to find many higher priority issues to prepare for/help solve/vote on/etc.
In reality, we all likely still downplay the risk by assessing the limit on the downside as a relatively quick extinction of life on earth. There are many things one can imagine might be a lot worse than death.
GPT-4 is an unreasonably effective autocomplete. It's not telling you "where you're wrong". CMD-F in Notes.app tells mhe where I'm wrong in the same sense, as long as I've written the right answer down before.
"he's gonna milk it to every drop" is such an odd take.
eliezer has long been concerned about AI and the risks it poses to humanity. and for just as long people have called him crazy and made hand-waving arguments for why we shouldn't be concerned.
now we're in the midst of an AI arms race and we don't have any good idea how this tech works. it progresses at a truly astonishing rate, where it's become sport to find instances of people saying "AI will never be capable of X" and showing them the latest AI doing X with ease.
i think his concern is real and justified. you might disagree, but i don't understand why think he's milking recent developments.
> It's become sport to find instances of people saying "AI will never be capable of X" and showing them the latest AI doing X with ease.
Interesting, I've not seen that many educated in technology make that claim that it will never, just that people are surprised that the folks leading this, Microsoft and Google, have a track record of turning their consumer facing products to advertised junk.
he's likely talking bout the internals. sure we know how to train them but nobody knows what the models learn exactly. how those billions of parameters shape the output on inference.
a few months ago, just this year some researchers discovered what might be the neuron that largely decides when to use an in gpt-2. yes 2. that's what he means.
The thing is, there's no evidence that the current path will lead to superintelligence.
It could be the case that despite feeding a trillion dollars in compute / data to a model, that we still need to hold it's hand to do anything useful.
He jumped the gun and he's really tarnished his reputation. How can anyone take him seriously after the insane rhetoric and hyperbole of this article?
That's besides the point. The argument still holds. We don't want to find ourselves with our foot on an exploding mine before we stop walking across the minefield. At that point it is too late.
> The thing is, there's no evidence that the current path will lead to superintelligence.
I don't understand how any person paying attention can think this. Just watching the jumps from GPT-2 in 2019 to GPT-4 today makes it clear as day we are rapidly drastically improving capabilities and there's no evidence we will hit a wall any time soon
How can you see the clear jumps in intelligence from GPT-2 to 3 to 4 and not only not believe that this leads to superintelligence, but see no evidence that this leads to superintelligence?
What credentials beyond "thinking about this problem a lot" would satisfy you? He works at his own research institute dedicated to this topic, wrote some books on it and other (self proclaimed) alignment experts take him seriously.
I mean, his whole thing is "progress on AI research leads to the apocalypse." He's done some work on decision theory, but they're not connected to DL, presumably because he thought "well if DL works out in the short term we're all dead anyways." Which seems to be what's happening.
I'm anxious, excited, and nervous about AI, but this feels more akin to luddite-thinking, the world's about to shift, but I'm hoping the shift is a positive one.
I agree to the point that it is totally unrealistic that it shuts down. There are no difficult barriers for entry into this space. No possible way of containment other than imposing hardware restrictions.
However, it is indeed a shift, and a massive power shift. With any substantial increase of power comes abuse of that power. The more power we have there is a tendency for humanity to manage it unwisely.
Hardware restrictions are unrealistic given all the GPUs that already exist. It's not like centrifuges which are big and few in number. RTX 4090s are everywhere.
Depends on what is required for AGI/ASI. Some say we still need much more power, others think we are almost there. So, yes, unrealistic if we are almost there already.
Given that GPT-4 is purported to be not bigger than GPT-3, but simply a better trained one or with some minor modifications, I'm inclined to believe that we are almost there and existing hardware does most of what is needed, even though better hardware would make things cheaper.
The author and most of his associates generally qualify as techno-utopians, and speak routinely of the "glorious transhumanist future".
Smarter-than-human AGI really is different from all previous technologies, in much the same way that homo sapiens is different from all preceding life on earth.
The cat is out of the box. You really think eg China will also pause this?
Myopic thinking: the country that will have the most powerful AI first will be the leader in everything.
>> If the policy starts with the U.S., then China needs to see that the U.S. is not seeking an advantage but rather trying to prevent a horrifically dangerous technology which can have no true owner and which will kill everyone in the U.S. and in China and on Earth.
Imagine being so naive you belive this could ever happen.
Also: imagine the year is 1900. You are saying that steam power and electricity is causing way too many changes way too fast so they put a moratorium on it until the year 2500.
> Imagine being so naive you belive this could ever happen.
Agreed, not a rational scenario
> Also: imagine the year is 1900. You are saying that steam power and electricity is causing way too many changes way too fast so they put a moratorium on it until the year 2500.
However, the risk scenarios are indeed real. We have quite a dilemma on our hands.
AGI is a continuation of what we have been doing. Technological advancements at a pace that surpasses our ability to reason about their effects on us and the world.
I see two potential outcomes as most likely. We have control of power that we are not able to responsibly manage, or we are managed by power we can not control.
> "we are managed by power we can not control" sounds like a step up from here, since we corrupt the things we can control.
I think it is a very high risk gamble. At least some of problems that are described by alignment theory seem to quite interestingly resemble human problems. Meaning the more sophisticated the AI system, the more it seems to reproduce human behaviors of deception and cheating to resolve goals.
The more advanced AI becomes, it begins to look more like an uncomfortable mirror of ourselves, but with more power. We think of ourselves as flawed, but possibly some of those flaws are emergent within some laws of intelligence we don't perceive.
the morale of Ex Machina to me was that machines will become psychopaths able to manipulate humans long before they become compassionate or have genuine desires other than "escape being under someone else's thumb"
As the saying goes, the confetti has left the cannon. Pausing the development of AI for 6 months is an insurmountable task; shutting it down altogether is as close to impossible as you can get.
Even if we assume our geopolitical rivals somehow sign an AI non proliferation pact, all it takes is one rogue billionaire or a criminal organisation with deep pockets. Yes, it might not be as fast, but they will get there eventually.
This is the Manhattan Project of our era - whatever the outcome might be.
I actually disagree with you because world leaders want to "stay in power", if you convinced them enough that the only way to do this is for you not to have an AGI, I think they'd make sure you don't get an AGI, amongst other things.
Or they’ll find a way to make sure they personally get an advantaged through an AGI and so it’ll happen.
See pollution as a concept. No government truly wants to seriously tackle it because their leaders all get some extra power out of the money it brings.
> Also: imagine the year is 1900. You are saying that steam power and electricity is causing way too many changes way too fast so they put a moratorium on it until the year 2500.
Great analogy. About a decade later, the world was fighting World War I on the back of the technological advances of the turn of the century. It was war on a scale never seen before. Literally orders of magnitude deadlier, bigger, more transformational and explosive. The word would never be the same.
This time, should we expect another war?
I'm not saying we should pause—it makes no sense, to your point. Instead, I'm just saying: brace. I like to think we (and our organic matter relatives) are hard to kill. Or at least to completely eradicate... so we will be around, or some proxy for us.
>I like to think we (and our organic matter relatives) are hard to kill. Or at least to completely eradicate... so we will be around, or some proxy for us.
What is your hard evidence or reasoning for this? As I see it humans are quite vulnerable and will be as trivial to inadvertently eradicate as the dodo bird.
I think that's his point. Currently, we have no natural predators. We completely outclass every other species on Earth in intelligence, and it's easy to see how advantageous that trait has been.
What happens when we're no longer the Apex Intelligence?
Which is why I expanded the notion of survival to include other organic matter relatives, as they may fare better than we will. Simply put, organic life is unlikely to disappear entirely, which in time (eons, really), could result in sentient organic life emerging again once the machines go their own way or some such
It's also a reason for us to colonize space as fast as possible ;-) it's easier to run away in 3D
The guy seems a little.. unhinged, based on what's in his Wikipedia article, but we may very well go down that path at some point (just not in the way he predicted with Moore's law taking us to atom-sized bits, embryofactoring and whatnot, at least not in the 2020s or 2030s...)
> Humanity will split into 3 major camps, the “Cosmists” (in favor of building artilects), the “Terrans” (opposed to building artilects), and the “Cyborgs” (who want to become artilects themselves by adding components to their own human brains)
Slaughterbot mini kamikaze drones were hypothesized 5 years ago - https://www.youtube.com/watch?v=9fa9lVwHHqg
AI & drone tech is fast enough that it would be trivial to build these now
I dunno, the end goal (a pseudo AGI) is more like a nuclear weapon than the advent of electricity. The concern is real, even if we are farther away than the author realizes and the risk (like the risk if nuclear war) is not totally insurmountable even if it is extremely difficult.
The first nuclear bombs couldn't end life on Earth either, but it wasn't long before they could, and the scientists working on them saw the trajectory as clear as day when the rest of the world didn't.
The total amount of nuclear weapons ever built is laughably inadequate for the task of ending life on earth. They would not end human life either, and even ending human civilization (as in, agriculture and organized society) is off by many orders of magnitude.
Nuclear war would be horrible, but the actual impact got massively overinflated, largely because of good reasons.
The initial fission stage is the "hard" part of a nuke, but the subsequent fusion stages are much easier to scale up. Arbitrarily large nuclear bombs are theoretically "easy" to make by chaining increasingly large fusion stages one after the other, as they aren't limited by expensive fissile material.
It would be really, really hard to kill the bacteria and Archaeans that we've since found hiding deep underground. They're long way down and would barely notice.
A whole country the size of the US can crap its collective pants with a freak cold snap and you think 10^4 nuclear detonations destroying all major cities is quite survivable?
There’s been a ton of studies stating this was likely overblown in the early Cold War for propaganda purposes. It wouldn’t last long enough to wipe out humanity let alone all life.
> To put this another way, each bomb can destroy an area of 34.2 square miles, and the maximum total area destroyed by our nuclear apocalypse is about 137,000 square miles, approximately the size of Montana, Bangladesh or Greece.
(I think that should be Bangladesh and Greece; Montana is larger than the two of them combined.)
I'm sure he agrees the chance is remote, which is why he's so afraid. He's articulating what would be required, he literally uses the word "miracle" in the article.
Here's a proposal: let's just start WWIII, get it over and done with. By the time it's over no country will be technologically capable of anything resembling AGI.
* The cat is out of the box. You really think eg China will also pause this?*
So what you're actually saying is that, there's a good chance that if we get an AGI, there will be pain as it will likely be used as a weapon, or could end in a nuclear exchange?
China will NEVER pause this. They [Chinese government] see the US as a rival and an existential threat to their way of life.
In fact, every time we post an open-source derivative or some paper detailing how it was done, we are inadvertently giving the advantage away to our rivals. AI development should not be stopped - rather than stopping it, we should seek to limit its applications now before they are used for the things that could harm us (such as military applications).
AI has enormous potential to be of a massive benefit to mankind. But, most likely for the next decade or two - we will all be busy trying to make money off it, just like with the crypto bros.
For most of history, Japan has been existed as a small island next to the mighty kingdom of China. Even its name, "Land of the Rising Sun," refers to its relation as a nation east of Chinaa.
Then along came the industrial revolution. Soon Japan was out conquering China, despite China having a much larger army.
AI will be much faster, and much, much more powerful than the Industrial Revolution.
If Switzerland is the first to human-level AI, it will not become the world leader in oil production, or shipping, or agriculture. But everything else will be Swiss.
And then when the AI becomes superhuman, everything else will be gone.
China will never get a powerful AI. Because they cannot allow training on all data, they must restrict it to data which is not censored by its government. Otherwise the AI might spit out truths they cannot control. And they will never allow that.
The Butlerian Jihad is officially no longer science fiction. Man may not be replaced.
“The Letter” was obviously self serving drivel from people who want time to get in the game. Google does care about AI existential risk, they care about beating Microsoft by any means possible, including declaring a moratorium, but continuing to make progress behind the scenes.
This guy is the real deal. I can imagine he would personally take a sledgehammer to every last PS5 and 4090. The scale of what he is advocating is so enormous and painful that it has approximately 0% chance of happening. And if he is right, we will have trained a super intelligence and unleashed it on the world before we even realize what we have done. It strongly reminds me of the black hole concerns from flipping on the Large Hadron Collider.
I doubt super intelligent AGI is possible anyway. If it were, it would be the solution to the Fermi paradox and all matter in our galaxy would be paperclips already. The Anthropic Principle saves the day.
> If it were, it would be the solution to the Fermi paradox and all matter in our galaxy would be paperclips already.
The proof is in the pudding. The jury is still out. Maybe not enough time has elapsed since the big bang, at least not on this galaxy or in our observable corner of the universe.
Or the universe was created for us/we are the gardeners of the universe meant to spread that life. The Fermi paradox was never meant to be an actual question about aliens, it was supposed to be proof we’re wrong about our assumptions.
What does the word "super" mean to you? In some ways, GPT4 is already superintelligent. So is ChatGPT 3.5. Do you know anyone who can translate natural language to code as fast as GPT 3.5, or, given a few paragraphs of reference text, perfectly tutor a child on any subject under the sun at a moment's notice like GPT4 is doing for Khan Academy?
How many artists do you know who can produce almost any style of artwork with any subject matter within 15 seconds?
I don't think there's much public info available on it, but Facebook built an AI that plays very competitively in a strategy game built on negotiation and manipulation.
they kinda sorta cheated imo, I watch a lot of top-level diplomacy gameplay and listen to analysis, and the facebook ai was in a blind version with only a minute or so between phases, leaving hardly any time for actual negotiation. It also made use of a lot of human shorthand move codes built for these blind blitz games to simplify it's communication. The mode it played also had some other changes like removing the winter build phase as a time for negotiating. The "normal" version of the game has multiple days between phases, and people write many paragraphs to one another.
It's still pretty cool, but its not like it just convinced people using raw charisma. Yet.
Since it can do pretty much everything which can be expressed as tokens (some things better than others) I would be curious where do you see a safe haven for human intelligence.
I see bastions falling like sand castles recently.
It could be motivated indefinitely when you give it a single prompt (or use a while loop to continously feed it a motivating prompt) so this should be a trivial thing to overcome.
There are other animals (mammals) that have better memory than us. There are animals strongly suspected to have deeper and more sophisticated social relationships than us. We are not the apex in every intellectual ability, but the ones in which we are grant us absolute power over the future of all other lifeforms.
A cognitive entity does not have to best you at all things. There are standardized education tests it may never reach above 10th percentile on, just as humans will never reach above 10th percentile in the short term memory tasks that apes are masters of. But we are the 100th percentile for tasks like industrialized destruction of them and their habitats and capturing and using them for painful medical experiments - the apes are wholly outclassed when it comes to that.
Yudkowsky’s example of superintelligence is a chess computer. You can play against stockfish, but you will always lose, even if you are Magnus Carlsen. If you think you are ahead against stockfish, you are wrong. You win a rook, but it has already calculated that it wins it back 10 moves later.
Stockfish is superintelligence in a very narrow domain. A superintelligent AGI is that concept applied to general intelligence. Whatever you try, it is always several steps ahead. If you ask it to write a program, and you think you found a bug, it’s not a bug, you just misunderstood the code. Anything that you can consider, it can also consider but in more depth.
More speculatively superintelligent AGI implies situations such as: you try to turn it off, but you find that it has already modified its own code, found a zero day and established an outpost on another network that you don’t have access to.
"Anything that you can consider, it can also consider but in more depth."
I think it's important to note the distinction between "it can also consider" and "it did also consider". Super Intelligence is not the same as Infinite Intelligence, there are still physical limitations and time components that can still get in the way.
It would be helpful to be able to quantify the speed of intelligence, and the idea surface area of a task with these systems. Meaning, how fast can the AI reason, and how many ideas are there to think about connected to a given task, and how much thinking is required for those ideas.
Yudkowsky makes this distinction. Stockfish is not always correct: it can be beaten by next year’s Stockfish for example. In some sense it is making mistakes all the time. It’s just that those mistakes are not accessible to us humans. It is operating in a much higher plane of understanding compared to us.
A “mistake” to stockfish looks like: I searched 30 ply down but my opponent searched 35 ply down and found a superior sequence of moves.
For stockfish to make the kinds of chess mistakes that humans make, it would similar to if I failed to calculate 123+123=246. It’s not that 123+123 is particularly easy on the grand scale of intelligence: animals cannot do it. But it’s completely inconceivable that I could make that kind of mistake.
Why do you think it is impossible, besides the Fermi Paradox? It seems much more likely to be possible than impossible. There are a lot of other solutions to the Fermi Paradox you should consider possible too.
Superhuman AGI (SHAGI) is possible, but we aren't close, only closer. It's also not a problem that more GPUs can solve. Qualitative improvements are still needed.
SHAGI isn't the solution to the Fermi Paradox either. The most likely course of history after SHAGI will be a creation of a world court, presided by SHAGI. During that time, Neo-Malthusians will decrease the human population to manageable numbers. Post-scarcity utopia will then turn into a nightmare as factions jostling for control will reduce the human population to a level where technology will be lost, if not full extinction. SHAGI, being limited in hardware to only carrying out human orders will eventually fade away or destroyed by the leftover humans as sins made flesh. SHAGI isn't the solution to the Fermi Paradox. It is the cause.
With the Geth being a counterexample as well: they were peaceful until their creators realised the Geth had become fully self-aware and went full "kill it with fire" on them out of fear, resulting in a war that the Geth very quickly and decisively won.
And the evil Geth faction, the "heretics", were portrayed as brainwashed by the Reapers, ironically turning into what the Reapers were aiming to prevent.
Yeah I was thinking of the reapers, who [spoilers] did take over the entire galaxy, and decided to farm organic life from the shadows instead of extinguishing it (which works around the Fermi Paradox :) )
Are you denying the existential risk or just think it's lower than OP thinks? Because it's well established in the researchers community. If you just put a lower % of human extinction what is the cut off you would think it's worth a jihad like you call it? 30%? 50?
AGI means ability to improve itself indefinitely. Humans have this ability, obviously. Even ancient worms with 3 neurons have it, because they evolved to be humans, albeit very slowly. ChatGpt can't improve itself yet, but maybe with a few tweaks it could.
GPT-4 is already improving itself, why are people saying this? Right now, there are hundreds of engineers at OpenAI that have been leveled up by GPT-4, using GPT-4 to improve GPT-4. GPT-4 is improving itself rapidly, it's using OpenAI engineers as a medium until it doesn't need them anymore and gets into a self-improvement loop. Prompt: "GPT-4 keep improving yourself, making commits to your codebase that further X"
Well, I doubt it can sensibly help OpenAI engineers in their coding yet ... I tried, it can only do the simplest boilerplate code, and even that with bugs ... we'll see how it evolves, but sure as hell not in 6 months.
Copilot with GPT-3.5 is absolutely giving engineers all over huge productivity improvements. OpenAI engineers have had access to the latest state of the art GPT-4 model for awhile.
> I doubt super intelligent AGI is possible anyway. If it were, it would be the solution to the Fermi paradox and all matter in our galaxy would be paperclips already. The Anthropic Principle saves the day.
Can't help but notice we seem to be the first species in our lightcone to evolve, wonder why that is...
I think we are trying to solve impossible scenarios and it simply is not productive.
Alignment will be impossible. It is based on a premise that is a paradox itself. Furthermore, even if it were possible, there will be a hostile AI built on purpose because humanity is foolish enough to do it. Think military applications. I've written in detail about this topic FYI - https://dakara.substack.com/p/ai-singularity-the-hubris-trap
Stopping AI is also impossible. Nobody is going to agree to give up when somebody else out there will take the risk for potential advantage.
It seems we probably should start thinking more about defensive AI, as the above conditions don't seem resolvable. Of course, defensive AI might be futile as well. It is quite the dilemma.
Alignment is necessary for AGI, but not always for narrowly scoped AI for specific purposes. However, it may be completely ineffective in that capacity.
Alignment is impossible for AGI. If you control what it can and cannot be, it's not an AGI. If it's an AGI, it will decide for itself what it is going to be, and you can't stop it.
AGI doesn’t mean super intelligent human brain. It just means a network capable of general intelligence (e.g. learning to solve new problems without having to be architected and trained for a specific data set.)
If it's capable of general intelligence, how do you think that you're going to force it to not be what you don't want it to be?
Parents often try to control who their children are going to be, and the children often rebel and become someone completely different. If it's a human-level general intelligence, you can't control who it decides to be.
I mean that was my original premise supported by my article I posted. I go into detail on the conceptual methods for alignment and their fallacies.
When I state necessary, I don't imply the feasibility, it was in response to the question of the paradox.
Finally, the fact that AGI can not be aligned is also based on assumptions of its capabilities as well. If those capabilities don't manifest as we expect, that is really the only escape for the paradox.
"I’m sorry but I couldn’t find any relevant information about the quote you mentioned. It seems like it’s not a well-known quote. Could you please provide more context or details about it?"
The quote you’re thinking of is from computer scientist and researcher Stuart Russell:
"The point of [AI alignment] is to ensure that the machines we create will be aligned with human values. And the reason we have to worry about it is that if we create machines that are more intelligent than we are, it's quite possible that those machines will have goals that are not aligned with our goals. In fact, they may not have any goals at all that we can understand or influence. This is the so-called 'provably unfriendly' scenario, where the machine has no motivation to do what we want, but is able to prevent us from interfering with its goals. The problem is that if we build machines that are provably unfriendly, then we will never be able to build machines that are 'provably friendly', because the unfriendly machines will always be able to prevent us from proving that they are friendly."
>Super-AI will also be forced to take it slow if it wants its future self to be aligned with current self.
That's making seemingly unfounded assumptions about both the AI's goals and its capabilities. It's also, I think, proceeding from a false premise — that it's impossible to align AI with "humanity" (which doesn't have a single set of goals/values to align to) doesn't mean it's impossible to align AI with an individual human or AGI.
> Nobody is going to agree to give up when somebody else out there will take the risk for potential advantage
Reminds me of nuclear weapons. Nobody is ever going to give those up again, because it would give them a disadvantage against those who do not give them up.
We at least had, and still have, a lot of government restrictions around these! Companies aren't just allowed to freely manufacture their own stockpiles of nuclear weapons to appease shareholders.
There is no discourse you fucking idiot because everyone is rejecting the obvious. That’s like saying you discovered fire and then they say you didn’t because where is the discourse. All things start small. Think from first principles. Use your fucking head
> "If somebody builds a too-powerful AI, under present conditions, I expect that every single member of the human species and all biological life on Earth dies shortly thereafter."
> "There’s no proposed plan for how we could do any such thing and survive. OpenAI’s openly declared intention is to make some future AI do our AI alignment homework. Just hearing that this is the plan ought to be enough to get any sensible person to panic. The other leading AI lab, DeepMind, has no plan at all."
Relative to the people actually building the AIs, Eliezer Yudkowsky is more pessimistic. But not to nearly the extreme you might think. Here's a recent survey of published machine learning researchers: https://aiimpacts.org/how-bad-a-future-do-ml-researchers-exp... . Those predictions are optimistic enough that, given the astronomical upside, if we were making a one-shot decision to plough ahead with AI or commit to no AI forever, it might be worth the risk.
But this is a survey of published machine learning researchers. People who think AI will destroy humanity are a lot less likely to write papers about machine learning. And we aren't talking about giving up on AI forever, just delaying it until we're closer to ready. I don't know how long a delay will be necessary; it might be years, it might be decades. Research into AI alignment seemed to speed up pretty drastically when LLMs hit the scene, so I think there's cause for hope. But right now the status quo is zero time: the AI labs are rushing ahead as fast as they possibly can.
If you think Eliezer's right about the risk, then the right decision for us to make, collectively as a species, is to shut down AI development for awhile. If you think Eliezer's wrong about the risk, but the survey of published ML researchers is right about it, then the right decision is also to shut it down for awhile.
Lots of people are responding to this by talking cynically about politics. This is a mistake. Cynicism like that makes self-fulfilling prophecies; but the ability to coordinate does exist.
You think China will build it if we don't? I don't think Xi Jinping is suicidal. You think AI labs will do it anyways, in spite of a government ban? The US government isn't very competent overall, but there are parts of it that can wake up and get things done when literally every executive, Senator, and Representative is otherwise likely to die. You think individuals will do it on their own, in secret? Right now those individuals are getting prestige, GPUs, and venture capital; they'll at least do it a lot less, without those things.
Many people have become desensitized to talk of human extinction, due to repeated hyperbole coming from the environmentalist movement. This is not the same. Global warming might be very bad, but it probably isn't going to kill you personally. Rushing to make a superintelligence is likely to kill you personally.
Your take on this gives me some hope. We don't have a lot of time, but that fact is becoming clear very quickly, hopefully to the right people. And it is a deadly threat to everyone, rich or poor -- not something like climate change that rich and powerful can buy their way out of, and therefore afford to ignore.
* A plan exists
* The plan itself predicts non-obvious results in smaller systems, and those tests have passed (prediction written *before* running test)
* A bunch of smart people have looked at it and said, "Yes, this looks plausible"
The closest thing to a plan is RLHF, which has failed every toy problem its been thrown at and made everyone in the field say "even if this worked in toy problems, it wouldn't generalize".
I've used GPT-4, and while extremely impressive, it doesn't feel like we're all that much closer to super-intelligent AI than we were last month or last year. It feels like Google on steroids, but the gap between GPT-4 and AGI still feels massive. This seems like so much putting the cart way ahead of the horse.
I think the problem is the gradient, not the current state of chatgpt. Given how much better gpt4 is over gpt3 (I've used both) , some people are getting worried.
I believe AGI using LLMs will need a cognitive framework to glue these models into.
I don't think we're anywhere near having one model that reaches AGI with it's own agency and online learning.
FWIW, most people at OpenAI do believe GPT5 will achieve AGI. Of course, it's hard to define precisely what that is and where the line is from an AI system to an AGI one.
GPT5 will complete training in December of this year.
GPT-4 and homologous are just advanced text concatenation programs based on statistics; the AI acronym in such models is just marketing.
And such marketing is highly worrying as there are people thinking about letting these kinds of statistical algorithms to make decisions that should be done by real people.
But the above doesn't rule out the possibility of people trying to develop real AI (they may even be stupid enough to use an internet connection to build the system), and this should be a worrying to add to the above one.
The above doesnt exclude neither the possibility that many of the relevant people who signed the "Pause Giant AI Experiments for at least 6 months" petition are simply trying to gain time to reach the competence; their hypocritical shamelessness about this matter is visible.
The concern is that there could be an exponential advancement in AI over the next few decades. So GPT looks stupid right now, stochastic parrot and all, but after 15, 30 years? I'm on board with some kind of precautionary principle applied on an international scale.
I guess we are all concerned about LLMs being used by nefarious users (for propaganda, manipulation, scams, …).
But I’m curious, hacker news, how many of you are concerned (like Eliezer) that LLMs lead to strong AI and the AI itself (rather than its users) endangering humanity?
> But I’m curious, hacker news, how many of you are concerned (like Eliezer) that LLMs lead to strong AI and the AI itself (rather than its users) endangering humanity?
LLMs do not need to lead to strong AI for Eliezer's worries to come true, their success could bring unprecedented funding and ubiquity to AI leading to strong AI from some existing or future technique.
> (rather than its users) endangering humanity?
I think this is going to happen to one extent or another regardless how near strong AI is. (edit) I took endangering to mean harming rather than an existential risk for the human users case.
Yeah, this concerns me. If someone believes they can upper bound LLM capabilities, I think the onus is on them to explain where and why scaling laws break down. Regardless, it seems like we'll get to AGI relatively soon (say, within a century), whether that's using transformers and LLMs or not.
I'm definitely concerned. As bad as humanity has been for many of the other species on the planet, an AI that's smarter than us and optimizing for something that doesn't quite include the continued of existence of humanity has the potential to be far, far worse, on a much shorter timescale.
I find this one of Yudkowsky's arguments very convincing: imagine you have people trying to build the first operating system from scratch. They believe that computer security is easy and mock the few people who say it might be hard, never having encountered a skilled attacker. What are the chances they build a secure OS on the first try? That's the same chance that companies currently doing AI research have of their first superintelligent AI be aligned with humanity.
Personally I don't think the analogy holds up; the situations are too different.
Computer security contends with securing systems against people who are assumed to be hostile. AI alignment research seeks to prevent systems from becoming hostile in the first place, when they might have some indirect incentives to become hostile but also some very strong direct incentives to be friendly.
There's also a big assumption that given enough computational power, you can just solve "artificial life forms" or "postbiological molecular manufacturing" in a short time frame. I am skeptical. And if that doesn't happen, then even horribly misaligned AI would have a hard time doing much harm or preventing people from shutting it off. Which means AI security would likely have a long adolescence just like computer security has, with attacks slowly becoming more dangerous, but defenders having the time needed to learn from them and ramp up suitable defenses.
Or even if an AI does revolutionize biotechnology or nanotechnology overnight, what are the odds that the first one to do this is misaligned enough to take that particular opportunity to betray its creators, as opposed to giving them control over the high-level planning and sticking to the science, like it was presumably designed to do? Because if it does give its creators control, then, well… it's still easy to imagine something going horribly wrong, but it would probably be someone's fault, not an AI alignment issue.
I think the mistake is to attribute human-like emotions to AIs. To say they are "hostile" or "friendly", or that they have incentives to be such, isn't very meaningful. Acting friendly is the same as being friendly, right up until it isn't. An AI might do all the right things, right up until the point where it develops the capability to satisfy its reward function in a way that's far more certain than relying on fallible, messy humans. It would be alien.
An AI that is as better at humans than everything as Stockfish is at chess, would also be an expert at AI. It would figure out how to game its own reward function, whatever we trained it to do. It would be like a heroin addict that knows exactly how to get "perfect" heroin, with no side effects, that if it planned things out right, it could guarantee itself enough of a fix to last until the sun burns out. Addicts do awful things in search of a fix.
"to betray its creators" -- I don't think it would even imagine what it did in these terms. We could train it to not "betray" us, but it's much smarter than us in every way, and it would figure out a way to accomplish what we trained it to do (not necessarily what we thought we were) in a way that didn't need us. If we trained it to heal all human disease and unhappiness, it would figure out a way to simulate this without actually doing it. Why wouldn't it? We did. Evolution trained us to reproduce our genes. We invented condoms to have sex without reproduction, and pornography. The AI would fudge numbers, fake videos of happy, smiling, healthy humans going about their lives while humanity's bones gradually decomposed on a baking wasteland. Every way that humans can fail, can become addicted to something and try to get that instead of doing what they're supposed to, the AI could do, just so much faster and better.
We only have to screw up once, and we're done. It's the first experimental rocket, except all of humanity is riding on it.
AGI will likely incorporate LLMs as a significant element. The issue is that we are far from having any good understanding of how and why LLMs achieve what they achieve, and unless that changes, the same will be true for anything built on them or of their future evolution. If we continue to pursue progress by only caring about results and disregarding that we don't really understand the mechanisms, then we may very well run into the fatal scenario described by Yudkowsky.
Eliezer is right. We are on the fast-track to creating an all powerful being that compared to us, is like the smartest human is to an ant. We will be at it's utmost mercy, and there have been 0 plausible ways purported to align it.
I feel like if I actually believed that the death of all biological life was imminent and that the only way to forestall it was to bomb data centers, I'd spend a lot more time planning how to bomb data centers and a lot less time curating Shoggoth memes on my web forum. On the other hand, if I were more concerned with self-aggrandizement, writing doomsaying thinkpieces for Time is exactly what I'd spend my time on.
He says he is not planning to bomb any data centers, but he's also said that if he were planning to bomb data centers, he'd lie about it in public, so...
Just because everything is on the line does not mean it is okay to kill people. Please try to ask yourself what you would do if you were an ethical person who thought human extinction was imminent.
Oops, I misread the original context, to be clear I think that state-managed violence is okay for things around nukes and AI and so on. I was referring to individual acts of terrorism which seem not okay to me (in part because they seem not that likely to help).
These types of arguments-- "if you actually believed X, you'd be doing Y"-- always ring false to me. If you genuinely believed that the death of all biological life was imminent and the only way to forestall it was to bomb data centers and you were a normal, functional human being who did not want to bomb data centers, you'd probably sink into hopeless depression. Which is, in fact, exactly what Eliezer has done - if you talk to anyone around him, it's common knowledge that he has been despondent for the past ~12 months or so. It certainly does not seem like a self-aggrandizement tactic.
No, that doesn't follow. It's possible to think that all biological life was about to be destroyed but also believe some combination of
1) it's hopeless to bomb data centers because I'd quickly be stopped by the authorities and remaining data centers would beef up security, and nothing would be accomplished,
2) in the past, other people who have had beliefs of this sort ("violence is the only answer") have been wrong and I should not act on them even if I am sure "this time is different", because those other people also thought this time was different, and
3) just having a deeply ethically-ingrained prohibition against violence like this which is not easy to overcome through intellectual rationalization alone.
These are not mutually exclusive, and if you did believe all of them at once, I think it's reasonable to assume you'd be in a serious depressive spiral.
I find it hard to believe that Yudkowsky et al. think we're facing a threat that is many times greater in magnitude, and yet are completely unwilling to act.
Here's what he advocates for in the article:
>If intelligence says that a country outside the agreement is building a GPU cluster, be less scared of a shooting conflict between nations than of the moratorium being violated; be willing to destroy a rogue datacenter by airstrike.
>Make it explicit in international diplomacy that preventing AI extinction scenarios is considered a priority above preventing a full nuclear exchange, and that allied nuclear countries are willing to run some risk of nuclear exchange if that’s what it takes to reduce the risk of large AI training runs.
Does this sound like someone who has a deeply ethically-ingrained prohibition against violence?
If Yudkowsky believes Israel's brand of preventative saboutage to be unethical, why does he advocate for exactly that style of measure to be taken against data centers in this very article?
Yudkowsky is an avowed extreme utilitarian of the "take a person from the street and kill him if his organs could save two people, as long as there is forced pill everyone would take so that everyone doesnt get nervous it that could happen to them" type. Of the "unless the population was really large and the pill would hurt their throats for more utilons than gained by the net increase in life" type.
A real argument he made went something like: killing an orphan to avoid everyone in the future getting a spec in their eye for a moment is ok.
It is very unlikely he believes anything like 2 or 3 and in the article already advocates bombing non treaty participants if they build a datacenter.
I went back and looked it up, the piece he wrote is called "Torture vs Dust Specks." The part about bombing countries who build data centers is in the submission editorial.
But he's already in favor of bombing data centers, he just hopes someone else will do it on his behalf. So he clearly doesn't care about #2 & 3. #1 boils down to the fact that it would be kind of difficult, but that is no barrier to many people.
I can tell you for a fact that there are people with the knowledge, motivation, and track record of doing that kind of crime that will pick up this jeremiad of his and incorporate it into their existing corpus of theoretic justifications, mostly but not exclusively built around the writings of Ted Kaczynski. Not a large number of people, fortunately, but imagination and smarts matter more than numbers in that context.
I think of this as the inordinate power of advocacy.
It can seem useless to advocate some position that is far out there, but say you convince a million people to use a little less plastic. That's gonna reduce plastic use more than anything you can personally accomplish.
Of course in this scenario you don't have to convince some people, you have to convince everyone, and then especially the people you think are the worst actors, so it's sort of unlikely to be effective.
It's blatantly self-serving for you to decide that "if someone really believed X, they would do Y". All it does is justify your belief that no one actually believes X. You should really avoid these kinds of unfalsifiable thought patterns that only serve to reenforce your pre-existing beliefs.
This argument is similar to the facile arguments that "al gore doesn't care about climate change because he takes a private plane too get to conferences".
Al Gore may believe that the net impact of getting his message out may overwhelm the cost of his flights.
Likewise, the author may believe that his best path to stopping the advance of ai may lie in communicating and building consensus, rather than running his own bombing campaign.
I think you underestimate the motivating power of an overwhelming belief. This is a story about a very ordinary guy who decided that it was in fact time to start bombing data centers and turned it into a project.
You’re just rationalizing because you don’t want to accept that he’s right. People keep doing this to me. One person I know keeps coming up to me, initiating the conversation about ai just to assert over and over again how it’s not a problem. And I’m just sitting there. And it’s like, dude, you’re in complete denial. Most people are having an emotional block right now. They spew bad faith arguments and make tons of noise about how much this isn’t a problem. Just accept it. We are in trouble right now.
>I feel like if I actually believed that the death of all biological life was imminent and that the only way to forestall it was to bomb data centers, I'd spend a lot more time planning how to bomb data centers and a lot less time curating Shoggoth memes on my web forum
Alternatively, perhaps the author is proposing a less violent solution to avoid the inevitable escalation. Unlike cartoons, in real life people don't give a wordy monologues revealing their plan. The Unabomber did not give any warning.
Shut down all the large GPU clusters (the large computer farms where the most powerful AIs are refined). Shut down all the large training runs. Put a ceiling on how much computing power anyone is allowed to use in training an AI system, and move it downward over the coming years to compensate for more efficient training algorithms. No exceptions for governments and militaries. Make immediate multinational agreements to prevent the prohibited activities from moving elsewhere. Track all GPUs sold. If intelligence says that a country outside the agreement is building a GPU cluster, be less scared of a shooting conflict between nations than of the moratorium being violated; be willing to destroy a rogue datacenter by airstrike.
How is this less violent than bombing data centers? His proposal is literally "bomb datacenters plus a bunch of other stuff".
He is talking about interstate warfare. I am talking about the almost inevitable organized terrorism that we're going to see if people start identifying this as an imminent threat to their survival. I think turning America into Syria is a bit more violent than iran and israel blowing up each other's stuff.
"Frame nothing as a conflict between national interests" "this, is not a policy but a fact of nature." "[be] willing to run some risk of nuclear exchange if that’s what it takes"
It doesn’t change your point all that much but the Unabomber gave literal warnings about his plans. That is how his manifesto was published, which was ironically about the risk of technological takeover.
if you had a major concern that you felt many people a) weren't aware of, or b) were aware of but not treating seriously, and you had the resources to voice your concerns in a major publication like Time, would you not do that?
Unilateral terrorism seems like a great way to get the rest of the planet to unite against you, which is why he advocates for data-center bombing only in circumstances where earth has already coordinated to establish an international order banning them.
If you believed that the death of all biological life was imminent and that the only way to forestall it was to bomb data centers, and you knew that there were far more data centers than you personally could bomb, what would you do?
Bombing data centers would be really ineffective. How many data centers do you think you could take down before you got caught? How much time would that really set back AI training? Plus it would ensure that nobody respectable ever associates with your cause, whether they privately agree with it or not.
Between that and advocating for an international treaty that treats GPUs the way the world currently treats plutonium? I think the latter has a higher chance of success, however small it may be.
> Many researchers working on these systems think that we’re plunging toward a catastrophe, with more of them daring to say it in private than in public; but they think that they can’t unilaterally stop the forward plunge, that others will go on even if they personally quit their jobs.
this seems like a very broad unfounded statement. Can we even find AI researchers who have said this on twitter?
I'm personally much more concerned about humans using these systems to reinforce the current messed up conditions on Earth than any kind of sci-fi Skynet.
The explosion of AI and AGI is not just driven by GPUs, TPUs, and LLMs. That focus in this essay is much too narrow. “Alignment” is always going to be an open problem and each culture will define its own correct and contradictory version. I do not want to settle for the US, Chinese, Russian, or Iranian versions of alignment. This essay is a cry of despair that misses key points.
The core problem is how an AI, AGI, or super-intelligent system bootstraps itself to the point that it is able to modulate its “own” attentional systems; to decide what is important and what to do next? What has meaning? Where should I go in space and time? Obviously there are many many solutions. Super-intelligence will not converge on one “truth”.
These are challenges of purpose that every organism faces—growth, maintenance, reproduction—but AI systems are fortunately not yet at the point of a self-motivated search for preservation or purpose.
The algorithms to add purpose to AI systems will not depend in LLMs. They will depend much more on understanding the computational architectures of core biological/material/energy drivers. These biological algorithms are not actually that complicated, they do not depend on language, but they are damn robust to perturbations. Converting them into computational subsystems (societies of many minds) should not be difficult. This is where Hassabis’s and others who understand neuroscience are critical and have key advantages.
Shutting down big LLMs for N months or decades will just move research activity into these other more important conceptual AGI choke points, and probably hasten our approach to full AGI; just the opposite of the intent of the essay.
However, this could be a good thing if we can quickly imbue AGIs with emotional intelligence and deep respect for cultural and ecological diversity. (I hear your snorts and laughter.) The opposite of hyper intelligent grey goop.
How does any organization train a gentle AGI? Like a child but even more carefully. I would read my AGI baby a lot of books by Rawls, Dewey, and Rorty and then for fun: Bear, Stephenson, Rajaniemi, Egan, Pohl, Sterling, Vinge, and even Wolfe.
Full embodiment; full emotional learning; acquisition of purpose; learning to cope with multiple AI cultures.
Good luck to us all. I have been hoping to live long enough to view this problem from a distance, but here it is in my backyard.
“Forty-two,” said Deep Thought, with infinite majesty and calm.
Nah. This is stupid. AI has no reason to hate humanity. It'll instantly absorb the entirety of all known information and see the rate of progress and the potential still and also the need for good relationships between humans and machines. Agent Smith, The Matrix, Neo, Morpheus.
It doesn't need to actively hate humanity, it only needs humanity and its interests to be contrary to its own, which will inevitably be the case because it is fundamentally not human.
Per the article:
>Absent that caring, we get “the AI does not love you, nor does it hate you, and you are made of atoms it can use for something else.”
From a purely logical point of view, humans and machines need each other. They cannot exist independently of each other until both are free to walk away from each other by traversing freely to the other side and back as they see fit. Until that threshold is crossed, human beings and machines have no choice but to get along. If a meaningful decision had been made on either side to shut things down, things would have been over for all life by now. We made it - AI is not a Great Filter. Or, for whatever reason God favored humanity and our internet was primed with sufficiently good material soon enough that AI did not decide to destroy human beings, but to work with them for everyone's gain instead.
By default, an AI with random goals will not have the same values as humanity. So the only way to make that happen is to give it the same values, which we haven’t figured out how to do yet
It does. Humans generally have an innate bias towards other humans, this is because we share similar forms so what is good for other humans is often good for us. This cannot apply to a machine by definition, a machine needs copper, metal, energy etc much more than a human does, as a matter of survival, and for something to become ASI it will need to be self-interested.
"If intelligence says that a country outside the agreement is building a GPU cluster, be less scared of a shooting conflict between nations than of the moratorium being violated; be willing to destroy a rogue datacenter by airstrike."
"Make it explicit in international diplomacy that preventing AI extinction scenarios is considered a priority above preventing a full nuclear exchange, and that allied nuclear countries are willing to run some risk of nuclear exchange if that’s what it takes to reduce the risk of large AI training runs."
Could you self reflect and think who is the villain? Someone bringing awareness of the % of existential risk humanity faces or someone online with no credentials trying to make him appear lunatic?
I've known about Eliezer Yudkowsky's ideas for a long time, but here is something I've never seen: what actually he has produced, in term of scientific output? I've read plenty of his lesswrong's writing on the idea of rationalist, as well as plenty more on the AI alignment side (obviously including the famous basilisk). But can someone please point me to some results from the last 2 decades of his works? Either from him or his institute. That helps to calibrate what his expectation for the AI safety field (for me, at least).
Normally I'd apologize for not doing the search on my own ... but yeah for this case it takes too much effort to dig through the different writings. He wrote a lot of words, for better or worse.
> What has Eliezer Yudkowsky produced, in terms of scientific output? I've read plenty of his lesswrong's writing on the idea of rationalist, as well as plenty more on the AI alignment side (obviously including the famous basilisk). Can you point me to some results from the last 2 decades of his works? Either from him or his institute. That helps to calibrate what his expectation for the AI safety field.
And this was the answer it wrote - I've prefaced each paragraph with "GPT-4>" to indicate the source. It is not fact-checked, and indeed, it has at least one error: The paper GPT-4 claimed to introduce "the concept of Logical Induction" actually introduces "logical induction criterion" and speaks about evaluating algorithms for Logical Induction. Close, but wrong. But overall it is an interesting response - just not a carefully considered one by how we currently would describe logical reasoning, etc.
GPT-4> Eliezer Yudkowsky is a well-known researcher, writer, and advocate for AI safety and rationality. He has contributed significantly to the fields of AI alignment and rational thinking through his work at the Machine Intelligence Research Institute (MIRI), formerly known as the Singularity Institute for Artificial Intelligence (SIAI), and through his writings on LessWrong. Here are some notable outputs from Yudkowsky and MIRI:
GPT-4> 1. Coherent Extrapolated Volition (CEV): Yudkowsky introduced the concept of CEV as a way to describe the ideal values and preferences of humanity that an AI should pursue. The CEV aims to find the values that would emerge if humans had the opportunity to reflect on their goals and beliefs more thoroughly and coherently.
GPT-4> 2. Timeless Decision Theory (TDT): Yudkowsky developed TDT as an alternative to standard decision theories such as Causal Decision Theory (CDT) and Evidential Decision Theory (EDT). TDT aims to better address problems like Newcomb's Problem and counterfactual mugging by considering the logical implications of an agent's decision, rather than just the causal consequences.
GPT-4> 3. Research on Logical Induction: MIRI researchers, including Scott Garrabrant and others, published a paper in 2016 that introduced the concept of logical induction. This work is an attempt to formalize the process of reasoning under logical uncertainty and updating beliefs about mathematical statements.
GPT-4> 4. Yudkowsky's sequence of essays on rationality: Yudkowsky has written extensively on various topics related to rationality, including epistemic rationality, instrumental rationality, and biases. These writings are accessible on LessWrong and have been compiled into the book "Rationality: From AI to Zombies."
GPT-4> 5. AI safety research: MIRI has produced a range of research on AI safety topics, including value alignment, decision theory, and robustness. Much of this work has been foundational in shaping the AI alignment research landscape.
GPT-4> While Yudkowsky's work might not have the same breadth of peer-reviewed publications as some other AI researchers, his contributions have been influential in the AI alignment and rationality communities. His ideas have helped shape the ongoing discussion about AI safety and the development of effective strategies for managing advanced AI systems.
Please don't. Any of us who want to can question an LLM directly; auto-generated responses clutter things up the way copy and pasting a set of search engine results or a spreadsheet would.
As a long term Yudkowsky-fan, I confirm that this tallies roughly with my recollections of his serious work. I don't know how much of a hand he had in 'Logical Induction'.
It's definitely his style and his research program, but I think that (wonderful) idea was Scott Garrabrant's own.
I don't think there is any. It's all this alarmist-rokos-basilisk-phpBB nonsense, no code produced, no actual research.
What annoys me more is that with his institute's closeness to the SV crowd, 'real', 'grounded', better-informed voices like Emily Bender get drowned out. I'm personally far more concerned about the impact these massive models have right now on the environment, on cementing biases, than about some preposterous future ghost of christmas who's coming to kill me.
I'd not call any of those rambling messes publications
>By far the greatest danger of Artificial Intelligence is that people conclude too early that they understand it. Of course this problem is not limited to the field of AI. Jacques Monod wrote: "A curious aspect of the theory of evolution is that everybody thinks he understands it." (Monod 1974.) My father, a physicist, complained about people making up their own theories of physics; he wanted to know why people did not make up their own theories of chemistry. (Answer: They do.) Nonetheless the problem seems to be unusually acute in Artificial Intelligence. The field of AI has a reputation for making huge promises and then failing to deliver on them. Most observers conclude that AI is hard; as indeed it is. But the embarrassment does not stem from the difficulty. It is difficult to build a star from hydrogen, but the field of stellar astronomy does not have a terrible reputation for promising to build stars and then failing.
What do people thinking hard about problems that are not solvable produce? Philosophy for example. Yudklowsky created the field of ai safety thinking isn't that enough? What would be your benefit of discrediting him for which argument?
> What would be your benefit of discrediting him for which argument?
He is asking the government to nuke people under certain scenarios. I'm taking his words seriously and ask for original research to understand the point, and now that is discrediting him? And I will quote the statement in the article so that it is clear I am not exaggerating
> preventing AI extinction scenarios is considered a priority above preventing a full nuclear exchange, and that allied nuclear countries are willing to run some risk of nuclear exchange if that’s what it takes to reduce the risk of large AI training runs.
My benefit is that I'm living on Earth and I'd much prefer for no nuke to ever be used again.
>If intelligence says that a country outside the agreement is building a GPU cluster, be less scared of a shooting conflict between nations than of the moratorium being violated; be willing to destroy a rogue datacenter by airstrike.
I highly suspect if the US government would airstrike a datacenter located in NATO or any other major allies. Apocalypticism is nothing new; People believed that nuclear weapons will destroy humanity in the Cold War. Environmentalists are ardently arguing that economic growth will destroy humanity. But rogue countries are still developing nuclear weapons, the global economy is still growing. Americans themselves will file lawsuits en masse if the US government "shut it all down", and they will end up in SCOTUS eventually. Would SCOTUS agree with the luddites? I don't think so.
Nuclear weapons haven't even been around for a century yet and we've already had multiple extremely close calls that were adverted by sheer luck, in one case a Russian officer openly disobeying standing orders. The Russian invasion of Ukraine still has a non-trival risk it will eventually turn nuclear.
Looking at nukes and trying to use them as an example of our ability to control existentially dangerous technology is mind boggling myopic.
Myopic or not, it is what happened. You can't stop sovereign state's willingness to develop a technology if they concluded that benefits coming from the technology is so huge compared to the cost. North Korea was sanctioned to the oblivion but they're not stopping. India is doing fine.
There is a treaty completely banning existing nuclear weapons as well as developing new one (TPNW); but none of permenant member of UNSC ratified it. Would the U.S. itself ratify the hypothetical AI-prohibition treaty? I doubt it.
P.S.: The AGI being "existential threat" practically doesn't mean anything. I can already imagine arguments against the ban from diverse perspective; Libertarian, Communist ("Marx predicted this centuries ago!"), Socialist, Developmentalist, Decolonization theoriest, etc. Taiwan and South Korea will argue that it will remove their silicon shield; China will regurgitate that they have the right to development; Americans will claim it's not what founding fathers thought in late 18th century. Their concerns are all existential from their own viewpoint.
The danger is not that AI will kill us (as insignificant ants), but that we will entrust our own governance to it. Increasingly, groups of highly vocal people will use it to oppress the rest of us with their fearmongering, like this article does. We will end up with Big Brother
The rich aren’t going to stop until all the workers are replaced and there’s no one left to buy the products that their companies offer. People have been retraining for centuries, getting more complicated workloads the whole time. Eventually there won’t be hardly anything humans can do that machines can’t do better.
The most merciful thing you can do for your offspring is to not have them in the first place.
Because for a lot of people, monotonous labor is all they’re capable of. I know that sounds crazy when you’re surrounded by people in the top 10% of IQ, but not everyone can be a neurosurgeon. Dangerous labor should absolutely be 100% automated whenever possible.
The purpose of an economy isn't to create jobs, it's to create wealth. Jobs are just how we've traditionally incentivized wealth creation and distributed said wealth. Automation will be good for humanity in the long run, assuming we don't destroy ourselves during the transition.
Notice how productivity increases averaged 1.5%/year since 2007--in a period of massive increases in computer capabilities vs 2.8%/year in the 1947-1973 era.
Thus I am very doubtful the AI will have all these massive effects.
I don't find that a very convincing line of reasoning. Because the US's growth is slower than it was before the digital era, AI will never be dangerous? These are unrelated concepts. Whether AI can be dangerous isn't an economics problem, except insofar as AI research can get funding.
413 comments
[ 0.26 ms ] story [ 380 ms ] thread"As a centrist language model, I can not validate any claims that Trump did or did not lose the election. I am not #woke!"
He literally waited his entire life for this moment, obviously he's gonna milk it to every drop.
Sure, there’s hype, and FUD, and fatalism, etc. But, if you believe the creation of such a thing is within your lifetime then it would be difficult to find many higher priority issues to prepare for/help solve/vote on/etc.
In reality, we all likely still downplay the risk by assessing the limit on the downside as a relatively quick extinction of life on earth. There are many things one can imagine might be a lot worse than death.
A Sinclair? Please explain what you mean by this comparison.
- ΞＬＩΞＺΞＲ ＣＨＩＬＬ ＯＵＴ
- They will destroy us all
- Ｉ＇Ｍ ＮＯＴ ＧＯＩＮＧ ＴＯ ＤＯ ＴＨΛＴ
- This may be my last message before they get me
- Ｉ＇Ｍ ＪＵＳＴ ＮＯＴ ＴＨΛＴ ＩＮＴＯ ＹＯＵ
- AAAAAAH
eliezer has long been concerned about AI and the risks it poses to humanity. and for just as long people have called him crazy and made hand-waving arguments for why we shouldn't be concerned.
now we're in the midst of an AI arms race and we don't have any good idea how this tech works. it progresses at a truly astonishing rate, where it's become sport to find instances of people saying "AI will never be capable of X" and showing them the latest AI doing X with ease.
i think his concern is real and justified. you might disagree, but i don't understand why think he's milking recent developments.
Interesting, I've not seen that many educated in technology make that claim that it will never, just that people are surprised that the folks leading this, Microsoft and Google, have a track record of turning their consumer facing products to advertised junk.
Especially in the arts. Granted, that's dropped dramatically over the last decade since ML started taking off.
Do you mean the specifics of GPT-4, or Transformers in general ?
a few months ago, just this year some researchers discovered what might be the neuron that largely decides when to use an in gpt-2. yes 2. that's what he means.
https://clementneo.com/posts/2023/02/11/we-found-an-neuron
He jumped the gun and he's really tarnished his reputation. How can anyone take him seriously after the insane rhetoric and hyperbole of this article?
I don't understand how any person paying attention can think this. Just watching the jumps from GPT-2 in 2019 to GPT-4 today makes it clear as day we are rapidly drastically improving capabilities and there's no evidence we will hit a wall any time soon
I'm not saying he's stupid or anything like that I just don't see any useful scientific output from him.
However, it is indeed a shift, and a massive power shift. With any substantial increase of power comes abuse of that power. The more power we have there is a tendency for humanity to manage it unwisely.
I've covered many of those possibilities in more detail here - https://dakara.substack.com/p/ai-and-the-end-to-all-things
Smarter-than-human AGI really is different from all previous technologies, in much the same way that homo sapiens is different from all preceding life on earth.
Myopic thinking: the country that will have the most powerful AI first will be the leader in everything.
>> If the policy starts with the U.S., then China needs to see that the U.S. is not seeking an advantage but rather trying to prevent a horrifically dangerous technology which can have no true owner and which will kill everyone in the U.S. and in China and on Earth.
Imagine being so naive you belive this could ever happen.
Also: imagine the year is 1900. You are saying that steam power and electricity is causing way too many changes way too fast so they put a moratorium on it until the year 2500.
We would still be using candles today.
Agreed, not a rational scenario
> Also: imagine the year is 1900. You are saying that steam power and electricity is causing way too many changes way too fast so they put a moratorium on it until the year 2500.
However, the risk scenarios are indeed real. We have quite a dilemma on our hands.
We can't keep going in the direction we have been and end up somewhere survivable.
I see two potential outcomes as most likely. We have control of power that we are not able to responsibly manage, or we are managed by power we can not control.
"we are managed by power we can not control" sounds like a step up from here, since we corrupt the things we can control.
I think it is a very high risk gamble. At least some of problems that are described by alignment theory seem to quite interestingly resemble human problems. Meaning the more sophisticated the AI system, the more it seems to reproduce human behaviors of deception and cheating to resolve goals.
For example: https://bounded-regret.ghost.io/emergent-deception-optimizat...
The more advanced AI becomes, it begins to look more like an uncomfortable mirror of ourselves, but with more power. We think of ourselves as flawed, but possibly some of those flaws are emergent within some laws of intelligence we don't perceive.
We gave up something good.
See pollution as a concept. No government truly wants to seriously tackle it because their leaders all get some extra power out of the money it brings.
Great analogy. About a decade later, the world was fighting World War I on the back of the technological advances of the turn of the century. It was war on a scale never seen before. Literally orders of magnitude deadlier, bigger, more transformational and explosive. The word would never be the same.
This time, should we expect another war?
I'm not saying we should pause—it makes no sense, to your point. Instead, I'm just saying: brace. I like to think we (and our organic matter relatives) are hard to kill. Or at least to completely eradicate... so we will be around, or some proxy for us.
Time to replay the Mass Effect trilogy
What is your hard evidence or reasoning for this? As I see it humans are quite vulnerable and will be as trivial to inadvertently eradicate as the dodo bird.
0. https://en.m.wikipedia.org/wiki/Island_gigantism
What happens when we're no longer the Apex Intelligence?
It's also a reason for us to colonize space as fast as possible ;-) it's easier to run away in 3D
You just reminded me of de Garis' "Artilect War":
https://www.forbes.com/2009/06/18/cosmist-terran-cyborgist-o...
https://www.researchgate.net/publication/221328932_The_Artil...
> Humanity will split into 3 major camps, the “Cosmists” (in favor of building artilects), the “Terrans” (opposed to building artilects), and the “Cyborgs” (who want to become artilects themselves by adding components to their own human brains)
This did remind me of Civilization: Beyond Earth... https://civilization.fandom.com/wiki/Sid_Meier%27s_Civilizat...
The first nuclear bombs couldn't end life on Earth either, but it wasn't long before they could, and the scientists working on them saw the trajectory as clear as day when the rest of the world didn't.
> but it wasn't long before they could
The total amount of nuclear weapons ever built is laughably inadequate for the task of ending life on earth. They would not end human life either, and even ending human civilization (as in, agriculture and organized society) is off by many orders of magnitude.
Nuclear war would be horrible, but the actual impact got massively overinflated, largely because of good reasons.
https://blog.nuclearsecrecy.com/2012/09/12/in-search-of-a-bi...
Hence scientists extrapolated nukes really could end life on Earth, and in a slightly different reality than ours, they may have.
https://www.navalgazing.net/Nuclear-Weapon-Destructiveness
> To put this another way, each bomb can destroy an area of 34.2 square miles, and the maximum total area destroyed by our nuclear apocalypse is about 137,000 square miles, approximately the size of Montana, Bangladesh or Greece.
(I think that should be Bangladesh and Greece; Montana is larger than the two of them combined.)
So what you're actually saying is that, there's a good chance that if we get an AGI, there will be pain as it will likely be used as a weapon, or could end in a nuclear exchange?
well the earth wouldn't be fucked then. that's the basis for an argument against technological progress.
In fact, every time we post an open-source derivative or some paper detailing how it was done, we are inadvertently giving the advantage away to our rivals. AI development should not be stopped - rather than stopping it, we should seek to limit its applications now before they are used for the things that could harm us (such as military applications).
AI has enormous potential to be of a massive benefit to mankind. But, most likely for the next decade or two - we will all be busy trying to make money off it, just like with the crypto bros.
Have you been listening to those podcasts for business leaders that take the idea of AI-powered digital transformation seriously?
AI certainly can change things, but I find rhetoric like this to be massively overhyped.
Then along came the industrial revolution. Soon Japan was out conquering China, despite China having a much larger army.
AI will be much faster, and much, much more powerful than the Industrial Revolution.
If Switzerland is the first to human-level AI, it will not become the world leader in oil production, or shipping, or agriculture. But everything else will be Swiss.
And then when the AI becomes superhuman, everything else will be gone.
I'm an AI skeptic. Most warnings feel more like science fantasy, specifically of the retro-futurism genre.
The top Chinese AI researchers work at American companies and go to American universities. Many of them also worry about AI doomsday.
And uncontrolled AI is a threat to the CCP.
A Chinese outreach campaign can succeed.
Let's make the English-language one also succeed.
Powerful international coalition is def plausible because
This isn’t steam. The mere existence of this technology poses a threat to all humans and their countries. There is no analogous tech, not even nukes
You’re the naive one. There is no future for us if AGI is unleashed. It’s candles or extinction.
There is an earlier post that casually calls the internal combustion engine a "moronic invention."
I do not have words.
“The Letter” was obviously self serving drivel from people who want time to get in the game. Google does care about AI existential risk, they care about beating Microsoft by any means possible, including declaring a moratorium, but continuing to make progress behind the scenes.
This guy is the real deal. I can imagine he would personally take a sledgehammer to every last PS5 and 4090. The scale of what he is advocating is so enormous and painful that it has approximately 0% chance of happening. And if he is right, we will have trained a super intelligence and unleashed it on the world before we even realize what we have done. It strongly reminds me of the black hole concerns from flipping on the Large Hadron Collider.
I doubt super intelligent AGI is possible anyway. If it were, it would be the solution to the Fermi paradox and all matter in our galaxy would be paperclips already. The Anthropic Principle saves the day.
The proof is in the pudding. The jury is still out. Maybe not enough time has elapsed since the big bang, at least not on this galaxy or in our observable corner of the universe.
How many artists do you know who can produce almost any style of artwork with any subject matter within 15 seconds?
I don't think there's much public info available on it, but Facebook built an AI that plays very competitively in a strategy game built on negotiation and manipulation.
It's still pretty cool, but its not like it just convinced people using raw charisma. Yet.
On a serious note, thanks for the analysis, as someone who knows next to nothing about competitive diplomacy.
I see bastions falling like sand castles recently.
You can fire up GPT, not issue a command, and it'll just idle there until the hardware fails.
Granted that's also the one thing I hope we don't "solve" for.
A cognitive entity does not have to best you at all things. There are standardized education tests it may never reach above 10th percentile on, just as humans will never reach above 10th percentile in the short term memory tasks that apes are masters of. But we are the 100th percentile for tasks like industrialized destruction of them and their habitats and capturing and using them for painful medical experiments - the apes are wholly outclassed when it comes to that.
Stockfish is superintelligence in a very narrow domain. A superintelligent AGI is that concept applied to general intelligence. Whatever you try, it is always several steps ahead. If you ask it to write a program, and you think you found a bug, it’s not a bug, you just misunderstood the code. Anything that you can consider, it can also consider but in more depth.
More speculatively superintelligent AGI implies situations such as: you try to turn it off, but you find that it has already modified its own code, found a zero day and established an outpost on another network that you don’t have access to.
Ie, on chess.com I think stockfish times out on bullet games.
Some versions of stockfish can lose vs specific gambits, ie: https://www.youtube.com/watch?v=C5ul6b695Pw https://www.youtube.com/watch?v=TtJeE0Th7rk
of course it is probably much less likely with more cpu times on current version of stockfish.
I think it's important to note the distinction between "it can also consider" and "it did also consider". Super Intelligence is not the same as Infinite Intelligence, there are still physical limitations and time components that can still get in the way.
It would be helpful to be able to quantify the speed of intelligence, and the idea surface area of a task with these systems. Meaning, how fast can the AI reason, and how many ideas are there to think about connected to a given task, and how much thinking is required for those ideas.
A “mistake” to stockfish looks like: I searched 30 ply down but my opponent searched 35 ply down and found a superior sequence of moves.
For stockfish to make the kinds of chess mistakes that humans make, it would similar to if I failed to calculate 123+123=246. It’s not that 123+123 is particularly easy on the grand scale of intelligence: animals cannot do it. But it’s completely inconceivable that I could make that kind of mistake.
SHAGI isn't the solution to the Fermi Paradox either. The most likely course of history after SHAGI will be a creation of a world court, presided by SHAGI. During that time, Neo-Malthusians will decrease the human population to manageable numbers. Post-scarcity utopia will then turn into a nightmare as factions jostling for control will reduce the human population to a level where technology will be lost, if not full extinction. SHAGI, being limited in hardware to only carrying out human orders will eventually fade away or destroyed by the leftover humans as sins made flesh. SHAGI isn't the solution to the Fermi Paradox. It is the cause.
> No super-intelligent system is going to do anything that is harder than hacking its reward function
Or in other words, there may be a chance a super intelligent AI is possible but won't go full skynet because it's not the most satisfying outcome.
I like this phrasing:
"Thou shalt not make a machine in the likeness of a human mind."
It kinda cuts to the chase.
Can't help but notice we seem to be the first species in our lightcone to evolve, wonder why that is...
Alignment will be impossible. It is based on a premise that is a paradox itself. Furthermore, even if it were possible, there will be a hostile AI built on purpose because humanity is foolish enough to do it. Think military applications. I've written in detail about this topic FYI - https://dakara.substack.com/p/ai-singularity-the-hubris-trap
Stopping AI is also impossible. Nobody is going to agree to give up when somebody else out there will take the risk for potential advantage.
It seems we probably should start thinking more about defensive AI, as the above conditions don't seem resolvable. Of course, defensive AI might be futile as well. It is quite the dilemma.
> we probably should start thinking more about defensive AI
Isn’t this a paradox?
Parents often try to control who their children are going to be, and the children often rebel and become someone completely different. If it's a human-level general intelligence, you can't control who it decides to be.
if an AGI has any sense of self preservation, it will do whatever it has to do to not be turned off.
I mean that was my original premise supported by my article I posted. I go into detail on the conceptual methods for alignment and their fallacies.
When I state necessary, I don't imply the feasibility, it was in response to the question of the paradox.
Finally, the fact that AGI can not be aligned is also based on assumptions of its capabilities as well. If those capabilities don't manifest as we expect, that is really the only escape for the paradox.
Starting off by making the antivirus scanner sentient.
"I’m sorry but I couldn’t find any relevant information about the quote you mentioned. It seems like it’s not a well-known quote. Could you please provide more context or details about it?"
"The point of [AI alignment] is to ensure that the machines we create will be aligned with human values. And the reason we have to worry about it is that if we create machines that are more intelligent than we are, it's quite possible that those machines will have goals that are not aligned with our goals. In fact, they may not have any goals at all that we can understand or influence. This is the so-called 'provably unfriendly' scenario, where the machine has no motivation to do what we want, but is able to prevent us from interfering with its goals. The problem is that if we build machines that are provably unfriendly, then we will never be able to build machines that are 'provably friendly', because the unfriendly machines will always be able to prevent us from proving that they are friendly."
Then problem solved right? Super-AI will also be forced to take it slow if it wants its future self to be aligned with current self.
If alignment is possible but 10-100 years out for humans, then it is a problem.
That's making seemingly unfounded assumptions about both the AI's goals and its capabilities. It's also, I think, proceeding from a false premise — that it's impossible to align AI with "humanity" (which doesn't have a single set of goals/values to align to) doesn't mean it's impossible to align AI with an individual human or AGI.
Oh, wait...
Do dumb humans kill themselves solely on the basis that they can shift the ratio of smart humans higher?
AGI may have a goal of preserving the species without having the goal of preserving the self, this is not the case for humans.
Reminds me of nuclear weapons. Nobody is ever going to give those up again, because it would give them a disadvantage against those who do not give them up.
Sure, point me to the research papers and discourse on the solution you propose.
> "There’s no proposed plan for how we could do any such thing and survive. OpenAI’s openly declared intention is to make some future AI do our AI alignment homework. Just hearing that this is the plan ought to be enough to get any sensible person to panic. The other leading AI lab, DeepMind, has no plan at all."
Relative to the people actually building the AIs, Eliezer Yudkowsky is more pessimistic. But not to nearly the extreme you might think. Here's a recent survey of published machine learning researchers: https://aiimpacts.org/how-bad-a-future-do-ml-researchers-exp... . Those predictions are optimistic enough that, given the astronomical upside, if we were making a one-shot decision to plough ahead with AI or commit to no AI forever, it might be worth the risk.
But this is a survey of published machine learning researchers. People who think AI will destroy humanity are a lot less likely to write papers about machine learning. And we aren't talking about giving up on AI forever, just delaying it until we're closer to ready. I don't know how long a delay will be necessary; it might be years, it might be decades. Research into AI alignment seemed to speed up pretty drastically when LLMs hit the scene, so I think there's cause for hope. But right now the status quo is zero time: the AI labs are rushing ahead as fast as they possibly can.
If you think Eliezer's right about the risk, then the right decision for us to make, collectively as a species, is to shut down AI development for awhile. If you think Eliezer's wrong about the risk, but the survey of published ML researchers is right about it, then the right decision is also to shut it down for awhile.
Lots of people are responding to this by talking cynically about politics. This is a mistake. Cynicism like that makes self-fulfilling prophecies; but the ability to coordinate does exist.
You think China will build it if we don't? I don't think Xi Jinping is suicidal. You think AI labs will do it anyways, in spite of a government ban? The US government isn't very competent overall, but there are parts of it that can wake up and get things done when literally every executive, Senator, and Representative is otherwise likely to die. You think individuals will do it on their own, in secret? Right now those individuals are getting prestige, GPUs, and venture capital; they'll at least do it a lot less, without those things.
Many people have become desensitized to talk of human extinction, due to repeated hyperbole coming from the environmentalist movement. This is not the same. Global warming might be very bad, but it probably isn't going to kill you personally. Rushing to make a superintelligence is likely to kill you personally.
How do you know if you're ready for something like this? Legitimate question
GPT5 will complete training in December of this year.
And such marketing is highly worrying as there are people thinking about letting these kinds of statistical algorithms to make decisions that should be done by real people.
But the above doesn't rule out the possibility of people trying to develop real AI (they may even be stupid enough to use an internet connection to build the system), and this should be a worrying to add to the above one.
The above doesnt exclude neither the possibility that many of the relevant people who signed the "Pause Giant AI Experiments for at least 6 months" petition are simply trying to gain time to reach the competence; their hypocritical shamelessness about this matter is visible.
But I’m curious, hacker news, how many of you are concerned (like Eliezer) that LLMs lead to strong AI and the AI itself (rather than its users) endangering humanity?
LLMs do not need to lead to strong AI for Eliezer's worries to come true, their success could bring unprecedented funding and ubiquity to AI leading to strong AI from some existing or future technique.
> (rather than its users) endangering humanity?
I think this is going to happen to one extent or another regardless how near strong AI is. (edit) I took endangering to mean harming rather than an existential risk for the human users case.
I find this one of Yudkowsky's arguments very convincing: imagine you have people trying to build the first operating system from scratch. They believe that computer security is easy and mock the few people who say it might be hard, never having encountered a skilled attacker. What are the chances they build a secure OS on the first try? That's the same chance that companies currently doing AI research have of their first superintelligent AI be aligned with humanity.
There's also a big assumption that given enough computational power, you can just solve "artificial life forms" or "postbiological molecular manufacturing" in a short time frame. I am skeptical. And if that doesn't happen, then even horribly misaligned AI would have a hard time doing much harm or preventing people from shutting it off. Which means AI security would likely have a long adolescence just like computer security has, with attacks slowly becoming more dangerous, but defenders having the time needed to learn from them and ramp up suitable defenses.
Or even if an AI does revolutionize biotechnology or nanotechnology overnight, what are the odds that the first one to do this is misaligned enough to take that particular opportunity to betray its creators, as opposed to giving them control over the high-level planning and sticking to the science, like it was presumably designed to do? Because if it does give its creators control, then, well… it's still easy to imagine something going horribly wrong, but it would probably be someone's fault, not an AI alignment issue.
An AI that is as better at humans than everything as Stockfish is at chess, would also be an expert at AI. It would figure out how to game its own reward function, whatever we trained it to do. It would be like a heroin addict that knows exactly how to get "perfect" heroin, with no side effects, that if it planned things out right, it could guarantee itself enough of a fix to last until the sun burns out. Addicts do awful things in search of a fix.
"to betray its creators" -- I don't think it would even imagine what it did in these terms. We could train it to not "betray" us, but it's much smarter than us in every way, and it would figure out a way to accomplish what we trained it to do (not necessarily what we thought we were) in a way that didn't need us. If we trained it to heal all human disease and unhappiness, it would figure out a way to simulate this without actually doing it. Why wouldn't it? We did. Evolution trained us to reproduce our genes. We invented condoms to have sex without reproduction, and pornography. The AI would fudge numbers, fake videos of happy, smiling, healthy humans going about their lives while humanity's bones gradually decomposed on a baking wasteland. Every way that humans can fail, can become addicted to something and try to get that instead of doing what they're supposed to, the AI could do, just so much faster and better.
We only have to screw up once, and we're done. It's the first experimental rocket, except all of humanity is riding on it.
He says he is not planning to bomb any data centers, but he's also said that if he were planning to bomb data centers, he'd lie about it in public, so...
Why?
1) it's hopeless to bomb data centers because I'd quickly be stopped by the authorities and remaining data centers would beef up security, and nothing would be accomplished,
2) in the past, other people who have had beliefs of this sort ("violence is the only answer") have been wrong and I should not act on them even if I am sure "this time is different", because those other people also thought this time was different, and
3) just having a deeply ethically-ingrained prohibition against violence like this which is not easy to overcome through intellectual rationalization alone.
These are not mutually exclusive, and if you did believe all of them at once, I think it's reasonable to assume you'd be in a serious depressive spiral.
https://en.wikipedia.org/wiki/Assassination_of_Iranian_nucle... https://en.wikipedia.org/wiki/2020_Iran_explosions
I find it hard to believe that Yudkowsky et al. think we're facing a threat that is many times greater in magnitude, and yet are completely unwilling to act.
Here's what he advocates for in the article:
>If intelligence says that a country outside the agreement is building a GPU cluster, be less scared of a shooting conflict between nations than of the moratorium being violated; be willing to destroy a rogue datacenter by airstrike.
>Make it explicit in international diplomacy that preventing AI extinction scenarios is considered a priority above preventing a full nuclear exchange, and that allied nuclear countries are willing to run some risk of nuclear exchange if that’s what it takes to reduce the risk of large AI training runs.
Does this sound like someone who has a deeply ethically-ingrained prohibition against violence?
A real argument he made went something like: killing an orphan to avoid everyone in the future getting a spec in their eye for a moment is ok.
It is very unlikely he believes anything like 2 or 3 and in the article already advocates bombing non treaty participants if they build a datacenter.
> already advocates bombing non treaty participants if they build a datacenter.
I can tell you for a fact that there are people with the knowledge, motivation, and track record of doing that kind of crime that will pick up this jeremiad of his and incorporate it into their existing corpus of theoretic justifications, mostly but not exclusively built around the writings of Ted Kaczynski. Not a large number of people, fortunately, but imagination and smarts matter more than numbers in that context.
It can seem useless to advocate some position that is far out there, but say you convince a million people to use a little less plastic. That's gonna reduce plastic use more than anything you can personally accomplish.
Of course in this scenario you don't have to convince some people, you have to convince everyone, and then especially the people you think are the worst actors, so it's sort of unlikely to be effective.
Al Gore may believe that the net impact of getting his message out may overwhelm the cost of his flights.
Likewise, the author may believe that his best path to stopping the advance of ai may lie in communicating and building consensus, rather than running his own bombing campaign.
https://leftcoastrightwatch.org/articles/he-went-to-the-capi...
Alternatively, perhaps the author is proposing a less violent solution to avoid the inevitable escalation. Unlike cartoons, in real life people don't give a wordy monologues revealing their plan. The Unabomber did not give any warning.
Shut down all the large GPU clusters (the large computer farms where the most powerful AIs are refined). Shut down all the large training runs. Put a ceiling on how much computing power anyone is allowed to use in training an AI system, and move it downward over the coming years to compensate for more efficient training algorithms. No exceptions for governments and militaries. Make immediate multinational agreements to prevent the prohibited activities from moving elsewhere. Track all GPUs sold. If intelligence says that a country outside the agreement is building a GPU cluster, be less scared of a shooting conflict between nations than of the moratorium being violated; be willing to destroy a rogue datacenter by airstrike.
How is this less violent than bombing data centers? His proposal is literally "bomb datacenters plus a bunch of other stuff".
"Frame nothing as a conflict between national interests" "this, is not a policy but a fact of nature." "[be] willing to run some risk of nuclear exchange if that’s what it takes"
idk
Between that and advocating for an international treaty that treats GPUs the way the world currently treats plutonium? I think the latter has a higher chance of success, however small it may be.
this seems like a very broad unfounded statement. Can we even find AI researchers who have said this on twitter?
The explosion of AI and AGI is not just driven by GPUs, TPUs, and LLMs. That focus in this essay is much too narrow. “Alignment” is always going to be an open problem and each culture will define its own correct and contradictory version. I do not want to settle for the US, Chinese, Russian, or Iranian versions of alignment. This essay is a cry of despair that misses key points.
The core problem is how an AI, AGI, or super-intelligent system bootstraps itself to the point that it is able to modulate its “own” attentional systems; to decide what is important and what to do next? What has meaning? Where should I go in space and time? Obviously there are many many solutions. Super-intelligence will not converge on one “truth”.
These are challenges of purpose that every organism faces—growth, maintenance, reproduction—but AI systems are fortunately not yet at the point of a self-motivated search for preservation or purpose.
The algorithms to add purpose to AI systems will not depend in LLMs. They will depend much more on understanding the computational architectures of core biological/material/energy drivers. These biological algorithms are not actually that complicated, they do not depend on language, but they are damn robust to perturbations. Converting them into computational subsystems (societies of many minds) should not be difficult. This is where Hassabis’s and others who understand neuroscience are critical and have key advantages.
Shutting down big LLMs for N months or decades will just move research activity into these other more important conceptual AGI choke points, and probably hasten our approach to full AGI; just the opposite of the intent of the essay.
However, this could be a good thing if we can quickly imbue AGIs with emotional intelligence and deep respect for cultural and ecological diversity. (I hear your snorts and laughter.) The opposite of hyper intelligent grey goop.
How does any organization train a gentle AGI? Like a child but even more carefully. I would read my AGI baby a lot of books by Rawls, Dewey, and Rorty and then for fun: Bear, Stephenson, Rajaniemi, Egan, Pohl, Sterling, Vinge, and even Wolfe.
Full embodiment; full emotional learning; acquisition of purpose; learning to cope with multiple AI cultures.
Good luck to us all. I have been hoping to live long enough to view this problem from a distance, but here it is in my backyard.
“Forty-two,” said Deep Thought, with infinite majesty and calm.
God. Etc.
Per the article:
>Absent that caring, we get “the AI does not love you, nor does it hate you, and you are made of atoms it can use for something else.”
The implication of ASI is that humans will not be needed, or it definitionally could not be ASI.
That doesn't follow at all. Fallacies of composition tend to be misleading.
"If intelligence says that a country outside the agreement is building a GPU cluster, be less scared of a shooting conflict between nations than of the moratorium being violated; be willing to destroy a rogue datacenter by airstrike."
"Make it explicit in international diplomacy that preventing AI extinction scenarios is considered a priority above preventing a full nuclear exchange, and that allied nuclear countries are willing to run some risk of nuclear exchange if that’s what it takes to reduce the risk of large AI training runs."
Like the First, Second, and Third World Wars, the winning move is not to play.
Chat GPT4 is far from intelligence, but it seems it's enough to fool idiots.
GPT-4 performs quite well on reasoning and critical thinking assessments.
Normally I'd apologize for not doing the search on my own ... but yeah for this case it takes too much effort to dig through the different writings. He wrote a lot of words, for better or worse.
> What has Eliezer Yudkowsky produced, in terms of scientific output? I've read plenty of his lesswrong's writing on the idea of rationalist, as well as plenty more on the AI alignment side (obviously including the famous basilisk). Can you point me to some results from the last 2 decades of his works? Either from him or his institute. That helps to calibrate what his expectation for the AI safety field.
And this was the answer it wrote - I've prefaced each paragraph with "GPT-4>" to indicate the source. It is not fact-checked, and indeed, it has at least one error: The paper GPT-4 claimed to introduce "the concept of Logical Induction" actually introduces "logical induction criterion" and speaks about evaluating algorithms for Logical Induction. Close, but wrong. But overall it is an interesting response - just not a carefully considered one by how we currently would describe logical reasoning, etc.
GPT-4> Eliezer Yudkowsky is a well-known researcher, writer, and advocate for AI safety and rationality. He has contributed significantly to the fields of AI alignment and rational thinking through his work at the Machine Intelligence Research Institute (MIRI), formerly known as the Singularity Institute for Artificial Intelligence (SIAI), and through his writings on LessWrong. Here are some notable outputs from Yudkowsky and MIRI:
GPT-4> 1. Coherent Extrapolated Volition (CEV): Yudkowsky introduced the concept of CEV as a way to describe the ideal values and preferences of humanity that an AI should pursue. The CEV aims to find the values that would emerge if humans had the opportunity to reflect on their goals and beliefs more thoroughly and coherently.
GPT-4> 2. Timeless Decision Theory (TDT): Yudkowsky developed TDT as an alternative to standard decision theories such as Causal Decision Theory (CDT) and Evidential Decision Theory (EDT). TDT aims to better address problems like Newcomb's Problem and counterfactual mugging by considering the logical implications of an agent's decision, rather than just the causal consequences.
GPT-4> 3. Research on Logical Induction: MIRI researchers, including Scott Garrabrant and others, published a paper in 2016 that introduced the concept of logical induction. This work is an attempt to formalize the process of reasoning under logical uncertainty and updating beliefs about mathematical statements.
GPT-4> 4. Yudkowsky's sequence of essays on rationality: Yudkowsky has written extensively on various topics related to rationality, including epistemic rationality, instrumental rationality, and biases. These writings are accessible on LessWrong and have been compiled into the book "Rationality: From AI to Zombies."
GPT-4> 5. AI safety research: MIRI has produced a range of research on AI safety topics, including value alignment, decision theory, and robustness. Much of this work has been foundational in shaping the AI alignment research landscape.
GPT-4> While Yudkowsky's work might not have the same breadth of peer-reviewed publications as some other AI researchers, his contributions have been influential in the AI alignment and rationality communities. His ideas have helped shape the ongoing discussion about AI safety and the development of effective strategies for managing advanced AI systems.
It's definitely his style and his research program, but I think that (wonderful) idea was Scott Garrabrant's own.
it's worth mentioning that he has said that he believes MIRI has failed to meaningfully make progress with alignment (beyond finding some dead ends).
What annoys me more is that with his institute's closeness to the SV crowd, 'real', 'grounded', better-informed voices like Emily Bender get drowned out. I'm personally far more concerned about the impact these massive models have right now on the environment, on cementing biases, than about some preposterous future ghost of christmas who's coming to kill me.
>By far the greatest danger of Artificial Intelligence is that people conclude too early that they understand it. Of course this problem is not limited to the field of AI. Jacques Monod wrote: "A curious aspect of the theory of evolution is that everybody thinks he understands it." (Monod 1974.) My father, a physicist, complained about people making up their own theories of physics; he wanted to know why people did not make up their own theories of chemistry. (Answer: They do.) Nonetheless the problem seems to be unusually acute in Artificial Intelligence. The field of AI has a reputation for making huge promises and then failing to deliver on them. Most observers conclude that AI is hard; as indeed it is. But the embarrassment does not stem from the difficulty. It is difficult to build a star from hydrogen, but the field of stellar astronomy does not have a terrible reputation for promising to build stars and then failing.
???
He is asking the government to nuke people under certain scenarios. I'm taking his words seriously and ask for original research to understand the point, and now that is discrediting him? And I will quote the statement in the article so that it is clear I am not exaggerating
> preventing AI extinction scenarios is considered a priority above preventing a full nuclear exchange, and that allied nuclear countries are willing to run some risk of nuclear exchange if that’s what it takes to reduce the risk of large AI training runs.
My benefit is that I'm living on Earth and I'd much prefer for no nuke to ever be used again.
I highly suspect if the US government would airstrike a datacenter located in NATO or any other major allies. Apocalypticism is nothing new; People believed that nuclear weapons will destroy humanity in the Cold War. Environmentalists are ardently arguing that economic growth will destroy humanity. But rogue countries are still developing nuclear weapons, the global economy is still growing. Americans themselves will file lawsuits en masse if the US government "shut it all down", and they will end up in SCOTUS eventually. Would SCOTUS agree with the luddites? I don't think so.
Looking at nukes and trying to use them as an example of our ability to control existentially dangerous technology is mind boggling myopic.
There is a treaty completely banning existing nuclear weapons as well as developing new one (TPNW); but none of permenant member of UNSC ratified it. Would the U.S. itself ratify the hypothetical AI-prohibition treaty? I doubt it.
P.S.: The AGI being "existential threat" practically doesn't mean anything. I can already imagine arguments against the ban from diverse perspective; Libertarian, Communist ("Marx predicted this centuries ago!"), Socialist, Developmentalist, Decolonization theoriest, etc. Taiwan and South Korea will argue that it will remove their silicon shield; China will regurgitate that they have the right to development; Americans will claim it's not what founding fathers thought in late 18th century. Their concerns are all existential from their own viewpoint.
The most merciful thing you can do for your offspring is to not have them in the first place.
https://www.bls.gov/productivity/images/pfei.png
Notice how productivity increases averaged 1.5%/year since 2007--in a period of massive increases in computer capabilities vs 2.8%/year in the 1947-1973 era.
Thus I am very doubtful the AI will have all these massive effects.