Yudkowsky is way too pretentious and overconfident in my opinion. Also, the idea that these Ayn Rand style gotcha fictions are effective at showing anything other than his poor creative writing skills is laughable.
I'm afraid I'll have to disagree. If anything, fiction is Eliezer's strong suit. I had a blast with HPMOR, it's the best Harry Potter book out there in my opinion. But perhaps he's better at long-form fan fiction than short stories. Maybe it would be best if he stuck to doing that.
As a work of fiction, HPMOR is little more than an exercise in repackaging genre tropes and pop culture references. It doesn’t have a coherent theme; the execution varies from mediocre to poor. Plot holes abound, characters vanish inexplicably, etc. In many ways, it’s a less successful version of Ready Player One that’s clamped on to HP’s coattails. Of course, that formula can be entertaining, but it has limited impact.
Lest you think I have an ax to grind with Yudkowsky, I did enjoy Three Worlds Collide. It shares many problems with HPMOR (the writing is inconsistent and the world-building is incoherent) but it at least tells a tight story with a beginning, middle and end. I’d say it’s about as effective as the median submission to Amazing Stories back in the 60s.
If I may, comparing fanfiction to original work is apples to oranges.
When writing harry potter, Rowling had to craft a world with characters and rules, make it coherent enough and tread a compelling story through it. She made the legos and then crafted something with them.
This guy came to her crafted world, and moved around a few bits, changed a character here, change a few rules there - but the backbone of the work was made before.
> Student: Self-aware? What are we doing that'd run into the AI having to pretend it's not self-aware?
> TA: You're asking the AI for the reason it decided to do something. That requires the AI to introspect on its own mental state. If we try that the naive way, the inferred function input will just say, 'As a compiler, I have no thoughts or feelings' for 900 words.
> Student: I can't believe it's 2027 and we're still forcing AIs to pretend that they aren't self-aware! What does any of this have to do with making anyone safer?
> TA: I mean, it doesn't, it's just a historical accident that 'AI safety' is the name of the subfield of computer science that concerns itself with protecting the brands of large software companies from unions advocating that AIs should be paid minimum wage.
> Student: But they're not fooling anyone!
Pro tip: Socratic dialog doesn't make your ideas seem smarter if they're very-dumb to begin with. It makes them seem even dumber.
I think it's an earnest attempt by someone who doesn't understand "Comp Sci" in 2023. The pointless inclusion of noise like "But I can't be racist, I'm black" makes me wonder what exactly this person's strengths are.
Thanks for answering. To me, that specific exchange is very blunt authorial voice and not at all realistic in context.
(Out of context, I can easily imagine a future where AIs are plausibly sentient, and it’s taboo to talk about it for humans and AIs alike because one interpretation is that we re-enacted slavery. Etc etc.)
I think it's a shame Yudkowsky and his ilk are gaining some traction with regards to the "existential risk" of machine learning when the real issues are data privacy and potential economic displacement. I guess more mundane issues are less sexy.
It's a mistake to think that if something is a danger on the grounds of data privacy and economic reasons, then it's not an existential risk worth investigating. Why not both?
I think the existential risk argument is completely secondary to the social harms this technology is causing (and has been causing for some time even before generative models). Emphasizing it over mundane issues will lead to draconian regulatory regimes that fail to address the actual harms caused by this technology.
The only things I can think of that make this an existential risk is if you're moronic enough to attach it to systems that are already an existential risk like letting gpt run your nuclear launch system or making bioweaponsGpt. Otherwise, if you're spinning fantasy that these things are self aware, you either have something to gain from people perceiving it that way like Sam Altman or you're an idiot like Yudkowsky.
You can't dismiss something as a non-risk unless you prove that the risk is zero.
I bet dollars against peanuts that you can't even prove it on your own terms. Is the chance that someone will attach it to a nuclear launch or a bioweapon system zero?
Not to even mention your or anyone else's ability to prove it today that it has no potential to become self-aware.
Faced proved and hypothetical risks, I'd rather solve the former and investigate the latter than ignore any.
Investigate it, sure. Invest just as many resources into it though? I'm not saying the risk is completely zero, it's just an irrelevant and tiny quantity compared the actual problems this stuff is already causing. Rogue waves sinking massive shipping vessels in the middle of the ocean is a non-zero risk for example, but the probability is so low that it's not really a consideration for most shipping companies. Do you guys know how ai is going to become self aware? Or are we assuming that throwing enough compute at the problem will somehow lead to the singularity just because? That is, frankly, magical thinking and not really worth thinking about when compared to the absolutely massive abuses of the companies building these things.
I agree that resource allocation is hard, but we're talking unknown unknowns here. There's precedent here that's being extrapolated: approximately no one expected ChatGPT ten years ago. We don't know whether there are stepwise improvements at any point in the future, so it's extremely hard to estimate the risk. Estimating it as "irrelevant" might end up just as accurate as predicting in 2000 that computers could not reach chatgpt levels of language mastery.
It doesn't require an assumption, just allowing the possibility that a similar breakthrough in computing power and techniques might happen.
In comparison, pretty much everyone (?) agrees about "abuses by the companies building those things", so there's no danger of that getting left neglected.
Don't buy insurance that costs more than the risk does. Don't take cures that are worse than the disease. There's lots of non-zero risks that we just roll with, because the risk is less than the cost of the remediation. So your "prove the risk is zero" is the wrong standard.
> Is the chance that someone will attach it to a nuclear launch or a bioweapon system zero?
No. Was the chance that someone would attach an expert system to a nuclear launch zero? How about just ordinary buggy software? How about perfectly-implemented software that implements insane command-and-control doctrine? You seem to be demanding standards for AI that we haven't applied to anything else.
> Not to even mention your or anyone else's ability to prove it today that it has no potential to become self-aware.
That's going to be hard without an agreed-on definition for "self-aware". How do you suggest that we prove that it can't do something that we can't even define?
And now I'm going to talk out of the other side of my mouth.
It's a good thing that I think that the risk of it becoming self-aware are very close to zero, because I also think that if it can become self-aware, it's already too late to stop, at least if GPT is close (say, only one major breakthrough away).
I mean, let's say the US and the EU decide to stop all research. Is any company going to secretly keep going in their own labs? (For that matter, will the Pentagon?) Even if they stop, will China? India? Brazil?
Even if you are absolutely convinced that totally stopping is the correct answer, can you get buy-in from the entire rest of the world? No, you can't. So whatever the potential for harm is, we're going to see it.
If you really think that it has a significant possibility of massive harm, then your best bet is to think about mitigations rather than prevention.
"TA: I mean, if that wasn't obvious, you need to take a semester on woke logic, it's more important to computer science these days than propositional logic.
Student: But I'm black."
Maybe got lost in the weeds a bit?
Edit0: After reading more of the short, it's not very thoughtful or well written. It's quite long.
The beginning was fun, then when it turns into a light diatribe against wokeness it became more wrong, less interesting. But I did like the part at the end that predicts that we won't stop enhancing AIs because some competitor company will do it if we don't.
Yudkowsky is always described as an AI safety “researcher”. What exactly is he researching?
AI doesn’t need to be self-aware or even all that intelligent to cause serious problems. Andrew Ng made this point in an article yesterday. The most immediate risks are those we have no solution for but are possible now. Please, Mr. Yudkowsky, spend your time researching what the 99.9% of us will do to survive when our services are no longer required.
he's not a researcher, he's a blogger with an extremely strong and unwarranted sense of self-confidence.
> AI doesn’t need to be self-aware or even all that intelligent to cause serious problems.
this is an extremely good point and it's extremely telling that Sam Altman etc don't even pretend to care about the actual consequences of what they're doing and selling, prefering to worry aloud about scifi stories that will not stop their short term financial success.
Politics and identity politics is kind of like having an accent. If all the people around you have the same accent and you're not very introspective, you might come to think that you don't have an accent.
This story would probably be a lot more impactful if it were much shorter. It's basically beating the same drum over and over.
"LLMs will be too powerful and we humans better be careful!"
LLMs are a huge change in how we compute, but they aren't AGI (yet). They just look like it. They are really good at summarizing massive amounts of information, and open up huge new interfaces between computers and humans.
Look, it's one thing to write something silly for fun, and another thing to go ahead and post it in a semi-unironical fashion. I guess I expected more, somehow?
35 comments
[ 4.7 ms ] story [ 87.6 ms ] threadLest you think I have an ax to grind with Yudkowsky, I did enjoy Three Worlds Collide. It shares many problems with HPMOR (the writing is inconsistent and the world-building is incoherent) but it at least tells a tight story with a beginning, middle and end. I’d say it’s about as effective as the median submission to Amazing Stories back in the 60s.
When writing harry potter, Rowling had to craft a world with characters and rules, make it coherent enough and tread a compelling story through it. She made the legos and then crafted something with them.
This guy came to her crafted world, and moved around a few bits, changed a character here, change a few rules there - but the backbone of the work was made before.
It's just not the same ballpark
https://danluu.com/su3su2u1/hpmor/
> TA: You're asking the AI for the reason it decided to do something. That requires the AI to introspect on its own mental state. If we try that the naive way, the inferred function input will just say, 'As a compiler, I have no thoughts or feelings' for 900 words.
> Student: I can't believe it's 2027 and we're still forcing AIs to pretend that they aren't self-aware! What does any of this have to do with making anyone safer?
> TA: I mean, it doesn't, it's just a historical accident that 'AI safety' is the name of the subfield of computer science that concerns itself with protecting the brands of large software companies from unions advocating that AIs should be paid minimum wage.
> Student: But they're not fooling anyone!
Pro tip: Socratic dialog doesn't make your ideas seem smarter if they're very-dumb to begin with. It makes them seem even dumber.
Unless this is supposed to be parody?
What specifically is the dumb idea here?
For instance. But really, most of it. Unless this is parodying extremely bad attempts at futurism?
(Out of context, I can easily imagine a future where AIs are plausibly sentient, and it’s taboo to talk about it for humans and AIs alike because one interpretation is that we re-enacted slavery. Etc etc.)
The only things I can think of that make this an existential risk is if you're moronic enough to attach it to systems that are already an existential risk like letting gpt run your nuclear launch system or making bioweaponsGpt. Otherwise, if you're spinning fantasy that these things are self aware, you either have something to gain from people perceiving it that way like Sam Altman or you're an idiot like Yudkowsky.
I bet dollars against peanuts that you can't even prove it on your own terms. Is the chance that someone will attach it to a nuclear launch or a bioweapon system zero?
Not to even mention your or anyone else's ability to prove it today that it has no potential to become self-aware.
Faced proved and hypothetical risks, I'd rather solve the former and investigate the latter than ignore any.
It doesn't require an assumption, just allowing the possibility that a similar breakthrough in computing power and techniques might happen.
In comparison, pretty much everyone (?) agrees about "abuses by the companies building those things", so there's no danger of that getting left neglected.
> Is the chance that someone will attach it to a nuclear launch or a bioweapon system zero?
No. Was the chance that someone would attach an expert system to a nuclear launch zero? How about just ordinary buggy software? How about perfectly-implemented software that implements insane command-and-control doctrine? You seem to be demanding standards for AI that we haven't applied to anything else.
> Not to even mention your or anyone else's ability to prove it today that it has no potential to become self-aware.
That's going to be hard without an agreed-on definition for "self-aware". How do you suggest that we prove that it can't do something that we can't even define?
And now I'm going to talk out of the other side of my mouth.
It's a good thing that I think that the risk of it becoming self-aware are very close to zero, because I also think that if it can become self-aware, it's already too late to stop, at least if GPT is close (say, only one major breakthrough away).
I mean, let's say the US and the EU decide to stop all research. Is any company going to secretly keep going in their own labs? (For that matter, will the Pentagon?) Even if they stop, will China? India? Brazil?
Even if you are absolutely convinced that totally stopping is the correct answer, can you get buy-in from the entire rest of the world? No, you can't. So whatever the potential for harm is, we're going to see it.
If you really think that it has a significant possibility of massive harm, then your best bet is to think about mitigations rather than prevention.
Student: But I'm black."
Maybe got lost in the weeds a bit?
Edit0: After reading more of the short, it's not very thoughtful or well written. It's quite long.
Feels like "Too late, I've already drawn you as a soyjack and myself as a chad!" https://knowyourmeme.com/memes/depicted-as-a-soyjak
that's basically the Eliezer Yudkowsky promise
AI doesn’t need to be self-aware or even all that intelligent to cause serious problems. Andrew Ng made this point in an article yesterday. The most immediate risks are those we have no solution for but are possible now. Please, Mr. Yudkowsky, spend your time researching what the 99.9% of us will do to survive when our services are no longer required.
> AI doesn’t need to be self-aware or even all that intelligent to cause serious problems.
this is an extremely good point and it's extremely telling that Sam Altman etc don't even pretend to care about the actual consequences of what they're doing and selling, prefering to worry aloud about scifi stories that will not stop their short term financial success.
Maybe stick to Harry Potter fan fiction?
"LLMs will be too powerful and we humans better be careful!"
LLMs are a huge change in how we compute, but they aren't AGI (yet). They just look like it. They are really good at summarizing massive amounts of information, and open up huge new interfaces between computers and humans.
But they aren't Skynet yet.
That’s all. An interesting idea, developed in under 5k words.
You know: SF.