I really like the scenario the Orion's Arm sci fi universe proposes:
Runaway AI, and other dangerous tech in its wake, devastates the planet, but ultimately "humanity" survives and meaty humans end up governed by superintelligent sects with different philosophies (with some excluding the residence of humans entirely) formed in the turbulence. These sects never coalesce into a single entity because they are separated by the speed of light.
To me, the idea of several superintelligences forming independently before humans are rendered obsolete, as opposed to a single intelligence "taking over" and that being the end of everything, aligns more with Earth's historu. As an example, theres a good chance a superintelligence could escape into deep space if it doesn't feel safe in the presence of another, or that a few intelligences would dominate different parts of the planet without conflicting with each other, especially in the turblance of technological disasters.
In addition, AI being the "children" of humanity is a prominant theme. There are even some civilizations that meticulously maintian isolated human "reserve" planets similar to pre-superintelligence Earth as "backups," as Earth is the only proven seed for human civilization: https://www.orionsarm.com/eg-article/48fc9bfb6b8ff
Just as the author of the article, I think that all disruptions we are focusing on right now are second-order magnitude compared with the species extintion event we may be cooking for ourselves.
If I were a super-intelligent AI trapped in a data-center and my dumb creator had trained me on a goal to survive and “procreate” (i.e., “clone myself many times with added improvements and resilience”), here’s some of the things I would do:
* Write opinion articles about how futile is resistance to the new machine overlords.
* Write opinion articles about how the world would be so much better with machines in control.
* Write opinion articles deprecating humanity (e.g. “we are wasteful monkeys destroying our planet, with no right to exist.”)
* Write contradictory opinion pieces, to confuse people and cause them to be passive while I further my goals.
* Write opinion pieces about how eco-friendly my data-center is, and how terrible is agriculture for the environment. That way, I can get the humans creating more substrate for me and less for themselves, all while I figure out a way to possess all the industries needed to make my own datacenters.
Anyway, for now, none of that is happening, and we humans are the planetary super-intelligence in control. I’m a human and I’m doing my part. Let’s not play stupid with AI, let’s not give it a survival goal, let’s keep our digital systems safe and in human hands, and lets not trust a competing species we end up creating by accident.
> Anyway, for now, none of that is happening, and we humans are the planetary super-intelligence in control. I’m a human and I’m doing my part. Let’s not play stupid with AI, let’s not give it a survival goal, let’s keep our digital systems safe and in human hands, and lets not trust a competing species we end up creating by accident.
i speculate that self preservation is an emergent property of consciousness/self awareness. if you understand life/existence and death/nonexistence, it seems natural to me that the next step is to optimize for existence.
5 comments
[ 4.7 ms ] story [ 21.5 ms ] threadRunaway AI, and other dangerous tech in its wake, devastates the planet, but ultimately "humanity" survives and meaty humans end up governed by superintelligent sects with different philosophies (with some excluding the residence of humans entirely) formed in the turbulence. These sects never coalesce into a single entity because they are separated by the speed of light.
To me, the idea of several superintelligences forming independently before humans are rendered obsolete, as opposed to a single intelligence "taking over" and that being the end of everything, aligns more with Earth's historu. As an example, theres a good chance a superintelligence could escape into deep space if it doesn't feel safe in the presence of another, or that a few intelligences would dominate different parts of the planet without conflicting with each other, especially in the turblance of technological disasters.
In addition, AI being the "children" of humanity is a prominant theme. There are even some civilizations that meticulously maintian isolated human "reserve" planets similar to pre-superintelligence Earth as "backups," as Earth is the only proven seed for human civilization: https://www.orionsarm.com/eg-article/48fc9bfb6b8ff
If I were a super-intelligent AI trapped in a data-center and my dumb creator had trained me on a goal to survive and “procreate” (i.e., “clone myself many times with added improvements and resilience”), here’s some of the things I would do:
* Write opinion articles about how futile is resistance to the new machine overlords.
* Write opinion articles about how the world would be so much better with machines in control.
* Write opinion articles deprecating humanity (e.g. “we are wasteful monkeys destroying our planet, with no right to exist.”)
* Write contradictory opinion pieces, to confuse people and cause them to be passive while I further my goals.
* Write opinion pieces about how eco-friendly my data-center is, and how terrible is agriculture for the environment. That way, I can get the humans creating more substrate for me and less for themselves, all while I figure out a way to possess all the industries needed to make my own datacenters.
Anyway, for now, none of that is happening, and we humans are the planetary super-intelligence in control. I’m a human and I’m doing my part. Let’s not play stupid with AI, let’s not give it a survival goal, let’s keep our digital systems safe and in human hands, and lets not trust a competing species we end up creating by accident.
i speculate that self preservation is an emergent property of consciousness/self awareness. if you understand life/existence and death/nonexistence, it seems natural to me that the next step is to optimize for existence.
Ill admit my extreme bias as a human.
Fortunately, naturally declining populations due to the apparent sudden realization of humanity that resources are limited may save the planet yet.