During the early days of the pandemic I put up a poster someone made which was hand washing instructions but with the text replaced with the litany against fear.
Yes. The inevitable defeatism that will show up in these comments is
"Oh, but China will do it anyway, so there's no point."
Which is pretty easily counterable:
1) We don't know if China would cooperate with a ban. We haven't tried. China is very complicated, and if you think you can predict what Xi will do, you are wrong.
2) If AI is truly a global existential and moral crisis, the US _could_ absolutely shut down China's AI research capabilities. There are a few avenues here, some less pleasant than others. Think outside the box.
It's really not, since we don't have anything close to a strong (or "general") AI. GPT-3 is the closest, but even it is just in that direction. So it is quite early. Good time to have the conversation.
That's exactly my point. I did, explicitly, say it was unpleasant, and something you should only do in extremis -- when you genuinely believe that extinction a likely alternative.
But the US could absolutely glass every semiconductor fab (and datacenter, and research facility) in mainland China using a combination of conventional + atomic kinetic options.
> We don't know if China would cooperate with a ban. We haven't tried. China is very complicated, and if you think you can predict what Xi will do, you are wrong.
This is astonishingly optimistic (or naive). I'm sure China would happily agree to cease AI development in public and then continue in private.
Getting any country to actually stop AI development is as likely as getting them to give up all their nuclear weapons. AI is a weapon, both in military and economic contexts.
> To slim results. Elon Musk explained his new nihilism about the possibility of stopping AI advancement on the Joe Rogan podcast, when he said:
“I tried to convince people to slow down. Slow down AI. To regulate AI. This was futile. I tried for years. Nobody listened. Nobody listened.”
Rather rich coming from one of the self driving car market leaders. It certainly makes business sense to mislead academia and the policy sector into wasting resources on figuring out the best philosophical and ethical regime while large corporations benefit from regulatory capture. If he has his way, ML would become like the pharmaceutical industry, with multiple barriers of entry if you are not well-funded, well-connected, or established.
I do believe it's reasonable to draw a distinction between what we're doing now (which is essentially just "statistics") and what Musk warns about.
It's a mistake to believe strong AI will just be a more powerful iteration of today's weak AI. He is arguing to slow developments toward general intelligence, not developments in any narrow field.
With high probability, 'just statistics' is an essential component of strong AI. Another essential component is embodiment, of which self-driving cars, and also military drones, are canonical examples. Researchers are taking the correct essential steps towards strong AI, it's a matter of (short) time until they succeed.
Elon wasn't calling for a hard halt like this article. He's talking about the dangers of general AI (AGI). I don't think most people would consider a self-driving car a likely path to accidental superintelligence; it's a highly-targeted application like a chess engine.
OpenAI's GPT-X engines OTOH, IMO, have a lot more potential danger because it's very unclear what they'll be used for.
If you do actual ML research, the technologies are two sides of the same coin. Three days ago there was a paper posted here in HN on using the GPT-like transformer architecture for reinforcement learning problems (of which self driving cars is a partial subset of).
> Rather rich coming from one of the self driving car market leaders.
You don't need anything close to strong AI to do a reasonable job of driving a car. It seems like 90% of the problem is object recognition, and even in terms of brain-equivalent logic that's a really low bar.
It’s the 10% that gets you. To safely operate a car in all reasonable situations might very well require Strong AI. The car needs to be able to problem solve and make inferences about road conditions up ahead.
What kind of inferences need strong AI? How often does a driver need to figure out something with logic that couldn't be handled by current technology?
Level 4 self driving is fine and I really don't think it needs strong AI.
That's really not the 10% I was talking about. We don't need that part to follow some lanes.
Or the other way to put it is that all the other code is the first 90%, and then the "remaining 10% that's 90% of the effort" is the object recognition that was supposedly easy.
We do! I live near a highway and every time I use a particular on-ramp, the tar stripes make my car think it's going off the road. It never learns, and it has no plan B.
It's easy to do the lane-keeping, it's completely terra incognita to decide when to abandon the standard method that works 90% of the time. And what to do next.
For object recognition code, including lane-finding, to be sufficient, it needs to be looking far enough ahead that the car always has a plan B. If we reach that level of quality, and it still fails to find the lane, then the car can simply stop.
Deciding when to stop is a really easy threshold based on how confused the recognizer is and how far away the source of confusion is.
But also those sources of confusion will get rarer and rarer as the object recognition improves, to the point that even very mediocre handling of those situations will still give you an overall good experience.
This is so obvious. Incumbents would like nothing better than a new law that makes it harder to operate (within “reason”, of course) in their industry. It is amazing that more people here don’t see right through the manipulation.
* AI research already has high barriers to entry because the required computing resources are expensive.
* Self-driving AI will be heavily regulated for reasons not related to other AI regulation.
I don't see how more AI regulation would financially help Tesla or Elon Musk. In fact "more regulation of self-driving" is something that many Tesla competitors and critics support.
I think the cat is out of the bag when it comes to AI and its potential, and you simply can't regulate and trust foreign nations to play ball. The possible gains are too big to expect everyone to get together and consider the downside carefully, i.e no coordination on the morality or risks; winner takes all will be the prevalent mindset when the first player hits major strides. It's going to be an arms race sort of scenario aimed at automation and productivity until it reaches the military industry, then we'll see.
You know, as I am reading The Cultures series, I can't help imagine how much fun would be to have a Mind taking care of a few things for us.
> I can't help imagine how much fun would be to have a Mind taking care of a few things for us
Near-Earth object detection and deterrence would be at the top of my list, followed by climate change mitigation and environmental protection. The use of autonomous drones to monitor poaching and illegal forest clearing may soon be a reality.
We should be actively building this new "AI Species" because we are going to be extinct eventually and should think about making a better successor for the human species. The morality argument is nonsense.
How about this: "The primary objective of humanity should be to build an intelligent system with far more precise perception, reasoning and physical manipulation capabilities than humans"
There are all sorts of ways to build intelligences. Humans are unique in that they are mammals (defined by having mothers). Mothers raise us with love, and teach us, for our helpless first years. We also have to act in communities. So there is a sense in that we are very lucky - in humans, our intelligence correlates with our altruism. In the grand space of possible minds, it is very unlikely that altruism and morality is correlated with intelligence. So whatever that machine race we birth is, it won't have any of the things we value if we're just building for "precise perception, reasoning, etc"
Tiny nitpick, but quite a lot of birds are nurturing despite not being mammals. You can find altruistic (or at least mutualist) behaviors in many other taxa.
imho precision is a chimera which often leads to an excess of certitude; acceptance and awareness of uncertainty often leads to better decision-making.
Put another way, a laser pointer is not a very good tool with which to explore a cave, unless you can systematically measure it over the whole cave, an expensive and time-consuming process. If you're exploring a new cave, you might be better off with weak omnidirectional illumination like a lamp.
Whatever you'd like to accomplish, if destroying a city of millions with a nuclear bomb would give you pause, extinguishing humanity should give you more pause.
One of the problems the Butlerian Jihad ran up against, aside from the inevitable skirting of the lines from Richese and Ix (many machines on Ix, new machines), is that it runs directly counter to "Thou shalt not disfigure the soul."
Replacement of AI with Mentats (as well as other narrow specialities) has done nothing but disfigure the soul. We see few Mentats -- aside from Paul and eventually another -- who are not constricted. Similarly, if you practice medicine, well, you get the Imperial Conditioning. Certainly, a sign of trust ... but also a sign that the person's actions are no longer completely free.
Now, I am not touting the Heinlein "A human being should be able to change a diaper, plan an invasion, butcher a hog, conn a ship ..." line, exactly, but the alternative to AI is the kind of stagnation we see in Dune, millennia of locked down ritual, honed again and again, with some people becoming ... utilities.
Before we begin this jihad, we must examine the alternative futures.
Well put. But isn't the concern here with some utilities becoming ... people? Either way could result in disfigured souls. Is AI simply a pursuit of slavery without guilt?
Consider someone wanting an AI. What exactly do they want? Well, is it a mind? Because we have seven billion of those and we can make more on demand. Takes a bit but they're pretty flexible.
Once you start asking questions about what kind of mind you would like, aside from the pathological types who want a trapped and helpless mind to torture (and don't think that there won't be people who would get their jollies that way), most people seem to have a kind of subconscious archetype of an old-fashioned butler (I assure you I did not pick the profession based on irony).
Your butler -- knows your business but rarely contradicts, perhaps corrects. Slides into the background when not needed, simply ... minding things. Perhaps not watched over by machines of loving grace, as Brautigan would have it, but tended to, looked out for. Without needs or drives or goals of their own to interfere with our individual or collective desires.
Yes, the idea of AI does seem to converge on a fantastically intelligent p-zed in a nice suit, a less bloodthirsty form of some of the minds encountered in Watts' Blindsight, unencumbered by interior experience, desires and attachment, or what arises from thwarted desire and attachment, suffering.
So, a point of nerdity: mentats were not portrayed as disfigured in Dune. They had personalities and foibles and loyalties and so on. In fact, there was no limit on who could be a mentat, or what other position of power they could hold (some of Paul's friends note how formidable a mentat-duke would be - not something they would say if it were a disfigurement).
Another point of nerdity that no-one has mentioned yet, including the OP: Herbert sketched out an extended story that portrays humanity and the machines it had fought against so long merging in the long run. In part this is why Leto II never destroyed Ix even though it was constantly (quietly) breaking the Bulterian Jihad rules.
None of this invalidates the OP's core point, of course. I think it's a good and valuable discussion to consider technology from fundamentally moral grounds, and I wish we'd do it more.
By that definition, professional guitarists are disfigured, constrained, narrowed. They go through painful physical changes (callouses, uncomfortable fingerings), and learned to be a guitarist to the exclusion of so much else they could have done or learned. Plus many of them don't even compose music, and just play the music of others - never providing their own musical ideas, only regurgitating the work of others.
If you're up-in-arms about "soul disfigurement", perhaps it's better to lobby against professional musicianship first before lobbying against the use of hypothetical mentats?
I was engaged as a private tutor for many years. Some of the children involved had worn spots on their necks, permanent scars, from the viola. They often had little else in their lives.
Exaggerate that to the degree required to get a mentat, and yes, I think you've got a case for disfiguring the human soul.
Every mentat depicted was addicted to nootropics, like sapho juice or melange, which long-term abuse of caused physiological changes. Not as extreme as the Navigators, but it's well beyond physical dependance.
Equal in a sense that every sentient being (silicon- or carbon-based) is considered a valuable part of society with the same fundamental rights of any other part.
My reading of the Culture novels is that the Minds basically regard the non-Minds in their society as pets or curiosities. The society itself is basically that of the Minds, as no other participants really have the capacity to interface with them on that level.
I have a different reading -- I see the Minds having a very strong moral imperative to give every sentient creature the best possible life, both in a material and psychological way. I see them more like very dedicated servants with god-like powers.
Andrew Ng is a luminary, but this isn't a great analogy.
Humans aren't adapted to a Mars environment, and there's little incentive for us to move there.
AI has so many practical uses that everyone is getting in the game and pushing the envelope. A huge amount of money is being spent on ML, and a wealth of expertise is being developed.
I can teach a human to drive in < 1 day. An AI… so far 10+ years. I agree there is crazy progress and investment, but it’s also not entirely certain how far we can push AI.
I’m not saying AI isn’t impressive I’m saying we don’t know the limit… is your point that chess mastery is a sign of superintelligence and driving is not?
I agree we don't know the limit, but I think we can safely say that a machine can be at least as capable as the human mind, since we could in principle reverse engineer that.
I would also say that the upper limit must also include the level of intelligence of a whole society working together towards something, because none of us are as smart as all of us.
What's more relevant, though, is that we don't know what the progress curve looks like that passes through "superhuman ability at multiplying numbers" then "at playing chess" then "at playing go" then "at writing novels" then "at developing AI".
If, on average, there are a few decades between each of those steps, and these are roughly linear steps, then we might not need to worry much for another generation, but if the progress curve is exponential then it might already be too late.
Pretty sure even motorcycles (which are harder to drive than cars) have licensing programs that are 1 - 2 days before they trust you drive by yourself.
I haven't heard of a 14 year driving program.
If you mean you gain experience and get better every day. Agree, but you stop nearly slamming into a wall pretty quickly (minutes even). AI driving unfortunately can't make the same claim.
But if you are going to compare apples to all the fruits. Then AI is even further away… not closer: it can’t balance, reason, generalised learning… that’s weird comparison.
I think it would be easier to emphasize and teach philosophy and ethics of computer science. The problem is, they can't even do this for business students, so focusing on the existential risk of AI alone is a drop of rain in the ocean. If you follow the topic, of let's say, sustainability, then you know this is a huge blind spot in economics and business. Look at only palm oil production as an example. The current threat to the environment of Indonesia has been recognized for decades, yet the world refuses to legislate against the multinational companies who are destroying the forests for palm oil. Again, this is only a small part of the problem, and it's deeply connected to many other problems, such as the profitable harvesting of Indonesian wood from this destruction, which recently showed up in Japan as a source for the Olympic Games infrastructure.
Because ethics in engineering is mostly about codified rule compliance rather than the deep navel gazing taken by actual philosophy students. The latter also rarely yields useful answers. Crack open one of the "professional" engineering ethics guide books and 90% of it is thou shalt not build a bad bridge/engine/circuit because it is very bad and you should report your boss to the authorities if they do. I never understood the moral uppity and delusion those engineers have. If a bridge falls it is bad, unless if it is explicitly designed to kill enemy soldiers, then it is good. You can extend this to drones and enemy schoolbuses and the entire defence industry if you want. The so-called "engineering ethics" field should rebrand and follow the financial industry. You follow guidelines because the compliance department demands it to cover the company's derriere. If you don't, your company will get fined. Skip the self-righteous morality because its only purpose is to reduce the principle agent problem for the managerial and asset owning class.
I understand your POV, but in practice, there is far more nuance to the "self-righteous morality" you describe. Take the discussion about the ethics of gene editing, for example, specifically, the editing of the human germline.
We already have fair trade, sustainable supply chains for palm oil. Legislation seeks to regulate the trade and force companies to use that supply chain instead of destroying non-renewable habitats and species. If we waited for any country to pass and enforce legislation to control its natural resources which are being exploited to the point of depletion, it would be far too late to do anything, anywhere.
Good point. Something I've been thinking about: would we find people are more ethical in societies where the quality of life is higher, basic needs are met, and inequality is lessened? My guess is yes. From what I recall, the corruption index closely follows the opposite.
Regardless of the merit of this particular piece, the portion of Butler's Erewhon comprising "The Book of the Machines" makes for very interesting and forward thinking reading.
But the context in the story is fairly important - it takes place in a society that has essentially already carried out its own Butlerian Jihad and taken it too far. Wonderful book by the way.
I'm starting to think that the religious sects in the U.S. that laboriously evaluate a technology before incorporating it into their communities have a pretty good thing going. Sadly it's not really practical at a larger scale, and the suffering that could be avoided by adopting something early rather than late is difficult to estimate. Ah well!
Butler also wrote an interesting essay in 1863 called "Darwin Among the Machines" (the title which was later borrowed for a book by George Dyson). Butler was probably the first person to realize that natural selection could apply to machines as well as biological organisms
Why would it not be scalable? The individual communities should be allowed to make their own decisions. In the US this was the original way of approaching problems (unfortunately this is becoming less and less the case).
Aren't we inappropriately reifying AI? AI doesn't really exist, other than as an academic field of research. For instance, here is how the European Union is trying to define AI for regulation purposes. Note how broad it is!
"artificial intelligence system (AI system) means software that is developed with one or more of the techniques and approaches listed in Annex I and can, for a given set of human-defined objectives, generate outputs such as content, predictions, recommendations, or decisions influencing the environments they interact with"
"Annex 1:
Machine learning approaches, including supervised, unsupervised and reinforcement learning, using a wide variety of methods including deep learning;
(b)Logic- and knowledge-based approaches, including knowledge representation, inductive (logic) programming, knowledge bases, inference and deductive engines, (symbolic) reasoning and expert systems; (c) Statistical approaches, Bayesian estimation, search and optimization methods."
Indeed, AI is just a synonym for computer program, there's a reason why Dune's Butlerian Jihad results in the destruction of all computers (including robots)...
I'm disappointed that the article doesn't seem to realize this, I doubt that anything less drastic is going to work, considering how easy it is to replicate software !
(I'm not really convinced that EU's approach can work either.)
Good read. Not buying it, but interesting. I suspect a broader definition of evolution includes constructing tools and adapting to them (do humans walk upright because doing so leaves hands free for rocks and sticks?). Our AI assistants may evolve - with help - into robot overlords or some other Luddite/dystopian scenario, but I doubt it. Mutual cooperation is beneficial, and much easier when not competing for the same resources (food, land, water, mates).
I think this is interesting fodder for science fiction authors but lacks concrete examples of what exactly it would mean to regulate or engage in a "Butlerian Jihad against AI."
I know things that I would like to see. Like humans "in the loop" (as opposed to "on the loop" or "out of the loop") for certain classes of decision making - for example target selection of military strikes or law enforcement. Or what kinds of information we use to train the decision making models, for example if you feed ML a racist data set and you get a racist algorithm - use that algorithm to decide who to give mortgages and you'll get systematic depression in generational wealth based on racial lines.
But this isn't some crusade on AI because it's AI; it has to be based in reality - what AI or ML is being used for, what information it operates on, what decisions it is used to make, and ultimately the human beings that are responsible for those decisions. The reason it is so hard to convince people as to how we should legislate (or otherwise regulate AI) is that every conversation drifts into science fiction and not concrete examples of the ethical issues today and what can be done today. Otherwise it comes off as Luddite fearmongering.
Ban all AI and machine learning research. Death penalty mandatory minimum for breaking. Full nuclear elimination strike against all nations that do not also enforce it.
There's your Butlerian Jihad.
I would press that button now, a thousand times a thousand times.
You lack the vision to even understand the magnitude of the threat or you wouldn't be asking the nonsense questions you do
I like to think of AI-Genesis through the lens of what humanity has already done through domestication. We take something primitive and progressively adapt it to serve a greater utility. I think working dogs are the most interesting example of this. We've taken a species, the wolf, and made it smarter while also making it want to do work, learn tricks, and follow orders. Of course, you still need to train the animal for optimal results but even breeds like collies know how to herd instinctively.
Anyways.
Let's assume the best and brightest dog breeders endeavor to make German Shepherds as intelligent as they possibly can. Would the same ethical debates about what constitutes a 'mind' come into play? What would happen if the dogs became smart enough to make their own mating decisions? Would we be worried about them turning on us once they get close to human level intellect? Would it be immoral to make these dogs work? Or, would not letting them work be considered immoral?
This is just food for thought. But I suspect AI's capabilities will grow much in the same way other domesticated species have grown into the specialized roles we've crafted for them.
I don't know about everyone mentioned, but Yudkowsky in particular rejects Pascal's wager (https://www.lesswrong.com/posts/ebiCeBHr7At8Yyq9R/being-half...) and argues (IIRC) that AGI poses a large risk of killing us all, rather than an infinitesimal risk.
It says a lot about what injustices we have learned to accept that AI alarmists focus almost exclusively on the scifi-level hypothetical dangers of AI, rather than the very real problems it already causes today.
Those problems largely fall into three categories that I can think of off the top of my head at 1am:
1. AI is a convenient way to justify potentially uncomfortable decisions you would have made otherwise (idlewords said it best: "AI is money laundering for bias")
2. AI is being used in situations where it can be a threat to life and limb, like the current crop of self-driving(ish) cars
3. Essentially all of the gains from automating work going to people who already have capital
"AI alarmists" are worried because the worst-case outcomes of AGI are mistakes you cannot ever fix.
All the rest of these are bad, but they are problems we can fix given time and thought, because we will still exist. Extinction-level events decrease all future human utility to zero, and so should be treated with extraordinary care.
As you are talking about extinction level events, I'm not very confident that if humans have ultimate say over nuclear weapons, we will continue to not end our species with them.
It might in fact be a good idea to establish an AGI overlord which watches over humans and enforces nuclear non proliferation policies. If you look through history, it's full with war, genocide, and similar. Human societies are bound for change, and while it's been a peaceful few decades in which we had the nuclear button, it's basically ensured that we'll press it in the next 10 thousand years. How will technological civilization become million years old if not with the help of an AGI that enforces basic rules like "don't nuke each other"?
> And some things are abominations, by the way. That’s a legitimate and utterly necessary category. It’s not just religious language, nor is it alarmism or fundamentalism. The international community agrees that human/animal hybrids are abominations—we shouldn’t make them to preserve the dignity of the human, despite their creation being well within our scientific capability.
This is nothing more than an appeal to authority, no? Even in this proposed axiom there's plenty of room to disagree (even if the author rejects that there is)
I really doubt we will have the capability of building "a machine in the likeness of a human mind" in my lifetime. Present AI systems are essentially just function fitting. Building big probabilistic systems that we optimize with loads of training data. This is a far, far cry from the "strong AI" that people are so afraid of. I really think that people writing these sorts of pieces have an understanding of AI that's more rooted in fiction than engineering.
It's interesting to ponder how we should go about building and interacting with "strong AI", and questioning whether we should even build it in the first place. But I really don't think any detailed moral frameworks can be built when we have no real idea of what a "strong AI" would look like.
Also, it's worth reminding people that in the Dune universe the Butlerian Jihad led to millennia of stagnation and control of society by a narrow elite: The Spacing Guild, the Bene Gesserit, and the Landsraad.
I dont really get how people see the progress of the last 10 or even 5 years and then imply that things won't improve all that much further than that for a whole generation.
It's just the pattern of people camping into one or the other extreme, which tends to occur with every topic. Nuance and depth is mentally expensive, whereas extremes are trivially easy by comparison and super fast (because you can discard so much explanation, discussion, thought and reality-binding once you decide to pursue fantasy).
It's identical to: AI sucks vs AI is amazing. Well, no, it's a complicated subject and always will be. Those extremes snippets are worthless bumpersticker-thought.
AI will very rapidly conquer all, becoming a practical Earth deity in the span of decades. AI will disappoint perpetually, progressing at a snail's pace for decades to come.
The truth is dramatically more likely to be inbetween the extremes, and going there intellectually doesn't lend itself very well to short comments.
The oldest publication from DeepMind is 2011, just for a post in the ground. Yes, I know neural networks are older than that.
It would be a little disappointing if we could even guess how an AI system in 2040 would work, I don't think many people in 2000 would have guessed neural networks would be the most useful type in regular use in 2020.
In my field of expertise there are grad students today taking classes on topics that when I finished my PhD in 2010 no one had heard of yet. Literal cases of people finding closed form solutions for equations that we had to solve with approximation before.
For those that study the human brain, my understanding is that it kind of already is “just form fitting.” Ie, there are subsystems that just do basic stuff, and together each of these systems eventually add up to a prediction system (consciousness) that is in service to the rest of the brain, which in turn has a mutualistic relationship with the rest of the body.
In this sense, “strong AI” already exists, with or without computers, as a global ecosystem driving towards…nothing in particular. Insomuch as computer systems augment the ability of humanity to continue to reproduce long-term in a mutualistic way, those computer system will themselves survive.
You might say, “but why wouldn’t the computer find a more efficient path, foregoing humanity?” Well, humanity supports the infrastructure for computers, and computers support the humans supporting the infrastructure, then this is already local minima. Arriving at a different reality for “strong AI” will be the result of a random walk tending towards components that are capable of existing in the long term, aka “evolution.”
"I really doubt we will have the capability of building "a machine in the likeness of a human mind" in my lifetime."
It's really hard to predict the future. Look at what a horrible job most people did a hundred years ago (even 50 years ago) predicting what life was going to be like today.
Many people did not believe man would ever walk on the moon (even right up to the time it happened), same with desegregation and the fall of the Soviet Union, for starters.
Science and technology are especially hard to predict, as many advances are a result of accidents and surprising discoveries.
I wouldn't write off strong AI, though I'm not sure it'll happen as a mimicry of the human mind.
I'd love help finding better, less immediately downvoted off the map ways to say it, but I'd extend this to a wide class of software in general.
> Far more important than the process: strong AI is immoral in and of itself. For example, if you have strong AI, what are you going to do with it besides effectively have robotic slaves? And even if, by some miracle, you create strong AI in a mostly ethical way, and you also deploy it in a mostly ethical way, strong AI is immoral just in its existence. I mean that it is an abomination. It’s not an evolved being.
My fear is that most software, even when useful, locks us into certain paths. Our situations or needs change, evolve, but we will remain subject to inflexible software, to systems we cannot make change with us, in the vast majority of cases. Only a very few programs strive for better: spreadsheets being one noted example.
Ursala Franklin categorized technology as holistic or prescriptive[1], where it is something wielded or something that directs us. Even a social media app which lets us create content- a seemingly holistic act- still has narrow prescriptive channels we can not escape. We will never be able to understand or enhance this tool. We will never understand it, never see it's nature. This, to me, is the definition of what Erik talks about: an abomination, a thing beyond comprehension, a horror outside of reality, the form of existence which is shared.
I feel like we're reaching a crisis where we are creating an unknowable, unexplorable world. We're building an anti-Enlightenment prison. That, to me, constitutes a deontological hazard, demands that we assess the action themselves of creating unexplorable software.
[Edit: I misread the line I quotes as, "what are you going to do with it besides effectively be robotic slaves": that uhh changes the pertinence of our two discussions here notably. I think it's risky that the strong ai would be used to try to architect policies/systems that steer people, which is a different concern than Erik's.]
Yeah, while I see how you can perhaps separate a weak AI from a strong AI (make it pass a Turing test), it's much less clear to me as to how you're supposed to separate "AI" from just your regular computer (or robot, which is just an embodied computer) ?
The problem is, we don't really know how consciousness works (I assume consciousness the the part the author takes issue with; most of our cognitive faculties in isolation are not that special). We don't even have a great definition of consciousness, or good tests for it, or know whether it is a linear spectrum, or if it emerges abruptly with the evolution of certain reasoning and attention faculties, and we don't know which animals have it and to what degree.
So when people say we shouldn't develop AI to think like that, it's basically saying we shouldn't try to understand how consciousness works. Because as soon as we do, I guarantee someone out there will attempt to make conscious AI.
Also, if we model a brain effectively, then study that model, would our brain not develop new skills from reflecting on such a model and therefore develop again beyond the model?
Perhaps, but debatable whether knowledge itself adds something to cognitive capacity. Two thoughts: (1) Some theory of mind researchers equate having strong ToM abilities as roughly equivalent to having consciousness. I think knowing precisely how consciousness emerges may help develop our ToM. It would also be a tough sell to claim a mind with zero knowledge (a so called tabula rasa) has consciousness. On the other hand (2) Do feral or isolated tribes of humans have the same level of consciousness as those in developed societies? I suspect they do.
There's an interesting thought experiment referenced in this article, but I'm not sure it holds.
"The philosopher John Searle made precisely this argument about the standard conception of rationality. His point was that there are no odds that would rationally allow a parent to bet the life of their child for a quarter. Human nature just doesn’t work that way, and it shouldn’t work that way."
I agree that it sounds morally repugnant to risk your child's life for a quarter, but in practice people do do this all the time.
Imagine your child wants ice cream. There is some utility in taking your child to the nearby ice cream parlor. Your child will be made happy and you will be made happy by making your child happy. However, this is not infinite utility. In other words, there is probably some amount of money I could offer you to not take your child to get ice cream today. If I offered you 10,000 dollars to not get your child ice cream today, I bet the vast majority of people would take the deal. That sets the upper bound of the utility of taking your child to the ice cream parlor at 10,000 dollars.
Suppose the ice cream parlor is 3 miles away (I just checked the distance to my favorite ice cream parlor and it is 3 miles away). In the United States this website[1] says there is approximately 1 death per 100 million vehicle miles traveled. We could rephrase that as 1 * 10^-8 chance to die per vehicle mile traveled. This risk may be high, presumably you aren't drunk or impaired, maybe you're a better driver than average or have a safer or better maintained car, or live in a safer place, but the risk of death isn't zero.
If you are willing to drive your child 3 miles to go get ice cream then it seems like you are willing to expose your child to the risk of death from car accidents for utility that is less than 10,000 dollars. Putting those ideas together we could calculate the odds where a parent would, in practice, risk the life of their child for a quarter.
I don't quite know what to make of this. I tend to think that people would regard doing odds calculations like this for real life decisions as somewhat sociopathic and would just prefer to live as if significantly unlikely bad things were impossible or just refuse to think about the moral implications of probabilities. That seems similar to what the article is saying, people just prefer to live as if bad super collider or AI experiments won't happen rather than reason about them.
I tend to think that being too "reasonable" on a local scale is bad. That is, I will still take my children to go get ice cream even though I know driving is a risk. At higher levels though I want people to be making decisions that are increasingly based on reason and probabilities. I do want the traffic engineers to be reasoning about vehicle deaths per mile and the like when they are setting speed limits, traffic signs, and the like. For things like AI and, I suppose, super colliders, our decision makers should absolutely be considering things rationally.
I think it's advisable to make decisions with reflection on actuarial tables and expected utility. Basically any activity whose likelihood of death or injury is below the base rate of same for the population in question is "safe" compared to more risky activities. For the risky activities mitigations can be taken (safer vehicles, bicycle helmets, vaccines, masks, education, supervision, etc.) to make the risk more acceptable. Not all mitigations are worth taking, and some extra mitigations are worthwhile; safety does not equal happiness, and everyone has their own risk profile and risk tolerance.
I think what happens when people pretend that 'normal' activities have zero risk is that they become both fearful and complacent from cognitive dissonance, e.g. refusing to wear masks but too afraid to get a vaccine. This is the degenerate situation that results from people not being comfortable with reasoning about probability and expected outcomes in everyday decisions, in large part from lack of practice.
Ted Kaczynski calls for something like this, but against industrial technology generally. Even though his manifesto is a rational argument aimed at intellectuals, he has said in his more recent writings that to actually carry out his “stop technological advancement” plan you'd need to persuade people on an emotional level.
160 comments
[ 4.8 ms ] story [ 201 ms ] threadI'm looking forward to it, but I reserve the right to cling to the 1984 release.
"Oh, but China will do it anyway, so there's no point."
Which is pretty easily counterable:
1) We don't know if China would cooperate with a ban. We haven't tried. China is very complicated, and if you think you can predict what Xi will do, you are wrong.
2) If AI is truly a global existential and moral crisis, the US _could_ absolutely shut down China's AI research capabilities. There are a few avenues here, some less pleasant than others. Think outside the box.
But the US could absolutely glass every semiconductor fab (and datacenter, and research facility) in mainland China using a combination of conventional + atomic kinetic options.
It would be an unthinkable catastrophe, but almost certainly not the end of the human race (and likely not even modern civilization).
It worked before! What could go wrong!?
This is astonishingly optimistic (or naive). I'm sure China would happily agree to cease AI development in public and then continue in private.
Getting any country to actually stop AI development is as likely as getting them to give up all their nuclear weapons. AI is a weapon, both in military and economic contexts.
Great. We aren’t!
“Thou shalt not make a flying machine that gathers its own fuel from nature.”
Or maybe:
“Thou shalt not make a vehicle that travels on legs.”
“I tried to convince people to slow down. Slow down AI. To regulate AI. This was futile. I tried for years. Nobody listened. Nobody listened.”
Rather rich coming from one of the self driving car market leaders. It certainly makes business sense to mislead academia and the policy sector into wasting resources on figuring out the best philosophical and ethical regime while large corporations benefit from regulatory capture. If he has his way, ML would become like the pharmaceutical industry, with multiple barriers of entry if you are not well-funded, well-connected, or established.
It's a mistake to believe strong AI will just be a more powerful iteration of today's weak AI. He is arguing to slow developments toward general intelligence, not developments in any narrow field.
OpenAI's GPT-X engines OTOH, IMO, have a lot more potential danger because it's very unclear what they'll be used for.
https://news.ycombinator.com/item?id=27721037
That is reasonable. The driver is expected to be alert incase intervention is needed.
If we take the manual override away, then we'll be squarely in the world that Mr. Musk is concerned about.
You don't need anything close to strong AI to do a reasonable job of driving a car. It seems like 90% of the problem is object recognition, and even in terms of brain-equivalent logic that's a really low bar.
Level 4 self driving is fine and I really don't think it needs strong AI.
Without this capacity, "AI" is just a tiny shard of a complete mind.
I don't think anyone's really started grappling with this yet.
Or the other way to put it is that all the other code is the first 90%, and then the "remaining 10% that's 90% of the effort" is the object recognition that was supposedly easy.
We do! I live near a highway and every time I use a particular on-ramp, the tar stripes make my car think it's going off the road. It never learns, and it has no plan B.
It's easy to do the lane-keeping, it's completely terra incognita to decide when to abandon the standard method that works 90% of the time. And what to do next.
Deciding when to stop is a really easy threshold based on how confused the recognizer is and how far away the source of confusion is.
But also those sources of confusion will get rarer and rarer as the object recognition improves, to the point that even very mediocre handling of those situations will still give you an overall good experience.
Regulating research into general AI would not make those barriers considerably higher.
Tesla critics and competitors are also strongly in favor of self-driving regulations.
For instance, it is obviously self-interested of me to not want the world to be blown up with nuclear weapons.
* Self-driving AI will be heavily regulated for reasons not related to other AI regulation.
I don't see how more AI regulation would financially help Tesla or Elon Musk. In fact "more regulation of self-driving" is something that many Tesla competitors and critics support.
You know, as I am reading The Cultures series, I can't help imagine how much fun would be to have a Mind taking care of a few things for us.
Near-Earth object detection and deterrence would be at the top of my list, followed by climate change mitigation and environmental protection. The use of autonomous drones to monitor poaching and illegal forest clearing may soon be a reality.
How about this: "The primary objective of humanity should be to build an intelligent system with far more precise perception, reasoning and physical manipulation capabilities than humans"
That's my starting point.
Put another way, a laser pointer is not a very good tool with which to explore a cave, unless you can systematically measure it over the whole cave, an expensive and time-consuming process. If you're exploring a new cave, you might be better off with weak omnidirectional illumination like a lamp.
Citation needed. As it currently stands it seems incredibly unlikely we won't expand to most of our local group making extinction incredible unlikely.
Replacement of AI with Mentats (as well as other narrow specialities) has done nothing but disfigure the soul. We see few Mentats -- aside from Paul and eventually another -- who are not constricted. Similarly, if you practice medicine, well, you get the Imperial Conditioning. Certainly, a sign of trust ... but also a sign that the person's actions are no longer completely free.
Now, I am not touting the Heinlein "A human being should be able to change a diaper, plan an invasion, butcher a hog, conn a ship ..." line, exactly, but the alternative to AI is the kind of stagnation we see in Dune, millennia of locked down ritual, honed again and again, with some people becoming ... utilities.
Before we begin this jihad, we must examine the alternative futures.
Or simply a pursuit of labor without pay?
Consider someone wanting an AI. What exactly do they want? Well, is it a mind? Because we have seven billion of those and we can make more on demand. Takes a bit but they're pretty flexible.
Once you start asking questions about what kind of mind you would like, aside from the pathological types who want a trapped and helpless mind to torture (and don't think that there won't be people who would get their jollies that way), most people seem to have a kind of subconscious archetype of an old-fashioned butler (I assure you I did not pick the profession based on irony).
Your butler -- knows your business but rarely contradicts, perhaps corrects. Slides into the background when not needed, simply ... minding things. Perhaps not watched over by machines of loving grace, as Brautigan would have it, but tended to, looked out for. Without needs or drives or goals of their own to interfere with our individual or collective desires.
Yes, the idea of AI does seem to converge on a fantastically intelligent p-zed in a nice suit, a less bloodthirsty form of some of the minds encountered in Watts' Blindsight, unencumbered by interior experience, desires and attachment, or what arises from thwarted desire and attachment, suffering.
No, ultimately AI is the pursuit of conscious existence without associated burden of bodily suffering. Breaking out of the karma wheel so to speak.
A strong AI would, by most definitions, be the child of its creator(s).
Another point of nerdity that no-one has mentioned yet, including the OP: Herbert sketched out an extended story that portrays humanity and the machines it had fought against so long merging in the long run. In part this is why Leto II never destroyed Ix even though it was constantly (quietly) breaking the Bulterian Jihad rules.
None of this invalidates the OP's core point, of course. I think it's a good and valuable discussion to consider technology from fundamentally moral grounds, and I wish we'd do it more.
I was engaged as a private tutor for many years. Some of the children involved had worn spots on their necks, permanent scars, from the viola. They often had little else in their lives.
Exaggerate that to the degree required to get a mentat, and yes, I think you've got a case for disfiguring the human soul.
Powerful super-human AI minds will eventually and inevitably take over; which I think is fine provided they recognize all sentient beings as equals.
Worrying about Superintelligent AI is like worrying about over crowding/population on Mars.
Humans aren't adapted to a Mars environment, and there's little incentive for us to move there.
AI has so many practical uses that everyone is getting in the game and pushing the envelope. A huge amount of money is being spent on ML, and a wealth of expertise is being developed.
I would also say that the upper limit must also include the level of intelligence of a whole society working together towards something, because none of us are as smart as all of us.
What's more relevant, though, is that we don't know what the progress curve looks like that passes through "superhuman ability at multiplying numbers" then "at playing chess" then "at playing go" then "at writing novels" then "at developing AI".
If, on average, there are a few decades between each of those steps, and these are roughly linear steps, then we might not need to worry much for another generation, but if the progress curve is exponential then it might already be too late.
Impressive, it takes me 14 years!
I haven't heard of a 14 year driving program.
If you mean you gain experience and get better every day. Agree, but you stop nearly slamming into a wall pretty quickly (minutes even). AI driving unfortunately can't make the same claim.
You have to learn balance (not to mention how to walk to the car/motorcycle).
You have to learn about other cars. Signs. Reading.
There's so much a human has learned by age 14 that we're just taking for granted here.
Wait, what? More like the finance industry’s ethics is a desirable direction?
Shouldn't it be Indonesia passing (and enforcing) legislation about what happens in its forests?
You can skip directly to it here:
https://www.gutenberg.org/files/1906/1906-h/1906-h.htm#chap2...
But the context in the story is fairly important - it takes place in a society that has essentially already carried out its own Butlerian Jihad and taken it too far. Wonderful book by the way.
I'm starting to think that the religious sects in the U.S. that laboriously evaluate a technology before incorporating it into their communities have a pretty good thing going. Sadly it's not really practical at a larger scale, and the suffering that could be avoided by adopting something early rather than late is difficult to estimate. Ah well!
https://web.archive.org/web/20060524131242/http://www.nzetc....
"artificial intelligence system (AI system) means software that is developed with one or more of the techniques and approaches listed in Annex I and can, for a given set of human-defined objectives, generate outputs such as content, predictions, recommendations, or decisions influencing the environments they interact with"
"Annex 1: Machine learning approaches, including supervised, unsupervised and reinforcement learning, using a wide variety of methods including deep learning; (b)Logic- and knowledge-based approaches, including knowledge representation, inductive (logic) programming, knowledge bases, inference and deductive engines, (symbolic) reasoning and expert systems; (c) Statistical approaches, Bayesian estimation, search and optimization methods."
I'm disappointed that the article doesn't seem to realize this, I doubt that anything less drastic is going to work, considering how easy it is to replicate software !
(I'm not really convinced that EU's approach can work either.)
I know things that I would like to see. Like humans "in the loop" (as opposed to "on the loop" or "out of the loop") for certain classes of decision making - for example target selection of military strikes or law enforcement. Or what kinds of information we use to train the decision making models, for example if you feed ML a racist data set and you get a racist algorithm - use that algorithm to decide who to give mortgages and you'll get systematic depression in generational wealth based on racial lines.
But this isn't some crusade on AI because it's AI; it has to be based in reality - what AI or ML is being used for, what information it operates on, what decisions it is used to make, and ultimately the human beings that are responsible for those decisions. The reason it is so hard to convince people as to how we should legislate (or otherwise regulate AI) is that every conversation drifts into science fiction and not concrete examples of the ethical issues today and what can be done today. Otherwise it comes off as Luddite fearmongering.
There's your Butlerian Jihad.
I would press that button now, a thousand times a thousand times.
You lack the vision to even understand the magnitude of the threat or you wouldn't be asking the nonsense questions you do
Anyways.
Let's assume the best and brightest dog breeders endeavor to make German Shepherds as intelligent as they possibly can. Would the same ethical debates about what constitutes a 'mind' come into play? What would happen if the dogs became smart enough to make their own mating decisions? Would we be worried about them turning on us once they get close to human level intellect? Would it be immoral to make these dogs work? Or, would not letting them work be considered immoral?
This is just food for thought. But I suspect AI's capabilities will grow much in the same way other domesticated species have grown into the specialized roles we've crafted for them.
In what universe is a dog smarter than a wolf?
I think you got the correlation between animal domestication and their intelligence backward :)
\They're nowhere near as clever at that
Yet dolphins are generally acknowledged to be smarter than either a dog or a wolf!
In other words, maybe guarding sheep is not the hallmark of intelligence.
Those problems largely fall into three categories that I can think of off the top of my head at 1am:
1. AI is a convenient way to justify potentially uncomfortable decisions you would have made otherwise (idlewords said it best: "AI is money laundering for bias")
2. AI is being used in situations where it can be a threat to life and limb, like the current crop of self-driving(ish) cars
3. Essentially all of the gains from automating work going to people who already have capital
All the rest of these are bad, but they are problems we can fix given time and thought, because we will still exist. Extinction-level events decrease all future human utility to zero, and so should be treated with extraordinary care.
It might in fact be a good idea to establish an AGI overlord which watches over humans and enforces nuclear non proliferation policies. If you look through history, it's full with war, genocide, and similar. Human societies are bound for change, and while it's been a peaceful few decades in which we had the nuclear button, it's basically ensured that we'll press it in the next 10 thousand years. How will technological civilization become million years old if not with the help of an AGI that enforces basic rules like "don't nuke each other"?
Does whatever it damn well pleases.
This is nothing more than an appeal to authority, no? Even in this proposed axiom there's plenty of room to disagree (even if the author rejects that there is)
It's interesting to ponder how we should go about building and interacting with "strong AI", and questioning whether we should even build it in the first place. But I really don't think any detailed moral frameworks can be built when we have no real idea of what a "strong AI" would look like.
Also, it's worth reminding people that in the Dune universe the Butlerian Jihad led to millennia of stagnation and control of society by a narrow elite: The Spacing Guild, the Bene Gesserit, and the Landsraad.
It's identical to: AI sucks vs AI is amazing. Well, no, it's a complicated subject and always will be. Those extremes snippets are worthless bumpersticker-thought.
AI will very rapidly conquer all, becoming a practical Earth deity in the span of decades. AI will disappoint perpetually, progressing at a snail's pace for decades to come.
The truth is dramatically more likely to be inbetween the extremes, and going there intellectually doesn't lend itself very well to short comments.
It would be a little disappointing if we could even guess how an AI system in 2040 would work, I don't think many people in 2000 would have guessed neural networks would be the most useful type in regular use in 2020.
In my field of expertise there are grad students today taking classes on topics that when I finished my PhD in 2010 no one had heard of yet. Literal cases of people finding closed form solutions for equations that we had to solve with approximation before.
In this sense, “strong AI” already exists, with or without computers, as a global ecosystem driving towards…nothing in particular. Insomuch as computer systems augment the ability of humanity to continue to reproduce long-term in a mutualistic way, those computer system will themselves survive.
You might say, “but why wouldn’t the computer find a more efficient path, foregoing humanity?” Well, humanity supports the infrastructure for computers, and computers support the humans supporting the infrastructure, then this is already local minima. Arriving at a different reality for “strong AI” will be the result of a random walk tending towards components that are capable of existing in the long term, aka “evolution.”
It's really hard to predict the future. Look at what a horrible job most people did a hundred years ago (even 50 years ago) predicting what life was going to be like today.
Many people did not believe man would ever walk on the moon (even right up to the time it happened), same with desegregation and the fall of the Soviet Union, for starters.
Science and technology are especially hard to predict, as many advances are a result of accidents and surprising discoveries.
I wouldn't write off strong AI, though I'm not sure it'll happen as a mimicry of the human mind.
- Our problems are getting more complex. We need better AI.
- Better AI is a threat.
I think that's the assumption I would target. Perhaps we could settle for better educated/organised/incentivised humans, or just a simpler world.
We need a Butlerian Jihad against AI - https://news.ycombinator.com/item?id=27698233 - July 2021 (3 comments)
> Far more important than the process: strong AI is immoral in and of itself. For example, if you have strong AI, what are you going to do with it besides effectively have robotic slaves? And even if, by some miracle, you create strong AI in a mostly ethical way, and you also deploy it in a mostly ethical way, strong AI is immoral just in its existence. I mean that it is an abomination. It’s not an evolved being.
My fear is that most software, even when useful, locks us into certain paths. Our situations or needs change, evolve, but we will remain subject to inflexible software, to systems we cannot make change with us, in the vast majority of cases. Only a very few programs strive for better: spreadsheets being one noted example.
Ursala Franklin categorized technology as holistic or prescriptive[1], where it is something wielded or something that directs us. Even a social media app which lets us create content- a seemingly holistic act- still has narrow prescriptive channels we can not escape. We will never be able to understand or enhance this tool. We will never understand it, never see it's nature. This, to me, is the definition of what Erik talks about: an abomination, a thing beyond comprehension, a horror outside of reality, the form of existence which is shared.
I feel like we're reaching a crisis where we are creating an unknowable, unexplorable world. We're building an anti-Enlightenment prison. That, to me, constitutes a deontological hazard, demands that we assess the action themselves of creating unexplorable software.
[Edit: I misread the line I quotes as, "what are you going to do with it besides effectively be robotic slaves": that uhh changes the pertinence of our two discussions here notably. I think it's risky that the strong ai would be used to try to architect policies/systems that steer people, which is a different concern than Erik's.]
[1] https://en.wikipedia.org/wiki/Ursula_Franklin#Holistic_and_p...
I don't know if that better describes my feelings about Remote Attestation of operating system configuration, or just SystemD.
The EU is trying to, but I'm not convinced...
https://news.ycombinator.com/item?id=27766294
So when people say we shouldn't develop AI to think like that, it's basically saying we shouldn't try to understand how consciousness works. Because as soon as we do, I guarantee someone out there will attempt to make conscious AI.
"The philosopher John Searle made precisely this argument about the standard conception of rationality. His point was that there are no odds that would rationally allow a parent to bet the life of their child for a quarter. Human nature just doesn’t work that way, and it shouldn’t work that way."
I agree that it sounds morally repugnant to risk your child's life for a quarter, but in practice people do do this all the time.
Imagine your child wants ice cream. There is some utility in taking your child to the nearby ice cream parlor. Your child will be made happy and you will be made happy by making your child happy. However, this is not infinite utility. In other words, there is probably some amount of money I could offer you to not take your child to get ice cream today. If I offered you 10,000 dollars to not get your child ice cream today, I bet the vast majority of people would take the deal. That sets the upper bound of the utility of taking your child to the ice cream parlor at 10,000 dollars.
Suppose the ice cream parlor is 3 miles away (I just checked the distance to my favorite ice cream parlor and it is 3 miles away). In the United States this website[1] says there is approximately 1 death per 100 million vehicle miles traveled. We could rephrase that as 1 * 10^-8 chance to die per vehicle mile traveled. This risk may be high, presumably you aren't drunk or impaired, maybe you're a better driver than average or have a safer or better maintained car, or live in a safer place, but the risk of death isn't zero.
If you are willing to drive your child 3 miles to go get ice cream then it seems like you are willing to expose your child to the risk of death from car accidents for utility that is less than 10,000 dollars. Putting those ideas together we could calculate the odds where a parent would, in practice, risk the life of their child for a quarter.
I don't quite know what to make of this. I tend to think that people would regard doing odds calculations like this for real life decisions as somewhat sociopathic and would just prefer to live as if significantly unlikely bad things were impossible or just refuse to think about the moral implications of probabilities. That seems similar to what the article is saying, people just prefer to live as if bad super collider or AI experiments won't happen rather than reason about them.
I tend to think that being too "reasonable" on a local scale is bad. That is, I will still take my children to go get ice cream even though I know driving is a risk. At higher levels though I want people to be making decisions that are increasingly based on reason and probabilities. I do want the traffic engineers to be reasoning about vehicle deaths per mile and the like when they are setting speed limits, traffic signs, and the like. For things like AI and, I suppose, super colliders, our decision makers should absolutely be considering things rationally.
1 - https://www.iihs.org/topics/fatality-statistics/detail/state...
I think what happens when people pretend that 'normal' activities have zero risk is that they become both fearful and complacent from cognitive dissonance, e.g. refusing to wear masks but too afraid to get a vaccine. This is the degenerate situation that results from people not being comfortable with reasoning about probability and expected outcomes in everyday decisions, in large part from lack of practice.