Am I the only one who immediately writes off authors that mention Microsoft's Syndey as something scary/to be worried about? The whole "it said <scary thing>" seems overblown to me
Yes, was my first reaction too, but then my 2nd reaction...
To put it as well as I can : after my 2nd reaction I went on to read the now available 2nd part, which also linked to this Open Letter by Yudkowski a month ago, which spells out my 2nd reaction probably better than I could :
> I agree that current AIs are probably just imitating talk of self-awareness from their training data. But I mark that, with how little insight we have into these systems’ internals, we do not actually know.
Also, I am not convinced that our current research won't hit a brick wall well before super-intelligence, which would then require a radically different approach... but why are we willing to gamble with something as dangerous as this ?!
I agree that we don't know exactly what is happening at any time in the model, but there's limits.
Sydney has no genitalia, no hormones, no sex drive. It does not "desire" someone. It may have a theory of mind, but clearly any AI saying that it loves someone is just working from its training data. The article treats this as though there was an actual intention there, rather than just a model repeating its training [0].
Just as obviously, this is a journo playing to our human desire to prevent aliens from running away with our women - any thought of non-humans breeding with humans triggers something pretty deep in us. Well played, sir. Take your paycheck, sit down, and please stop pushing the buttons.
The AIs are not going to take over the world in the sense that this article talks about. They are not invented by supernatural forces beyond our ken, as it suggests. This is a human emotional reaction to things that appear to be intelligent. I agree that this generation of AI is going to hit a brick wall and the shortcomings will become obvious, but the emotional reaction will be the same. I think it will be another 10-20 years before we solve some of the technical challenges in this generation of AI. But we're having the discussion about AI, and the emotional reaction to it, now. Which is probably good, because by the time we develop actual AI we'll have been living with our current level of "almost-AI" since we were kids and we'll be used to it.
And to answer the question: Every technology is dangerous. That has never stopped us from using it.
[0] Though, of course, that raises questions about what an "intention" is in human terms, too. Are we just "repeating our training"? I think not, at least for sex & love - there are too many hormones involved.
Does a spider desire to build a web? Maybe, maybe not, but it takes actions to try to build one. Is human desire a real thing, or is it an illusion? Some think that consciousness itself is an illusion. What is human sensation but the world sensing itself?
Does a country have desires? An economy? Do the millions of transactions and price changes and signals going between people and companies mean that an economy has thoughts, in some sense? When an economy reacts to a war in a foreign land like an ant flinching away from something hot, is it feeling pain in some very primitive sense?
Sure, it's entirely possible that there's no real sensation behind an LLM, or rather, it senses and reacts to text input like a bacterium reacting to chemical nutrient gradients or an ant reacting to food. But at what level of mental complexity does consciousness and true desire arise? I think it's a continuum, and to think that there's something special about biological intelligence vs machine intelligence is to believe in the supernatural. We wouldn't say that aliens don't have true desires because they have genetic material based on something other than DNA, or use something other than glucose for energy in their bodies.
I do think you're right that current LLMs proclaiming love likely don't understand what they're saying. But at some point we're going to build machine intelligences that do, and it may be hard to tell when we cross that line. People used to worry a lot more about the ethics of that.
There is another technology that produced a similarly startling mystery -- the ouja board, introduced in 1890. That one, we figured out. The messages came from the depths of the human minds attached to the technology. The answers to the three numbered questions presented in the substack article are not known to me, but, of course, the nature of the presentation inclines me to think that the answers are likely to explain more about the operation of human minds when they attempt to collaborate in teams connected by unfathomed autonomous technology than they explain about the technology itself.
First sentence says:
> The Internet and its consequences have been a disaster for the human race.
Root cause analysis gets me this far: The Internet is a consequence of the human race. The human race loves consequences. We cannot resist the allure of the search for advantageous consequences, even though we have declining brain size and increasing need for wisdom. It looks to me as if the fight for the wheel will include several factions of humans, several species of AI, and a jumble of alliances, hybrids, coalitions, strategies, conspiracies, and standoffs amongst them. The future is going to be just like a trip to the movies today -- nothing but sequels.
Don't we? Maybe someone with more architectural knowledge of LLM systems can fill in the gaps in my knowledge, but where's the feedback loop that would allow actual self-awareness to develop? FAFAIK, LLM's are feed-forward networks. There are neither small cycles spanning a few nodes, nor integral cycles allowing the model to see what it's outputting -- the only feedback it gets is through the training process, which does not act as a mirror.
I struggle to see how self-awareness would ever manifest from a system that cannot experience its own output.
> Maybe someone with more architectural knowledge of LLM systems can fill in the gaps in my knowledge, but where’s the feedback loop that would allow actual self-awareness to develop? FAFAIK, LLM’s are feed-forward networks. There are neither small cycles spanning a few nodes, nor integral cycles allowing the model to see what it’s outputting – the only feedback it gets is through the training process, which does not act as a mirror.
This is, AFAIK, true of bare LLMs. Systems consisting of LLMs and a control interface that does reprompting which includes the earlier LLM responses (e.g., chat interfaces like ChatGPT) do provide the model with its output as feedback and show learning within the loop (though obviously this learning doesn’t reach back to the base model). The cognitive features of such a loop, such as they are, may well be different than those of a bare LLM considered in isolation.
I still think Yudkowsky is a crank and that taking him seriously is a bigger danger than AI.
I think Sydney is worrisome in that it shows that even Microsoft, spending tons of money, can’t (or doesn’t want to) stop LLMs from going “off-script”. If your concern is “will Microsoft et. al be careful with deploying new tech”, Sydney was evidence towards “no”. You could guess that the PR incident will have encouraged them to be more careful in the future, but I’m not confident in that.
> Out there, said all the old tales from all the old cultures, is another realm. It is the realm of the demonic, the ungodly and the unseen: the ‘supernatural.’ Every religion and culture has its own names for this place. It lies under the barrows and behind the veil, it emerges in the thin places where its world meets ours. And the forbidden question on all of our lips, the one which everyone knows they mustn’t ask, is this: what if this is where these things are coming from?
We are doing this because our individual incentives are not aligned with those of the group. Moloch is the god of misaligned incentives, of coordination problems, of arms races and races to the bottom. It's a property of any system with selfish actors, which means any system that humans are part of, and probably any intelligent being that arose out of natural selection would have been hypothetically been part of.
From `Meditations on Moloch`:
> There’s a passage in the Principia Discordia where Malaclypse complains to the Goddess about the evils of human society. “Everyone is hurting each other, the planet is rampant with injustices, whole societies plunder groups of their own people, mothers imprison sons, children perish while brothers war.”
> The Goddess answers: “What is the matter with that, if it’s what you want to do?”
Looks like it worked on you then. Good writing evokes feelings in its readers but as you've discovered, not all feelings feel good. Bad or good though, feelings make things stick. As you worship whichever deity you relate to (even if that deity is technology), you'll remember that passage for far longer than if that substitution literary device hadn't been used.
If it shook you up so much that you had to stop reading then it was quite successful.
Meh, I have seen other versions, and this Christian one isn't even particularly interesting... though did he discuss Teilhard de Chardin at some point in one of them ?
Signature properties of human language and consciousness are recursion and self-awareness. As ChatGPT and its ilk continue to vacuum up basically all of the human-generated content on the net, more and more of that content will be discussion of ChatGPT and its ilk. These AI's already do an impressive job of composing essays in response to questions about the impact of AI's (that is, themselves). It feels like we are (mataphorically) probing around in the dark, with electric probes, connecting parts of circuits to other parts of circuits, and we are getting close to inadvertently closing circuits that will "levitate" these technologies into something like internally stimulating and augenting self reference. I would prefer we not do this.
I don’t see why people are so worried about AI in the future. AI has been exploiting humans for 500 years already, and destroying Earth’s habitability for the last 150 years.
I’m talking about our first AIs, the corporations. They maximize their goals at any cost, live forever, assimilate other corporations, and frequently trick humans to engage in unsafe behavior for their own benefit. They expand to control more and more territory, concentrate power to themselves, and seek to make humans dependent on them — while at the same time polluting everywhere to make the human race weaker. They don’t intend this, they just have an alien code of ethics that worships shareholder value as the height of virtue.
How is computer AI going to be worse than what we’re already doing?
If AIs will be “corporations but faster and/or smarter”, and you accept that corporations behave unethically, AIs seem plainly worse.
I don’t think “AI = corporation” is a useful framework, current AI doesn’t particularly resemble corporations and corporations are much more human and less “intelligent” than I’d expect future AI to be.
> What I want to know is this: what force lies behind the screens and wires of the web in which we are now entangled like so many struggling flies, and how we can break free of it... In short: What is this thing? And how should it be faced?
I had read up until there, but stepped away to do something else. I was looking forward to coming back to something more ambitious beyond the recent spate of "AI" talk, like some kind of analysis within a larger context of the forces pushing us towards technological/economic determinism (slash digital authoritarianism) for the past few decades - the elephant in the room that all of these alarmist "OMGAI" articles conveniently leave out.
Instead, this article disappointingly nose dived into the superstitious/religious, with its characteristic lack of constructive ideas - essentially just lamenting that we're no longer in the past when things were oh-so-simple. Not that there isn't some fundamental wisdom in the religious tales and prognostications of old, but to take them literally as descriptive models that are directly applicable to novel phenomena is utterly vacuous.
This is starkly demonstrated in the second installment of the essay, where the author leans on "I won’t have a smartphone in the house" as if that is an example of ascetic restraint. Sorry, abstention is not principled engagement!
I've developed my own personal philosophy mostly revolving around libre software and personal computing autonomy, which I think has done a decent job of letting me dodge many attention-economy bullets. But it doesn't feel particularly apt for confronting what ML is poised to do. Which is why I was hoping for something a bit more general and constructive, but alas this isn't it.
> Question One: why does digital technology feel so revolutionary?
The digital revolution, which began some 30 years hence, has little to do with the more recent advancements in AI/ML. The reason digital technology feels so revolutionary is because it is - it's our first glimpse into a post-scarcity future. Distribution costs for a digital work to all of humanity is low, and approaches zero if we get rid of copyright.
Copyright is an antiquated system, suitable only for a pre-digital age. We need a better system. One that incentivizes the creation of works while also admitting that DRM to make bits uncopyable is like trying to make water not wet. If instead every copy that was distributed over the Internet, including via Bittorrent paid the author back, suddenly it's not a problem anymore. There are other problems/details to be worked out, but the first step is in admitting you have a problem.
> Even in the mid-1980s, Black had noticed how hours spent on a computer were changing him. ‘I noticed that my thinking became more refined and exact,’ he wrote, ‘able to carry out logical analyses with facility, but at the same time more superficial and less tolerant of ambiguity or conflicting points of view.’
Well this skewers the worst tendencies of the HN comment section…
32 comments
[ 4.4 ms ] story [ 73.8 ms ] threadTo put it as well as I can : after my 2nd reaction I went on to read the now available 2nd part, which also linked to this Open Letter by Yudkowski a month ago, which spells out my 2nd reaction probably better than I could :
https://news.ycombinator.com/item?id=35364833
> I agree that current AIs are probably just imitating talk of self-awareness from their training data. But I mark that, with how little insight we have into these systems’ internals, we do not actually know.
Also, I am not convinced that our current research won't hit a brick wall well before super-intelligence, which would then require a radically different approach... but why are we willing to gamble with something as dangerous as this ?!
Sydney has no genitalia, no hormones, no sex drive. It does not "desire" someone. It may have a theory of mind, but clearly any AI saying that it loves someone is just working from its training data. The article treats this as though there was an actual intention there, rather than just a model repeating its training [0].
Just as obviously, this is a journo playing to our human desire to prevent aliens from running away with our women - any thought of non-humans breeding with humans triggers something pretty deep in us. Well played, sir. Take your paycheck, sit down, and please stop pushing the buttons.
The AIs are not going to take over the world in the sense that this article talks about. They are not invented by supernatural forces beyond our ken, as it suggests. This is a human emotional reaction to things that appear to be intelligent. I agree that this generation of AI is going to hit a brick wall and the shortcomings will become obvious, but the emotional reaction will be the same. I think it will be another 10-20 years before we solve some of the technical challenges in this generation of AI. But we're having the discussion about AI, and the emotional reaction to it, now. Which is probably good, because by the time we develop actual AI we'll have been living with our current level of "almost-AI" since we were kids and we'll be used to it.
And to answer the question: Every technology is dangerous. That has never stopped us from using it.
[0] Though, of course, that raises questions about what an "intention" is in human terms, too. Are we just "repeating our training"? I think not, at least for sex & love - there are too many hormones involved.
Does a country have desires? An economy? Do the millions of transactions and price changes and signals going between people and companies mean that an economy has thoughts, in some sense? When an economy reacts to a war in a foreign land like an ant flinching away from something hot, is it feeling pain in some very primitive sense?
Sure, it's entirely possible that there's no real sensation behind an LLM, or rather, it senses and reacts to text input like a bacterium reacting to chemical nutrient gradients or an ant reacting to food. But at what level of mental complexity does consciousness and true desire arise? I think it's a continuum, and to think that there's something special about biological intelligence vs machine intelligence is to believe in the supernatural. We wouldn't say that aliens don't have true desires because they have genetic material based on something other than DNA, or use something other than glucose for energy in their bodies.
I do think you're right that current LLMs proclaiming love likely don't understand what they're saying. But at some point we're going to build machine intelligences that do, and it may be hard to tell when we cross that line. People used to worry a lot more about the ethics of that.
There is another technology that produced a similarly startling mystery -- the ouja board, introduced in 1890. That one, we figured out. The messages came from the depths of the human minds attached to the technology. The answers to the three numbered questions presented in the substack article are not known to me, but, of course, the nature of the presentation inclines me to think that the answers are likely to explain more about the operation of human minds when they attempt to collaborate in teams connected by unfathomed autonomous technology than they explain about the technology itself.
First sentence says:
> The Internet and its consequences have been a disaster for the human race.
Root cause analysis gets me this far: The Internet is a consequence of the human race. The human race loves consequences. We cannot resist the allure of the search for advantageous consequences, even though we have declining brain size and increasing need for wisdom. It looks to me as if the fight for the wheel will include several factions of humans, several species of AI, and a jumble of alliances, hybrids, coalitions, strategies, conspiracies, and standoffs amongst them. The future is going to be just like a trip to the movies today -- nothing but sequels.
Don't we? Maybe someone with more architectural knowledge of LLM systems can fill in the gaps in my knowledge, but where's the feedback loop that would allow actual self-awareness to develop? FAFAIK, LLM's are feed-forward networks. There are neither small cycles spanning a few nodes, nor integral cycles allowing the model to see what it's outputting -- the only feedback it gets is through the training process, which does not act as a mirror.
I struggle to see how self-awareness would ever manifest from a system that cannot experience its own output.
This is, AFAIK, true of bare LLMs. Systems consisting of LLMs and a control interface that does reprompting which includes the earlier LLM responses (e.g., chat interfaces like ChatGPT) do provide the model with its output as feedback and show learning within the loop (though obviously this learning doesn’t reach back to the base model). The cognitive features of such a loop, such as they are, may well be different than those of a bare LLM considered in isolation.
I still think Yudkowsky is a crank and that taking him seriously is a bigger danger than AI.
Both yes and no, in a sense.
https://slatestarcodex.com/2014/07/30/meditations-on-moloch/
We are doing this because our individual incentives are not aligned with those of the group. Moloch is the god of misaligned incentives, of coordination problems, of arms races and races to the bottom. It's a property of any system with selfish actors, which means any system that humans are part of, and probably any intelligent being that arose out of natural selection would have been hypothetically been part of.
From `Meditations on Moloch`:
> There’s a passage in the Principia Discordia where Malaclypse complains to the Goddess about the evils of human society. “Everyone is hurting each other, the planet is rampant with injustices, whole societies plunder groups of their own people, mothers imprison sons, children perish while brothers war.”
> The Goddess answers: “What is the matter with that, if it’s what you want to do?”
> Malaclypse: “But nobody wants it! Everybody hates it!”
> Goddess: “Oh. Well, then stop.”
> I cheated a bit there, I admit. I changed one of the words. The name that the saint used in that passage was not ‘Ahriman’. It was ‘Antichrist.’
That's manipulation. That's gross. That's a shame. There are some interesting ideas in the post.
He's illustrating how a description of the Antichrist from over 100 years ago so neatly describes modern man's deity.
You clearly understood the parallel, for you felt manipulated. His manipulation is not only intentional, it is admitted by the author!
If it shook you up so much that you had to stop reading then it was quite successful.
His part Two is available (to subscribers at least)
I’m talking about our first AIs, the corporations. They maximize their goals at any cost, live forever, assimilate other corporations, and frequently trick humans to engage in unsafe behavior for their own benefit. They expand to control more and more territory, concentrate power to themselves, and seek to make humans dependent on them — while at the same time polluting everywhere to make the human race weaker. They don’t intend this, they just have an alien code of ethics that worships shareholder value as the height of virtue.
How is computer AI going to be worse than what we’re already doing?
I don’t think “AI = corporation” is a useful framework, current AI doesn’t particularly resemble corporations and corporations are much more human and less “intelligent” than I’d expect future AI to be.
I had read up until there, but stepped away to do something else. I was looking forward to coming back to something more ambitious beyond the recent spate of "AI" talk, like some kind of analysis within a larger context of the forces pushing us towards technological/economic determinism (slash digital authoritarianism) for the past few decades - the elephant in the room that all of these alarmist "OMGAI" articles conveniently leave out.
Instead, this article disappointingly nose dived into the superstitious/religious, with its characteristic lack of constructive ideas - essentially just lamenting that we're no longer in the past when things were oh-so-simple. Not that there isn't some fundamental wisdom in the religious tales and prognostications of old, but to take them literally as descriptive models that are directly applicable to novel phenomena is utterly vacuous.
This is starkly demonstrated in the second installment of the essay, where the author leans on "I won’t have a smartphone in the house" as if that is an example of ascetic restraint. Sorry, abstention is not principled engagement!
I've developed my own personal philosophy mostly revolving around libre software and personal computing autonomy, which I think has done a decent job of letting me dodge many attention-economy bullets. But it doesn't feel particularly apt for confronting what ML is poised to do. Which is why I was hoping for something a bit more general and constructive, but alas this isn't it.
The digital revolution, which began some 30 years hence, has little to do with the more recent advancements in AI/ML. The reason digital technology feels so revolutionary is because it is - it's our first glimpse into a post-scarcity future. Distribution costs for a digital work to all of humanity is low, and approaches zero if we get rid of copyright.
Copyright is an antiquated system, suitable only for a pre-digital age. We need a better system. One that incentivizes the creation of works while also admitting that DRM to make bits uncopyable is like trying to make water not wet. If instead every copy that was distributed over the Internet, including via Bittorrent paid the author back, suddenly it's not a problem anymore. There are other problems/details to be worked out, but the first step is in admitting you have a problem.
https://wn.rudolfsteinerelib.org/RelArtic/BlackDavid/DB1981/...
Seems the author does technical due diligence at a VC firm now:
https://www.blackliszt.com/2021/10/what-is-technical-due-dil...
Animus against that catalyst is so much hormonal groaning.
Human nature is constant.
Well this skewers the worst tendencies of the HN comment section…