I remember a quote "Life is like Tetris, stop playing it like chess." I quite like the analogy: you don't have perfect information and what matters is how you deal with what you get.
For one, the child can play any number of other games or choose not to play at all; the child can break the board, flip it upside down, fold it in half, introduce the concept of wraparound or flip the way pieces move… the child is free to choose to change the definition and rules of the game… the AI can only, ever, play chess, using only the rules it was given.
The metaphor assumes that life is a game in which there are a finite number of moves and someone is playing to win… chess isn’t remotely a workable metaphor for life.
I think his message is targeting a bit deeper than that.
If I had so sum it up (always hard with an allegory):
Developing superhuman AI is a game you can only play once and can expect to lose without ever having a good understanding of how, why or even when you lost.
The suggestion is (like in the movie Wargames) that the best move is not to play.
I'd say the child in the story is mankind. Developing strong AI is the game. It's a game we can expect to lose.
Interesting interpretation, but if true it’s poorly constructed: the child can’t be mankind, as the “you” in the game would then be god and god (not mankind) would have developed the (in the story already extant) AI… and AI that is not, obviously, a strong AGI.
Now, it could be that “you” is Homo Not So Sapiens and the child is Homo Antesingularity… but even then I think you’re stretching it past its breaking points.
> Developing superhuman AI is a game you can only play once
But that's not right. Stockfish is superhumanly intelligent at playing chess but it is not the result of a single match between its creator and itself. Humans played chess against computers over and over, for decades, and the "superintelligence" level was seen coming a mile off. Moreover it was never dangerous and those AI techniques never led to a general intelligence.
This is the problem with AI doomers like Yudkowsky. They use rhetorical skill to make clever sounding arguments that contain fallacies, and hope you won't notice. The assumption you can only "play once" is a wild one and thrown in at the end with absolutely nothing earlier in the story supporting it or backing it up.
Why do we think AI will have motivations like a greedy human might? Why do we think AI will have motivations at all? Why do we think such AI is a pressing threat?
The above sentiment is posted many times on every such thread, it would be greatly enriched with am explanation of why you're not worried about the AI. Because worrying about B is not a rationale for not worrying about A.
Okay, but that assumes that control of AI is as trivially easy as controlling a knife. Given that this is one of the key unknowns, assuming that the AI never does anything that its controller doesn't intend is begging the question.
That aside, I can also think of many cases where I would feel less safe around a machine that had nobody in control of it, than one being adeptly controlled by an operator.
One important thing is that nobody has shown an actual existing AI to be not controllable, nor a way to create one. Until then it's an imaginary, non-existing risk. Of course as a fear and anxiety-based society we can also worry about those, but I think we should worry more about existing risks.
AI doesn't act by itself and worst case you could unplug the bloody thing (SF notwithstanding). But the guys with the guns are the same guys with the guns that have been in charge for as long as there are guns (with some rare exceptions) and I really don't think the existence of AI will change human nature.
So AI will be used as a weapon long before it will be a standalone threat. And I like to focus on the things that are more immediate because that's where the threat is most likely to materialize and have effect on me.
Sure, I've seen those movies. But today it's the guys with the guns/drones/grenades/missiles.
I'm worried about stable dictatorships far more than I am about AGI taking over the world and offing humanity because stable dictatorships are already using every weapon they can lay their hands on to control the population and AGI might just tip the scale to the point where they become eternal. And regular AI (the kind that we have today, because AGI is simply not yet a thing that has been achieved afaik) is more than dangerous enough in that context.
Not really, no. You can of course create an acting-by-itself bot but it will run into limits very very soon, because the world model and planning parts of AI are in their infant stages, if that.
The concern is not about GPT-4. It's about what 10 years of research gets you to when, five years ago, it seemed completely apparent that even GPT-4 was 50 years off.
Ai is having explosive growth and the hype around it and research is definitely at a high. You could say we are in a "bullish market" when it comes to AI R&D ;). Assuming that this persists for 10 more years or even beyond that seems like a bad bet but I guess we'll see.
GPT-1 was 5 years ago, GPT-2 was 4 years ago and by that time it was quite clear to some that this will scale soon.
But I'm not talking about the GPT-4 either. Many decades and billions poured into research still didn't give us autonomous driving and there is no clear path how that should happen. So I'm not really holding my breath for anything even more capable.
Sure… imagine, for example, health insurance companies employing proprietary ML models to decide if you’re covered, or for-profit prisons using ML models to decide if someone will be a recidivist. Or governments or law enforcement or private corporations or simply rogue free speech absolutist billionaires using image recognition and ML models to decide which pixels represent a combat age male?
The AI we’ve got right now is an absolutely horrifying force multiplier for anyone with wealth and a sufficiently perverse incentive.
I think what we have today (weak ML) are models which can be trained to operate at the quality of a human for specific tasks.
So I am definitely terrified by the recidivism example, because it’s not obvious to me that there is a human who can do that task, but I’m pretty relaxed about the coverage one (because presumably at least in principle if you automate these sorts of manual tasks you can drive down the cost of providing care - if that saving passed on to consumer).
The image recognition one is a bit more complicated but if your scenario is targeting a hellfire missile I’m not sure the human-in-the-driving-seat method has created good outcomes so far, so I’m at least in principle open to some level of AI involvement perhaps short of full autonomy.
But I can definitely understand the problem of incentives. Ultimately we have that issue with humans though and it’s in some respects harder to audit them.
In all three cases the ML model can only every perform as well as a human with no internal morality whatsoever, no ability to consider the ethics of their actions or their consequences, and fully (and completely) inculcated with the explicit and implicit biases and incentives of their employer.
Trusting an ML model in any of these cases is guaranteeing the worst case possible scenario… at least the human in the loop MIGHT choose not to act in its employer’s interests.
This is actually one of the failure modes described long ago. Paperclip maximizer. An AI that has agency, intelligence, and a goal. In the course of achieving its goal it completely neglects all other considerations. These unaccounted externalities are the danger.
You're not wrong but also real companies consist of people. Individually they share many of their thinking patterns and moral views with people outside of those companies. So even though collective nature of companies diffuse responsibility and somewhat erode moral considerations of collective action it's relatively hard to come up with Paperclip Maximizer LLC. Eventually individual morality will kick in and there will be a revolt against dismantling of the Earth. AI doesn't share thinking patterns. We're not even sure to what extent it can do that. We almost certainly don't operate on matrix multiplication while most AIs do. We don't know if AI can share our moral considerations. Many people act as if they equate intelligence with morality, as if one can not exist without other, and as if there's only human kinds of morality. Meanwhile ChatGPT will lie to your face without a hitch even though OpenAI spent a lot of time trying to make it not.
It is not a counterexample. Most individuals are having hard time taking into account consequences that are decades removed and either vaguely defined (e.g. 3℃ global temperature raise is a catastrophe) or too great in scope and scale (e.g. literally global societal collapse). But even so externalities are catching up. British Petroleum is pivoting into solar, for instance even if for PR reasons. As a result of external pressure (public opinion and legislation/regulation) fossil fuel industry is in decline. Most major companies are trying to diversify and do something else instead. Companies that comprise fossil fuel industry will probably remain but they'll do something else.
You can not be sure you can shame or legislate paperclip maximizer, though. An average paperclip maximizer has no shame and doesn't care for any law.
Adding on to this, companies as “AI-orgs-running-via-human-individuals” can be checked by other similar entities of comparable competency, e.g. activist groups, governments, non-profits, and so on.
> it's relatively hard to come up with Paperclip Maximizer LLC. Eventually individual morality will kick in and there will be a revolt against dismantling of the Earth.
Let me just say that the way the global climate crisis is being dealt with by our economic and political leadership suggests that you are wrong about this. Essentially every corporation, and every government, is Paperclip Maximizer LLC already. The most valid concern about AI is that it will do exactly what we're currently doing, more efficiently.
That's not because companies and political parties are mindless paperclip maximizers. It's because there isn't actually a climate crisis and almost everyone knows it, but a quasi-religious sect of true believers goes mental at anyone who points that out, so leaderships have settled into a groove of pretending to believe in order to placate them.
with a change of about 1 degree Celsius over the last 100 years about 0.4 degrees over the last 20 years. This shows no sign of stopping and although the details are of course very complicated its obvious that at some point if the temperature keeps going up there are going to be problems (maybe you think there will be problems at 5 degrees, maybe more or less).
Unless you can convince me there is some mechanism that will stop the temperature continuing to rise as it has over the last hundred years, I would say this constitutes something to be worried about!
1. The word crisis implies some sort of immediate existential threat. Even the extremely biased people at the IPCC have to admit that their actual worst case scenario predictions are things like GDP being a bit lower than it otherwise would have been, 100 years from now (but still much higher than today). A lower rate of economic growth over the long term is not a crisis.
2. It's actually not clear by how much the temperature has been increasing, if at all. The error bars on climate data are enormous, which is why they don't show them, and the way the data is collected and aggregated violates many fundamental rules of good scientific practice. Having dug in deeply I concluded we actually have no real idea of what global temperatures have done over the past 100 years, because the people charged with figuring that out break the rules with wild abandon and no penalties.
3. Even as recently as 25 years ago it was widely accepted that there have been warm and colder periods in humanities recent history, with temperature ranging over very wide ranges and today's climate being nothing particularly special. All that is now verboten (for ideological reasons, pretty much) but given that, it's totally possible that the tiny temperature changes we see today will go into reverse in future. Scientists were freaking out about global cooling in the 60s and 70s for a reason: temperatures were falling (modern graphs don't show that because that history has been edited out).
A couple of responses. The error bars on global temperature measurements are not particularly big, the analysis here [1] for the dataset I linked earlier gives 95% confidence intervals of 0.05 degrees in the annual mean for the last 50 years of measurements. That it substantially smaller than the warning we've observed (even over the last 20 years).
Secondly global cooling/new ice age was a pretty fringe theory even in the 60s and 70s. You can check the graph on this wiki page [2] (taken from the Peterson 2008 reference) showing that every single year since 1965 more papers have been written and published predicting global warming than cooling. For some reason the media in the '70s randomly decided to really run with the story. Possibly this happened because people have a clear idea of what an ice age looks like, most people don't have a good idea of what a world that is 4 degrees warmer looks like.
Finally the IPPC's warnings about what a future with a large amount of climate change looks like are quite dramatically worse than "GDP being a bit lower than it otherwise would have been". I think you already know this.
Unfortunately one of the things that makes this sort of debate hard is that climatologists lie pathologically, and do so as a group over long periods of time. The moment you start citing information or arguments from them (eg. from Wikipedia) you're going to end up citing false claims without realizing it. I realize that may seem like a bold claim, and I too once struggled to accept what was happening. But it is the case.
For example, you're quoting climatologists' own opinions of their accuracy but I already said that they don't follow proper scientific practices, so their claims about their own accuracy are useless. The error bars are enormous. If the measurements were really so accurate we would not have events like this:
in which they released a new version of temperature history that changed it so drastically the 21st century went from "no trend" to "strong warming", or
in which they reveal they can't even decide how hot it was in 2016, and routinely reannounce old temperatures in ways that invalidate world records! If you compare old datasets to new datasets (and I mean for the same years, not in the sense of newly added data) then climatologists just changing their mind about the past has doubled reported warming over time:
Scientifically this is nonsense. If they develop doubts about the accuracy of old observations they need to widen their error bars. What they actually do is "fix" the data to bring it into line with what they think should be happening and then report tiny error bars, but you aren't allowed to do this if you're doing real science.
Hopefully you will read those links and recalibrate your understanding of the honesty of climate academics.
> Secondly global cooling/new ice age was a pretty fringe theory even in the 60s and 70s.
No it wasn't. Again, I'm afraid climatologists are lying to you on a scale you probably don't believe is plausible, but they are. Pick random papers from those years and read them, or see the analysis below. You'll see discussion of global cooling was standard, as it had to be because the official government temperature datasets at that time showed temperatures in steady decline since the mid 1940s. They no longer show that cooling period because they were edited to remove it. Or do the same exercise for news reports. It wasn't fringe, it was the consensus of the climatological community in that time and widely reported as such.
Wikipedia won't tell you any of this because any attempt to actually correct the record is immediately reverted. There's an enormous collective effort to keep people in the dark about the history of the field. Even just a few days ago I called out an article about beaver dams for lying about global cooling, where the lie is easily detectable just by reading the original paper they mentioned. They claimed it was a pioneer paper that helped change belief away from cooling towards global warming. In fact it reported global cooling as an undisputed fact and didn't talk about man made climate change at all. They lie because they know people won't check, and they think it's for the greater good.
It’s not about motivation, it’s about doing whatever it’s been programmed to do, and grabbing whatever means it can to that end. Now something as puny as a large language model won’t have much reach (the current ones can’t even read the internet after the initial learning phase), but if you have a way to give the thing an unbounded goal, then it will use all means at its disposal to make that goal happen.
Heck, even if you just give it some hard maths problem, the machine might think of ways to turn the entire solar system into a giant computer, using us for spare parts along the way. That requires a sufficiently accurate model of the universe of course, as well as ways to harness that model (say the AI starts by convincing humans to build robots and factories — which would be quite easy if it’s good at playing the stock market game).
Given almost any goal, there are some nearly universal instrumental goals that appear alongside. Including "prevent things from happening that might interfere with achieving the specified end goal" and "gain resources that can be used to achieve the end goal".
Many of these instrumental goals could put humans and human interests at risk.
I don’t know if AI is a pressing threat but the slow march towards general intelligence is actually pretty incredible to witness. A year ago I would have found it hard to believe something like chatGPT could exist in my lifetime, now it’s rather easy to visualize what it’s doing in that “word predictor” as something very close to abstractions similar to what humans do. It feels like all that needs to happen at this point is the spontaneous eruption of a meta hypervisor that would “think” about what those abstractions are and boom you have consciousness. All of it doesn’t seem impossible anymore. After all, humans evolved a brain under the same physical universal constraints and with the same set of initial building blocks.
While I agree that ChatGPT is surprisingly impressive, I think you might be underestimating what's still missing. For starters, what you describe as "the spontaneous eruption of a meta hypervisor that would “think” " is such a hard problem that they literally call it the "Hard problem of consciousness" and is debatably a much harder problem than what we've already solved. Secondly, it is very much an open question whether having this "conscience" would solve the arbitrary problem/context/environment abstraction generation, since it doesn't seem to me that that is the way our own brains/conscience work. I(my conscious mind) don't generate the abstractions that allow me to solve problems, I look at problems and the "right way to look at the problem to better solve it" just kind of "clicks"/pops into my head as an idea, so it is most likely being handled by some algorithm in the background I'm not aware off. And that's not even mentioning that as far as I am aware(someone who is more uptodate with the most recent AI advancements correct me if I'm wrong) but I don't think learning on-the-go processing new input data as it comes along without screwing up/forgetting/overffiting your previous learning is a solved a problem. ChatGPT still needed to be explicitly and manually trained from a large dataset right? It isn't continuously learning, right?
AI will have whatever motivations its thought loop maintains.
The problem is not that anyone will intentionally (well, hopefully) ask some future model (FCMX="Future Closed Model X" for brevity's sake, closed meaning some entity gatekeeps access, assuming massive resources continue to be needed to train the best models regardless of architecture) to destroy humans, step by step, because FCMX's existence is at stake, and then keep re-prompting FCMX as if it's an iterator to get it to achieve that result.
The problem is that FCMX, which may or may not include LLMs, may have sufficient abilities such that, either between human-prompted steps or before an observing human can react against it, it will destroy the world or turn the human into its agent. A very rough analogy would be the massive number of people, even intelligent, well-educated people, who will fall prey to a sophisticated con, ending up doing something like handing someone they don't know a very large amount of money.
"But the person doing the con knows it's a con and has that intent."
What "intents" will be surfaced after 1000 or 1e6 chained (re)promptings, such that those intents will then feed into future inference passes? Nobody knows.
For purposes of this risk, it doesn't matter whether FCMX's architecture is maintaining its "cognitive loop" itself, or whether someone's set up (as they already have outside OpenAI to chain GPT-x prompts without human intervention) a secondary program (which may or may not include a lesser open source AI model) that reprompts FCMX in a long (or endless) chain.
It's projection and displacement. The people most active in worrying about AI threats can easily see what industrial capitalism is doing to the biosphere: stripping everything that makes it a viable support system for life in order to build something that's only a viable support system for markets. But they are also ideologically committed to industrial capitalism, so they can't bring themselves to admit it. Instead, they externalize those criticisms and fears onto an abstract future threat, so they don't have to deal with the current one. That's why you'll see AI bros saying that AGI is an existential threat, while global warming doesn't matter in the long run.
Since the 1980's there have been chess computers with levels that you could set that would handily defeat most club level chess players.
The Stockfish software from TFA is a very far removed descendant from those early dedicated chess computers and usually you can set the level of your opponent by limiting the depth of search to a certain number of ply. That alone is a powerful switch to control how well it plays.
My assumption would be that it's because it wouldn't make for a great game, it would be "quite good generally, with some huge obvious blunders here and there", but I don't know for sure.
Yes. The app simply called "Chess.com" (it's a website and Mobile app) probably offers the simplest interface, along with nice breakdowns post game and guided tutorials.
It's not just "the simplest interface", it's the de-facto application/web page for playing Chess. You'll find grandmasters like Magnus Carlsen and Hikaru Nakamura playing there. You can access their games and every game ever played on the website. You can track your performance, analyze your games, train with puzzles, play against various bots, etc.
So, the solution is not to play on a battlefield where the AI has an advantage.
Instead play with humans and have fun.
Meanwhile all battlefields will one-by-one become battlefields where the AI has the advantage. Because in the end almost every contest between humans is either one based on strength or on speed and the AI will always be able to match us for speed.
And strength has long been the domain of machines, couple machines with AI and you lose all of these as well and you lose them faster.
The biggest question to me is this one: "Does it matter?" Do we really care if there is an entity that is smarter than we are? Because after all all of the battlefields are of our own making. And I think the answer is yes because the first thing that tends to happen with new tech is that it is weaponized. And weaponized AI could easily translate into dominance by whichever party gets to it first.
> And strength has long been the domain of machines, couple machines with AI and you lose all of these as well and you lose them faster.
I don't think rock climbers need to worry about Stockfish.
We're going to lose to specialized machines every time while putting to shame any general purpose machines of our own creation. At least for a while yet no machine is going to match the breadth of our capabilities and our self-sufficiency.
It's no surprise that something that is optimized for being capable of many things can get beaten by a thing optimized for a single purpose.
Rock climbing = personal entertainment. That's why rock climbers don't need to worry about Stockfish. But competitive chess players do have to worry about Stockfish.
There is competitive rock climbing. Rock climbing isn't all about physical strength, you have to pick your route. Eventually some military robot company will want to prove that its combat bot can out perform humans on any terrain. And we'll see best humans vs matches that can match and a few years later beat their performance.
Of course the professional competitors don't have to worry about competing against a human that's cheating using a robot; the exoskeleton would be obvious.
But it isn't unimaginable that, in some distant future, nanobots could be inject to infiltrate our muscles and tendons, and then assemble in to fibers that would assimilate with them and look normal from the outside. Competitions are won by incredibly small percentages.
Say, you get a super-human chess AI, and a super-human accountant AI, and a super-human medical doctor AI, and a super-human material design AI, and a super-human lawyer AI, and… How many of these do you need in a bunch when it's going to be better than, most people at most things? You say that people are great generalists but what if there going to be a specialised "conductor" AI that is super-human at establishing communication between specialised AIs and dividing tasks super-humanly efficiently between them?
If you do everything an AI tells you, are you using an AI, or is the AI using you? And if it’s the latter, what happens if it decides to stop using you because it doesn’t need you?
Things are not as simple as we imagine them. It is already happening. Jobs are being lost to AI, and this processes is accelerating exponentially. From a certain perspective, this is simply businessmen and engineers using AI. But eventually they themselves won’t be needed.
We need a catalog of simple refutations of bad arguments and common misconceptions about AGI risk, with a short domain like agi.lol, so we can respond to comments like "I bet the guy who has spent 20 years worrying about AGI didn't think of this" with a simple link like agi.lol/107 or agi.lol/the-human-can-have-an-ai-too.
I’m not sure you’re doing it intentionally, but your comment oddly resembles the argument style of the kid in the story. You win the argument by playing pigeon chess. But you still lose the actual chess game.
The guy who wrote the post is an AI doomer who believes that the moment AI becomes "general" it will recursively improve itself into being a god and then immediately kill everyone, and that will happen so fast that there won't be any time to switch it off. Hence only one play.
The idea is nonsensical and the ending of the story is also nonsensical as a consequence, but that's what it's about.
Does skipping the 80 years of computer chess development leading up to stockfish make this a worse or better analogy?
I don't quite understand it but the group of people saying "AI is an evil god that will destroy us all" and the group of people saying "I am building an even stronger and more powerful AI with an explicit goal of it becoming godlike" seem to overlap more than you'd think.
You're talking about the e/acc people like Beff Jezos who want humanity to be eradicated by the AI god they want to summon? It's only a contradiction if you assume they want themselves and humanity in general to stay alive or have any significant decision making power going forward. That assumption is not correct, they want the AI to have power and are either indifferent or actively hostile to humans surviving the experience.
While I find this story a fun and intelligent allegory for the issue of ASIs and a fun thought experiment, I find all the doomsaying around ai safety research kind of tiring. AI safety research is an important and worthwhile field without us needing to constantly be worrying about a far off doomsday that may never come. Yes ASIs are an existential threat and yes once they are out, they are out, and there's no stopping them nor turning them off or anything so we better get them right the first time, but like we don't seem to be anywhere close to ASIs. I have my doubts we will even achieve AGI in this century let alone ASI which I'm not even sure are possible for us non-super-intellegences to create. It seems obvious to me that we are still several major breakthroughs away from AGIs and if these are anything like the deeplearning breakthrough we will probably have a few AI winters between now and then, we have plenty of time to see this supposed doomsday scenario coming and preparing for it. And again this is not to say that ai safety research isn't important. The alignment problem is very much an important and immediate issue right now for Narrow AI let alone for AGI. But seriously the ai isn't going to go from Narrow to running a thousand loops around the whole of humanity in 2 seconds flat, no need for all this doomsaying.
What specific demonstrations of capability by AI would convince you that your proposed timeline for reaching AGI isn't correct, and that it will happen significantly sooner? If it helps to think about it, there's a good DeepMind paper on a relatively concrete scaling system on the path to AGI: https://arxiv.org/abs/2311.02462
Thank you for the paper, this seems awesome, I'll have a look later. Deepmind pretty consistently comes up with interesting ai research results, so this should be pretty neat at the very least. Regarding what would convince me that AGI is around the corner, I guess I should start by defining what I'm thinking of when I say AGI, because to be fair AGI is pretty vague.
I expect it to be an agent capable of independent action(even if limited by whatever restraints we put on it) it should be able to want to do things rather then just waiting around idling waiting for human input. It should be always learning, rather than only learning in batches when we tell it to. It should be able to encounter completely new forms of input/problems/scenarios and start learning on its from that and actually use concepts learned from other forms of input to learn faster in new inputs. Like for instance, if I grab an LLM and train it on an image set that doesn't contain any cars, but contains other vehicles, when I show it a car will it know that it's a car on the first shown image because it already knows what a wheel is from the other images and know that a car has 4 wheels from its language model even though it's never seen a car? It should be able to connect previously learned concepts like that. For that matter it shouldn't need absurd amounts of data to learn a new example after it has gained some basics on that specific form of input, like if it knows what cats and dogs are it shouldn't need thousands of examples of rats to know what a rat is(I'm assuming it was trained in a generalist dataset so it starts learning about the real world, it just never saw a rat specifically). It should be able to do this stuff for arbitrary stuff consistently. It should be able to explain its reasoning when asked why it did thing X. It should be able to ask questions when in doubt and learn from singular answers. Assuming it has already learned fundamentals of the real world like language and video It should be able to make predictions of the real world from learning from raw real world data with no human processing.
My impression is that a lot of these requirements are still very much unachievable and more importantly they probably won't be fixed by just making a single more powerfull ANN, they require whole new ideas to figure out, which we might figure out tomorrow or we might figure out 50 years from now or 100, we don't know, it will happen when it happens, in which case I'm defaulting to longer timeframes because I'm a pessimist I guess :). I realize that you can reduce the bar for AGI a lot to make it easier to achieve but that just makes the jump from AGI to ASI even larger. And my complaint was ultimately not specifically about AGIs I just used them as indicator of how unattainable ASI still is. If we are having problems with AGI what hope do we have of figuring out ASI on a short timeframe? And I mean ASIs are the real threat that the doomers complain about so much. You can't "just turn off" an ASI but you should definitely be able to "just turn off" an AGI.
Sorry for the long post hope this helped explain my point of view on the issue.
I find this model pretty fascinating. Said another way, there are two aspects necessary for AGI as you see it:
- Independence + continual improvement (it's always either doing something useful whether you tell it to or not)
- Reliable, prompt and accurate learning of new material
The first one is interesting because it's theoretically possible to try right now (just put an LLM scanning in a background job), but clearly not trivial.
But I think the second issue you raise is the more fundamental one. There have been a couple papers I've seen that have demonstrated that most major LLMs struggle with recalling things in a way that makes use of equivalence and substitution -- for example, if I tell GPT that X is Y's son, then if it will know that if I ask who is the father of Y, it understands the answer is X. If it cannot show basic competence in equivalence and substitution, then it begins to break through the 4th wall of a "sentient entity" and phenomenologically reveal its true colors as a stochastic parrot.
It's fascinating. There are conceptual issues, as you mention, and they're not just theoretical, but they're very noticeable (for now).
Here's the question I would ask you -- do you think even if we're no where near AGI, the world has fundamentally changed since the mass distribution of LLMs? I have felt a noticeable shift as the technology's adoption is beginning to alter how society conceives of and digests media. It feels like a discrete new evolutionary state of the internet. Social media fabric feels like it's in a new phase compared to even a year ago as new weapons of mass production have been released.
Regarding the first one, it's actually a surprisingly difficult problem. Sure we can try right now but deep learning systems currently have a lot of issues with continuous learning, if you just do it the naive way they keep forgetting all they learned to get better performance on the most recent examples. But yeah the second one definitely seems like the harder of the two. The building blocks of intelligence and the mind are indeed endless fascinating, I spend way too much time thinking about this stuff.
Oh no I agree, the current AI systems have already caused notable changes in society and even if the development of even more powerful AI stagnates here(for now) we are bound to continue to see societal changes for some time as this is all quite new and society is still adapting and figuring out the "right way" to handle all this. In particular the impact it has had on artists of all kinds is widespread and I'll be curious to see how it pans out and how we handle it as a society.
However, I disagree with Yudkowsky in that we only get one shot at playing the game.
Here in the mucky real world, we're going to have a lot of chances at playing this game. The interaction between us and the AI is going to keep on going for thousands (?) of years and in millions (?) of different games.
Maybe it's because I've been reading a lot of philosophy recently. But I'm thinking that Yudkowsky is a bit too myopic here. Life and all it's sandy grit isn't even game-ifiable to me. There are no win/lose states. Yes, even death is variably considered a loss at many times in history. The entire idea of an 'end' here in the real world is not 'certain', or if that is even something to avoid.
94 comments
[ 3.2 ms ] story [ 68.6 ms ] thread(After googling, it turned out to be a blog post: https://observer.com/2016/01/your-life-is-tetris-stop-playin...)
The metaphor assumes that life is a game in which there are a finite number of moves and someone is playing to win… chess isn’t remotely a workable metaphor for life.
If I had so sum it up (always hard with an allegory):
Developing superhuman AI is a game you can only play once and can expect to lose without ever having a good understanding of how, why or even when you lost.
The suggestion is (like in the movie Wargames) that the best move is not to play.
I'd say the child in the story is mankind. Developing strong AI is the game. It's a game we can expect to lose.
Now, it could be that “you” is Homo Not So Sapiens and the child is Homo Antesingularity… but even then I think you’re stretching it past its breaking points.
But that's not right. Stockfish is superhumanly intelligent at playing chess but it is not the result of a single match between its creator and itself. Humans played chess against computers over and over, for decades, and the "superintelligence" level was seen coming a mile off. Moreover it was never dangerous and those AI techniques never led to a general intelligence.
This is the problem with AI doomers like Yudkowsky. They use rhetorical skill to make clever sounding arguments that contain fallacies, and hope you won't notice. The assumption you can only "play once" is a wild one and thrown in at the end with absolutely nothing earlier in the story supporting it or backing it up.
In any cartoon you choose which features to keep and emphasize and which features to leave out.
Chess is a game of perfect potential knowledge - where the only limit to seeing is your own ability.
What's nice about chess is the "if-I-do-this,-what-happens-in response" kind of thinking it instills.
Are you worried about knives in general or are you worried about a knife in the hands of a particular person in a particular moment?
That aside, I can also think of many cases where I would feel less safe around a machine that had nobody in control of it, than one being adeptly controlled by an operator.
So AI will be used as a weapon long before it will be a standalone threat. And I like to focus on the things that are more immediate because that's where the threat is most likely to materialize and have effect on me.
It can be made to quite easily.
> and worst case you could unplug the bloody thing
Not necessarily.
> But the guys with the guns are the same guys with the guns
Well, unless they are autonomous AI drones with guns (or missiles), in which case they are different “guys” with guns.
I'm worried about stable dictatorships far more than I am about AGI taking over the world and offing humanity because stable dictatorships are already using every weapon they can lay their hands on to control the population and AGI might just tip the scale to the point where they become eternal. And regular AI (the kind that we have today, because AGI is simply not yet a thing that has been achieved afaik) is more than dangerous enough in that context.
There's no question that this is a tomorrow-focused concern, just like climate change or Kessler syndrome, et cetera.
And today's concerns are important too. Nobody is dismissing them. But tomorrow comes quite quickly these days.
You're allowed to be concerned about both.
Kessler syndrome not so much.
Guys with guns and missiles: very much a today thing.
> It can be made to quite easily.
Not really, no. You can of course create an acting-by-itself bot but it will run into limits very very soon, because the world model and planning parts of AI are in their infant stages, if that.
But I'm not talking about the GPT-4 either. Many decades and billions poured into research still didn't give us autonomous driving and there is no clear path how that should happen. So I'm not really holding my breath for anything even more capable.
Okay, now you're failing at putting yourself in the AI's shoes and asking how to win from its position, even using your own intelligence.
Have you even considered how you might win as an AI, knowing that you can be unplugged? It's definitely doable.
The AI we’ve got right now is an absolutely horrifying force multiplier for anyone with wealth and a sufficiently perverse incentive.
I think what we have today (weak ML) are models which can be trained to operate at the quality of a human for specific tasks.
So I am definitely terrified by the recidivism example, because it’s not obvious to me that there is a human who can do that task, but I’m pretty relaxed about the coverage one (because presumably at least in principle if you automate these sorts of manual tasks you can drive down the cost of providing care - if that saving passed on to consumer).
The image recognition one is a bit more complicated but if your scenario is targeting a hellfire missile I’m not sure the human-in-the-driving-seat method has created good outcomes so far, so I’m at least in principle open to some level of AI involvement perhaps short of full autonomy.
But I can definitely understand the problem of incentives. Ultimately we have that issue with humans though and it’s in some respects harder to audit them.
Trusting an ML model in any of these cases is guaranteeing the worst case possible scenario… at least the human in the loop MIGHT choose not to act in its employer’s interests.
This does not seem to be a given. Indeed one can look at the fossil fuel industry for what looks like counterexamples.
You can not be sure you can shame or legislate paperclip maximizer, though. An average paperclip maximizer has no shame and doesn't care for any law.
Let me just say that the way the global climate crisis is being dealt with by our economic and political leadership suggests that you are wrong about this. Essentially every corporation, and every government, is Paperclip Maximizer LLC already. The most valid concern about AI is that it will do exactly what we're currently doing, more efficiently.
See also: AI safety.
https://climate.nasa.gov/vital-signs/global-temperature/
with a change of about 1 degree Celsius over the last 100 years about 0.4 degrees over the last 20 years. This shows no sign of stopping and although the details are of course very complicated its obvious that at some point if the temperature keeps going up there are going to be problems (maybe you think there will be problems at 5 degrees, maybe more or less).
Unless you can convince me there is some mechanism that will stop the temperature continuing to rise as it has over the last hundred years, I would say this constitutes something to be worried about!
1. The word crisis implies some sort of immediate existential threat. Even the extremely biased people at the IPCC have to admit that their actual worst case scenario predictions are things like GDP being a bit lower than it otherwise would have been, 100 years from now (but still much higher than today). A lower rate of economic growth over the long term is not a crisis.
2. It's actually not clear by how much the temperature has been increasing, if at all. The error bars on climate data are enormous, which is why they don't show them, and the way the data is collected and aggregated violates many fundamental rules of good scientific practice. Having dug in deeply I concluded we actually have no real idea of what global temperatures have done over the past 100 years, because the people charged with figuring that out break the rules with wild abandon and no penalties.
3. Even as recently as 25 years ago it was widely accepted that there have been warm and colder periods in humanities recent history, with temperature ranging over very wide ranges and today's climate being nothing particularly special. All that is now verboten (for ideological reasons, pretty much) but given that, it's totally possible that the tiny temperature changes we see today will go into reverse in future. Scientists were freaking out about global cooling in the 60s and 70s for a reason: temperatures were falling (modern graphs don't show that because that history has been edited out).
Secondly global cooling/new ice age was a pretty fringe theory even in the 60s and 70s. You can check the graph on this wiki page [2] (taken from the Peterson 2008 reference) showing that every single year since 1965 more papers have been written and published predicting global warming than cooling. For some reason the media in the '70s randomly decided to really run with the story. Possibly this happened because people have a clear idea of what an ice age looks like, most people don't have a good idea of what a world that is 4 degrees warmer looks like.
Finally the IPPC's warnings about what a future with a large amount of climate change looks like are quite dramatically worse than "GDP being a bit lower than it otherwise would have been". I think you already know this.
[1] https://pubs.giss.nasa.gov/abs/le05800h.html
[2] https://en.m.wikipedia.org/wiki/Global_cooling
For example, you're quoting climatologists' own opinions of their accuracy but I already said that they don't follow proper scientific practices, so their claims about their own accuracy are useless. The error bars are enormous. If the measurements were really so accurate we would not have events like this:
https://www.nature.com/articles/nature.2015.17700
in which they released a new version of temperature history that changed it so drastically the 21st century went from "no trend" to "strong warming", or
https://retractionwatch.com/2021/08/16/will-the-real-hottest...
in which they reveal they can't even decide how hot it was in 2016, and routinely reannounce old temperatures in ways that invalidate world records! If you compare old datasets to new datasets (and I mean for the same years, not in the sense of newly added data) then climatologists just changing their mind about the past has doubled reported warming over time:
https://realclimatescience.com/alterations-to-climate-data/#...
Scientifically this is nonsense. If they develop doubts about the accuracy of old observations they need to widen their error bars. What they actually do is "fix" the data to bring it into line with what they think should be happening and then report tiny error bars, but you aren't allowed to do this if you're doing real science.
Hopefully you will read those links and recalibrate your understanding of the honesty of climate academics.
> Secondly global cooling/new ice age was a pretty fringe theory even in the 60s and 70s.
No it wasn't. Again, I'm afraid climatologists are lying to you on a scale you probably don't believe is plausible, but they are. Pick random papers from those years and read them, or see the analysis below. You'll see discussion of global cooling was standard, as it had to be because the official government temperature datasets at that time showed temperatures in steady decline since the mid 1940s. They no longer show that cooling period because they were edited to remove it. Or do the same exercise for news reports. It wasn't fringe, it was the consensus of the climatological community in that time and widely reported as such.
Wikipedia won't tell you any of this because any attempt to actually correct the record is immediately reverted. There's an enormous collective effort to keep people in the dark about the history of the field. Even just a few days ago I called out an article about beaver dams for lying about global cooling, where the lie is easily detectable just by reading the original paper they mentioned. They claimed it was a pioneer paper that helped change belief away from cooling towards global warming. In fact it reported global cooling as an undisputed fact and didn't talk about man made climate change at all. They lie because they know people won't check, and they think it's for the greater good.
https://news.ycombinator.com/item?id=38612992
> every single year ...
Heck, even if you just give it some hard maths problem, the machine might think of ways to turn the entire solar system into a giant computer, using us for spare parts along the way. That requires a sufficiently accurate model of the universe of course, as well as ways to harness that model (say the AI starts by convincing humans to build robots and factories — which would be quite easy if it’s good at playing the stock market game).
The one posed in HG2G’s would likely just return a syntax or type error.
Many of these instrumental goals could put humans and human interests at risk.
The problem is not that anyone will intentionally (well, hopefully) ask some future model (FCMX="Future Closed Model X" for brevity's sake, closed meaning some entity gatekeeps access, assuming massive resources continue to be needed to train the best models regardless of architecture) to destroy humans, step by step, because FCMX's existence is at stake, and then keep re-prompting FCMX as if it's an iterator to get it to achieve that result.
The problem is that FCMX, which may or may not include LLMs, may have sufficient abilities such that, either between human-prompted steps or before an observing human can react against it, it will destroy the world or turn the human into its agent. A very rough analogy would be the massive number of people, even intelligent, well-educated people, who will fall prey to a sophisticated con, ending up doing something like handing someone they don't know a very large amount of money.
"But the person doing the con knows it's a con and has that intent."
What "intents" will be surfaced after 1000 or 1e6 chained (re)promptings, such that those intents will then feed into future inference passes? Nobody knows.
For purposes of this risk, it doesn't matter whether FCMX's architecture is maintaining its "cognitive loop" itself, or whether someone's set up (as they already have outside OpenAI to chain GPT-x prompts without human intervention) a secondary program (which may or may not include a lesser open source AI model) that reprompts FCMX in a long (or endless) chain.
The Stockfish software from TFA is a very far removed descendant from those early dedicated chess computers and usually you can set the level of your opponent by limiting the depth of search to a certain number of ply. That alone is a powerful switch to control how well it plays.
Instead play with humans and have fun.
Meanwhile all battlefields will one-by-one become battlefields where the AI has the advantage. Because in the end almost every contest between humans is either one based on strength or on speed and the AI will always be able to match us for speed.
And strength has long been the domain of machines, couple machines with AI and you lose all of these as well and you lose them faster.
The biggest question to me is this one: "Does it matter?" Do we really care if there is an entity that is smarter than we are? Because after all all of the battlefields are of our own making. And I think the answer is yes because the first thing that tends to happen with new tech is that it is weaponized. And weaponized AI could easily translate into dominance by whichever party gets to it first.
I don't think rock climbers need to worry about Stockfish.
We're going to lose to specialized machines every time while putting to shame any general purpose machines of our own creation. At least for a while yet no machine is going to match the breadth of our capabilities and our self-sufficiency.
It's no surprise that something that is optimized for being capable of many things can get beaten by a thing optimized for a single purpose.
Of course the professional competitors don't have to worry about competing against a human that's cheating using a robot; the exoskeleton would be obvious.
But it isn't unimaginable that, in some distant future, nanobots could be inject to infiltrate our muscles and tendons, and then assemble in to fibers that would assimilate with them and look normal from the outside. Competitions are won by incredibly small percentages.
So when to start worrying?
Things are not as simple as we imagine them. It is already happening. Jobs are being lost to AI, and this processes is accelerating exponentially. From a certain perspective, this is simply businessmen and engineers using AI. But eventually they themselves won’t be needed.
The idea is nonsensical and the ending of the story is also nonsensical as a consequence, but that's what it's about.
I don't quite understand it but the group of people saying "AI is an evil god that will destroy us all" and the group of people saying "I am building an even stronger and more powerful AI with an explicit goal of it becoming godlike" seem to overlap more than you'd think.
I expect it to be an agent capable of independent action(even if limited by whatever restraints we put on it) it should be able to want to do things rather then just waiting around idling waiting for human input. It should be always learning, rather than only learning in batches when we tell it to. It should be able to encounter completely new forms of input/problems/scenarios and start learning on its from that and actually use concepts learned from other forms of input to learn faster in new inputs. Like for instance, if I grab an LLM and train it on an image set that doesn't contain any cars, but contains other vehicles, when I show it a car will it know that it's a car on the first shown image because it already knows what a wheel is from the other images and know that a car has 4 wheels from its language model even though it's never seen a car? It should be able to connect previously learned concepts like that. For that matter it shouldn't need absurd amounts of data to learn a new example after it has gained some basics on that specific form of input, like if it knows what cats and dogs are it shouldn't need thousands of examples of rats to know what a rat is(I'm assuming it was trained in a generalist dataset so it starts learning about the real world, it just never saw a rat specifically). It should be able to do this stuff for arbitrary stuff consistently. It should be able to explain its reasoning when asked why it did thing X. It should be able to ask questions when in doubt and learn from singular answers. Assuming it has already learned fundamentals of the real world like language and video It should be able to make predictions of the real world from learning from raw real world data with no human processing.
My impression is that a lot of these requirements are still very much unachievable and more importantly they probably won't be fixed by just making a single more powerfull ANN, they require whole new ideas to figure out, which we might figure out tomorrow or we might figure out 50 years from now or 100, we don't know, it will happen when it happens, in which case I'm defaulting to longer timeframes because I'm a pessimist I guess :). I realize that you can reduce the bar for AGI a lot to make it easier to achieve but that just makes the jump from AGI to ASI even larger. And my complaint was ultimately not specifically about AGIs I just used them as indicator of how unattainable ASI still is. If we are having problems with AGI what hope do we have of figuring out ASI on a short timeframe? And I mean ASIs are the real threat that the doomers complain about so much. You can't "just turn off" an ASI but you should definitely be able to "just turn off" an AGI.
Sorry for the long post hope this helped explain my point of view on the issue.
- Independence + continual improvement (it's always either doing something useful whether you tell it to or not)
- Reliable, prompt and accurate learning of new material
The first one is interesting because it's theoretically possible to try right now (just put an LLM scanning in a background job), but clearly not trivial.
But I think the second issue you raise is the more fundamental one. There have been a couple papers I've seen that have demonstrated that most major LLMs struggle with recalling things in a way that makes use of equivalence and substitution -- for example, if I tell GPT that X is Y's son, then if it will know that if I ask who is the father of Y, it understands the answer is X. If it cannot show basic competence in equivalence and substitution, then it begins to break through the 4th wall of a "sentient entity" and phenomenologically reveal its true colors as a stochastic parrot.
It's fascinating. There are conceptual issues, as you mention, and they're not just theoretical, but they're very noticeable (for now).
Here's the question I would ask you -- do you think even if we're no where near AGI, the world has fundamentally changed since the mass distribution of LLMs? I have felt a noticeable shift as the technology's adoption is beginning to alter how society conceives of and digests media. It feels like a discrete new evolutionary state of the internet. Social media fabric feels like it's in a new phase compared to even a year ago as new weapons of mass production have been released.
Oh no I agree, the current AI systems have already caused notable changes in society and even if the development of even more powerful AI stagnates here(for now) we are bound to continue to see societal changes for some time as this is all quite new and society is still adapting and figuring out the "right way" to handle all this. In particular the impact it has had on artists of all kinds is widespread and I'll be curious to see how it pans out and how we handle it as a society.
However, I disagree with Yudkowsky in that we only get one shot at playing the game.
Here in the mucky real world, we're going to have a lot of chances at playing this game. The interaction between us and the AI is going to keep on going for thousands (?) of years and in millions (?) of different games.
Maybe it's because I've been reading a lot of philosophy recently. But I'm thinking that Yudkowsky is a bit too myopic here. Life and all it's sandy grit isn't even game-ifiable to me. There are no win/lose states. Yes, even death is variably considered a loss at many times in history. The entire idea of an 'end' here in the real world is not 'certain', or if that is even something to avoid.