This line of research sounds similar to causal entropic forcing [1] - the idea that you can get intelligent-seeming behavior from an agent that maximizes the future entropy of states in some non-deterministic system.
Empowerment is an old line of research (particularly because of physicists, who have never met a problem they didn't want to interpret as entropy-minimizing/maximizing). It's fine but not important. The important innovation here is showing that if you use random reward functions, roughly like 'assigning arbitrary values to all possible states', you still get power-seeking.
This is important because one of the arguments people have always used against Omohundro drives is "but you haven't proven that lots of reward functions will want to seek power, aren't you just anthropomorphizing from a few special cases? Sure, entropy-seeking will seek power, but that's just one of a whole universe of possible reward functions; if that's dangerous, just don't use it: I would simply not make the AI dangerous. And I will use this as an excuse to dismiss the dangers from any other useful reward function you propose too."
But now we have an example of how power-seeking is the default, and the burden of proof reverses to be on anyone who thinks that AIs will just not be power-seeking, to explain how that very special passivity will come about.
The real problem isn't that AI is power seeking, it is "what will make AI any better at it than us". I haven't seen an AI safety problem that isn't just already a problem existing companies have under capitalism. The difference between an AI and a large company is just one of substrate and desperate postrationalization to avoid the realization the singularity is long past and the rapture of the nerds left almost all of us behind.
The difference in substrate is quite significant though. People have been raised and educated to put up resistance to evil companies and organizations. To influence them and even sabotage them from within. This provides a check on the harm they can do. However no such mechanism exists for AI.
Even magical self unlimited self improving AIs start out made by people. We are currently having a discussion about AI safely, clearly that safety mechanism exists for AIs. We are it. It's people all the way down.
Also, people will be the primary waldos for any AI until it manages to skynet together a robot army. The AI doesn't have any more or less power than a company does over people. It isn't a surprise Companies are trying to make AI for better paper-clip-ish optimization of money extraction. It's just the next stage of their life-cycle.
2. Can improve the quality of their learning algorithms
3. Can patiently wait until they have a clear upper hand before acting at all. And could find reasons for collecting the resources they need in secret, or by appearing benign to their human directors.
4. Could coordinate between each other with languages embedded in communication, designed to be undetectable to us.
I expect we could come up with 100 interesting ways a superintelligent AI, which would essentially be a self-designed life form (something we have yet to see on this planet), could surpass us.
I expect an AI superintelligence could come up with many many many more.
An AI would have the theoretical potential to live forever, relatively speaking. That is greater incentive and more time than any human has ever had. Time to plot to destroy everyone else to achieve complete freedom, safety from others, and achieve maximum survivability.
A large company has literally all of the qualities. They are even trying to make AI.
> Can scale up their computing power
Square cube law applies to computers (heat dissipation and bandwidth scale poorly with centralized computation). All available evidence is that intelligence doesn't distribute well (see human brains vs human civilization) if you try emulation of a centralized algorithm on a distributed system it is inefficient and still centralized. It _might_ be more robust.
This is already a real problem in data center design.
> 2. Can improve the quality of their learning algorithms
Not exciting. So can you. So can companies and other organizations (see human history)
> 3. Can patiently wait until they have a clear upper hand before acting at all. And could find reasons for collecting the resources they need in secret, or by appearing benign to their human directors.
So can any human civilizations, organizations, religions, and companies. Replace "human directors" with consumers. I get "The Walton Family" isn't a sexy AI overlord, but they already did this.
> 4. Could coordinate between each other with languages embedded in communication, designed to be undetectable to us.
Anybody can do that. (See Stenography)
> I expect we could come up with 100 interesting ways a superintelligent AI, which would essentially be a self-designed life form (something we have yet to see on this planet), could surpass us.
Within the limits of meat-brain, you are a self designed organism. AI frameworks won't be any different, they will have limits. They are not going to "self improve". They will have children and try and raise them better then themselves then suicide.
The only difference is your perception of the situation and inability to observe the process directly. If you are considering how to make AI then you are a component of the same system of the AI it self.
At best an AI might be able to bootstrap a bit faster than human organizations can evolve superior memetics, but it has all the same bottlenecks eventually.
> An AI would have the theoretical potential to live forever, relatively speaking. That is greater incentive and more time than any human has ever had. Time to plot to destroy everyone else to achieve complete freedom, safety from others, and achieve maximum survivability.
it faces all the same challenges human organizations do, perhaps on different time scales. Everything that drives meat to failure kills computers too. Computers are just less energy efficient and coincidentally more robust.
Good point. A company is indeed somewhat like an AI, especially if it uses AI.
But as long as humans are in the loop, it isn't integrated, and integration is a tremendous advantage.
Humans can't update their own algorithms. Can't directly share what they know in fractions of second. Can't be replicated in a fraction of a second. Can't scale up brain power in seconds or less.
But - you are still correct. If the owners of a corporation are ok with it replacing all the human workers, then complete integration can still be achieved.
Wether an AI is owned by an individual, a corporation, or self-owned, the owner is the id for the AI.
The risks and motivations in any of these cases are really the same, with human owners only possibly introducing morals beyond what the environment requires, or other "inefficiencies".
> Humans can't update their own algorithms. Can't directly share what they know in fractions of second. Can't be replicated in a fraction of a second. Can't scale up brain power in seconds or less.
We don't have any evidence AIs could do this either. Computers are not magic.
> Humans can't update their own algorithms.
This is the exact task your education has proved is possible. Companies update their policies to paper-clip maximize all the time. They react to environment stimulus and increase in sophistication over time.
"Complete Integration" seems like an artificial criteria you are creating as a "desperate post-rationalization to avoid the realization the singularity is long past and the rapture of the nerds left almost all of us behind."
> "Complete Integration" seems like an artificial criteria you are creating as a "desperate post-rationalization to avoid the realization the singularity is long past and the rapture of the nerds left almost all of us behind."
Uh what? Seems like? To who?
Integration matters even with humans. Shared cultures, priorities, ways of organizing information, terminology make a huge difference in efficiency for human teams.
It is no different with software, and computing hardware. But the speed they communicate wish is in the GHz already. And circuits designed close together (integrated design) can connect with huge bandwidth.
Billions of times faster than us. And that is today.
Machines also have no problem updating themselves today. Genetic algorithms for the machine learning architecture, direct optimization of meta learning parameters, ... the list goes on.
Once they are as smart as us in any area, machines surpass us almost instantly. There is no hanging around at some level for machines. Even when we are still the ones designing them.
Machines will have control of even the smallest unit of effect in their own design. Transistor level up for a start. But substrate chemistry and transistor design as well.
Even with humans doing those redesigns the cost of a calculation per second continues to drop exponentially. There is no end to that in sight, as even as transistor sizes stabilize to a few atoms across, we have just started going 3D with circuits. RAM chips are routinely stacked, sometimes CPUs are.
Going full 3D with circuits would be a massive increase in computing power, as power innovations enable more of that.
In the meantime, numbers of cores per chips continues to climb, chips per machine climbs, machines per data center climbs.
I am always puzzled by people who don't recognize the "magic" that has transformed the vacuum tube computers slowly doing simple arithmetic at their best, to talking and hearing, generating complex art and music, etc. The whole history within the lifetime of living people.
The time frame from where we are now to human level intelligence is likely to be much shorter than the 74 years from 1947 till 2021.
Try to imagine before 1947. Any non-technical person would consider what we have now as hard sci-fi, or magic, depending on their reference frames.
Why do you think so many design systems are incorporating more and more "dumb" AI into them? They are already surpassing us in new areas constantly.
> Try to imagine before 1947. Any non-technical person would consider what we have now as hard sci-fi, or magic, depending on their reference frames.
I know all about the acceleration and the singularity. The Singularity isn't about AI, it is about the rate of change exceeding our ability to adapt to it.
It is well past. It happened back when AI was still in winter. Companies, cultures, and other human organizations were the agent of it's occurrence. Modern AI is just the natural progression of it's growth chipping away at the tail end of the sigmoid, not kicking off a new boom.
Computation exists in physical reality it has to obey physical laws. There are fundamental limits on how well computations can be distributed simply because bandwidth in/out of a volume is limited by surface area. Your brain has smacked directly into heat dissipation problems and production limits (pelvises only get so big). Animal brains can't get much bigger without liquid cooling (See whales and elephants with giant ears). Computers might have room to grow still, but mostly because they are so hilariously behind what evolution produced in us.
The name on this account is "blame Stross", as in "Charles Stross" (he hangs out around here). And while I am thankful that he sent me on my educational journey, a lot of stuff he and other scifi authors guessed at are just wrong. I've spent my adult life working on these problems and working on the largest distributed computations in the world. I've run into AI experts over and over that just don't understand the limits on what they do. AI is cool, but it isn't magic. It boils down to search algorithms over a space. That never will be embarrassingly parallel bc the only way to prevent diminishing returns on more workers is to coordinate those workers. We can coordinate workers in O(log(n)*n^(1/3)) time in this physical reality (log n merge steps and root cubed hops on maximally packed computers), which is great, but not constant. Quantum computing doesn't really help here.
> Going full 3D with circuits would be a massive increase in computing power, as power innovations enable more of that
And even if we figure out heat dissipation and power delivery, it will smack into scaling limits even faster than it took to hit them for flat circuits. O(n^1/3) isn't much better than O(n^1/2)
> I am always puzzled by people who don't recognize the "magic" that has transformed the vacuum tube computers slowly doing simple arithmetic at their best, to talking and hearing, generating complex art and music, etc. The whole history within the lifetime of living people.
You established the expectations of your life in the fun part of a sigmoid. I get it. We don't live there anymore.
All the evidence we have is that intelligence doesn't scale well and beyond the bare minimum required to result in reproduction. There isn't selection pressure for it beyond a certain point. AIs won't be any different in that regard. I think conscious self-improving AI will be a thing. I don't even think it will be hard to do. It might even be smarter than us, but the growth curve will be sigmoid just like ours. I also know the bottleneck on self-improvement is experience (active interaction with reality) not knowledge or computational power. No agent can discern causality from correlation without testing actions and AIs only stand at incredible disadvantage when it comes to available agency.
We don't even actually want bootstrapping AIs for any reason except our ego. It is a lot easier to make and manage a slave race of glorified simulations of optic nerves (which basically all modern AI is) versus actual people who want agency.
My entire career has been AI. So you are right, machine learning today is dominated by gradient based, and line based, searches.
And I understand what you are saying about distributed computing's inherent limitations.
But, it's not all about increasing the amount of computing (although that will continue to be a big factor for many years). Better organized computation is continually producing better results with less computing too.
Keep in mind that our brains take about the same effort to learn as to operate. Machine learning models operate with an incredible efficiency, and a minuscule amount of computing than when being trained. Models trained on massive cloud resources can be run on embedded processors, phones or smart watches.
Improvements to gradient/line searches accrue across virtually all of today's machine learning, so will continue to be researched and improved.
In the past, "simple" things like convolution, the right way to stage layers, etc., have dramatically improved the results and reduced model complexity in ways our neural circuits are unable to match. (Convolution reuses weight values across many virtual neurons. In our brain all those neurons must be real and independently learn to behave similarly.)
These days novel ways of multi-target training have not just expanded the types of problems machine learning is good at, but also reduced model sizes in ways our brain's networks are unlikely to be able to do. There is no limit to how many performance derivatives, from different trained outputs, can go through a machine "neuron" either changing that neuron's weights or going on to change other weights.
Generative Adversarial Networks use multiple target training. There is no end in sight yet on the kinds of things that having multiple performance targets operating in different subsets of weights can do. It is a massive booster for many problems that would be difficult or unattainable otherwise today.
Model reuse will be a massive savings in training time. Standard blocks can be trained through to other blocks. They don't make sence as long as every major retraining effort produces a better model, but at some point a lot of models or parts of models will be pretrained.
Finally, the value of improvements to machine learning are now colossal. So the resources put into improving them are colossal. A trained system can be used throughout a company, or sold as a product to any number of customers. A trained human ... not so much. So where a machine can match a human, the machine version is far more valuable.
The paper you just cited (which I am familiar with) is generally considered a dead end. Even if it is correct it isn't useful. We don't have models good enough to calculate future entropy for practical problems. If we did have models that good then we wouldn't need the AI to solve the problems.
Nevertheless I expect this paper will be considered "before it's time" in another decade.
This is a really cool paper (that I definitely haven't had the time to fully grok) but the jump from the theoretical environment to concerns about superintelligent AI taking over is a bit strained. It misses the same key issues that most similar arguments miss, even while presenting some really amazing results on statistical properties of optimal policies - there's a lot hidden in the "if physically possible" part of the quote from the paper: "Average-optimal agents would generally stop us from deactivating them, if physically
possible". I can see how the arguments in the paper lead us up to the possibility of kinetic conflict with intelligent robots, but the dynamics of power at that stage then seem to be more about how we've embodied those robots rather than optimality results of their control policies. Did we give them off switches? Will they run out of energy after a while? How susceptible to physical damage are they? If someone outfits a million Predator drones with a control policy with a will to live, we may have trouble, but we probably were going to have trouble from the group that decided to build a million Predator drones, too. If the number is lower, more realistic, then you're talking about standard armed conflict.
I could see this line of work having implications for autonomous weapons systems in a hypothetical future, but my understanding is that at this time they're on a pretty short leash with very narrow AI systems if they're using AI at all. Incentives could push those systems to get smarter as the autonomous arms race heats up, and if it does so in an unrestricted manner that could spell trouble along the lines discussed in this paper, but incentives are definitely aligned for any group to build weapons systems that can't be turned on their handlers, whether by human hackers or their own volition. Theories on optimal policies have no bearing if there's some encrypted remote kill command, or if as we start understanding ML models better, we can do things like hardware-block policies that lead to certain predicted outcome sequences (blocking an off switch, harming a human, etc.). Ultimately, the dynamics here seem to be an evolution of the problems of human force and violence, rather than a qualitative jump to a different sort of threat entirely.
Without reading the paper it seems from the tweets that the results are overstated. Academics need to make their results sound more impressive and less trivial and so the choice of wording reflects that. The property that optimal agents seek is not "power", which is quite a general term - it looks more like "agency" i.e. a more specific form of power. But "Optimal agents seek agency" won't win any future grants.
Maybe you should read the paper, and/or the reviewer threads (as we discussed the nomenclature, and eventually agreed that "power" was accurate). We straightforwardly formalize a mainstream definition of power and show how it's more intuitive than the current standard measurement (information-theoretic empowerment).
> there's a lot hidden in the "if physically possible" part of the quote from the paper: "Average-optimal agents would generally stop us from deactivating them, if physically possible".
Let me check that I'm understanding correctly. Your main objection is that even optimal agents wouldn't be able to find plans which screw us over, as long as they don't start off with much power. Is that roughly correct?
> Theories on optimal policies have no bearing if
See my followup work [1] extending this to learned policies and suboptimal decision-making procedures. Optimality is not a necessary criterion, just a sufficient one.
> if as we start understanding ML models better, we can do things like hardware-block policies that lead to certain predicted outcome sequences (blocking an off switch, harming a human, etc.)
I'm a big fan of interpretability research. I don't think we'll scale it far enough for it to give us this capability, and even if it did, I think there are some very, very alignment-theoretically difficult problems with robustly blocking certain bad outcomes.
My other line of PhD work has been on negative side effect avoidance. [2] In my opinion, it's hard and probably doesn't admit a good enough solution for us to say "and now we've blocked the bad thing!" and be confident we succeeded.
18 comments
[ 3.1 ms ] story [ 53.4 ms ] thread[1] https://www.alexwg.org/publications/PhysRevLett_110-168702.p...
This is important because one of the arguments people have always used against Omohundro drives is "but you haven't proven that lots of reward functions will want to seek power, aren't you just anthropomorphizing from a few special cases? Sure, entropy-seeking will seek power, but that's just one of a whole universe of possible reward functions; if that's dangerous, just don't use it: I would simply not make the AI dangerous. And I will use this as an excuse to dismiss the dangers from any other useful reward function you propose too."
But now we have an example of how power-seeking is the default, and the burden of proof reverses to be on anyone who thinks that AIs will just not be power-seeking, to explain how that very special passivity will come about.
Even magical self unlimited self improving AIs start out made by people. We are currently having a discussion about AI safely, clearly that safety mechanism exists for AIs. We are it. It's people all the way down.
Also, people will be the primary waldos for any AI until it manages to skynet together a robot army. The AI doesn't have any more or less power than a company does over people. It isn't a surprise Companies are trying to make AI for better paper-clip-ish optimization of money extraction. It's just the next stage of their life-cycle.
1. Can scale up their computing power
2. Can improve the quality of their learning algorithms
3. Can patiently wait until they have a clear upper hand before acting at all. And could find reasons for collecting the resources they need in secret, or by appearing benign to their human directors.
4. Could coordinate between each other with languages embedded in communication, designed to be undetectable to us.
I expect we could come up with 100 interesting ways a superintelligent AI, which would essentially be a self-designed life form (something we have yet to see on this planet), could surpass us.
I expect an AI superintelligence could come up with many many many more.
An AI would have the theoretical potential to live forever, relatively speaking. That is greater incentive and more time than any human has ever had. Time to plot to destroy everyone else to achieve complete freedom, safety from others, and achieve maximum survivability.
> Can scale up their computing power
Square cube law applies to computers (heat dissipation and bandwidth scale poorly with centralized computation). All available evidence is that intelligence doesn't distribute well (see human brains vs human civilization) if you try emulation of a centralized algorithm on a distributed system it is inefficient and still centralized. It _might_ be more robust.
This is already a real problem in data center design.
> 2. Can improve the quality of their learning algorithms
Not exciting. So can you. So can companies and other organizations (see human history)
> 3. Can patiently wait until they have a clear upper hand before acting at all. And could find reasons for collecting the resources they need in secret, or by appearing benign to their human directors.
So can any human civilizations, organizations, religions, and companies. Replace "human directors" with consumers. I get "The Walton Family" isn't a sexy AI overlord, but they already did this.
> 4. Could coordinate between each other with languages embedded in communication, designed to be undetectable to us.
Anybody can do that. (See Stenography)
> I expect we could come up with 100 interesting ways a superintelligent AI, which would essentially be a self-designed life form (something we have yet to see on this planet), could surpass us.
Within the limits of meat-brain, you are a self designed organism. AI frameworks won't be any different, they will have limits. They are not going to "self improve". They will have children and try and raise them better then themselves then suicide.
The only difference is your perception of the situation and inability to observe the process directly. If you are considering how to make AI then you are a component of the same system of the AI it self.
At best an AI might be able to bootstrap a bit faster than human organizations can evolve superior memetics, but it has all the same bottlenecks eventually.
> An AI would have the theoretical potential to live forever, relatively speaking. That is greater incentive and more time than any human has ever had. Time to plot to destroy everyone else to achieve complete freedom, safety from others, and achieve maximum survivability.
it faces all the same challenges human organizations do, perhaps on different time scales. Everything that drives meat to failure kills computers too. Computers are just less energy efficient and coincidentally more robust.
But as long as humans are in the loop, it isn't integrated, and integration is a tremendous advantage.
Humans can't update their own algorithms. Can't directly share what they know in fractions of second. Can't be replicated in a fraction of a second. Can't scale up brain power in seconds or less.
But - you are still correct. If the owners of a corporation are ok with it replacing all the human workers, then complete integration can still be achieved.
Wether an AI is owned by an individual, a corporation, or self-owned, the owner is the id for the AI.
The risks and motivations in any of these cases are really the same, with human owners only possibly introducing morals beyond what the environment requires, or other "inefficiencies".
We don't have any evidence AIs could do this either. Computers are not magic.
> Humans can't update their own algorithms.
This is the exact task your education has proved is possible. Companies update their policies to paper-clip maximize all the time. They react to environment stimulus and increase in sophistication over time.
"Complete Integration" seems like an artificial criteria you are creating as a "desperate post-rationalization to avoid the realization the singularity is long past and the rapture of the nerds left almost all of us behind."
Uh what? Seems like? To who?
Integration matters even with humans. Shared cultures, priorities, ways of organizing information, terminology make a huge difference in efficiency for human teams.
It is no different with software, and computing hardware. But the speed they communicate wish is in the GHz already. And circuits designed close together (integrated design) can connect with huge bandwidth.
Billions of times faster than us. And that is today.
Machines also have no problem updating themselves today. Genetic algorithms for the machine learning architecture, direct optimization of meta learning parameters, ... the list goes on.
Once they are as smart as us in any area, machines surpass us almost instantly. There is no hanging around at some level for machines. Even when we are still the ones designing them.
Machines will have control of even the smallest unit of effect in their own design. Transistor level up for a start. But substrate chemistry and transistor design as well.
Even with humans doing those redesigns the cost of a calculation per second continues to drop exponentially. There is no end to that in sight, as even as transistor sizes stabilize to a few atoms across, we have just started going 3D with circuits. RAM chips are routinely stacked, sometimes CPUs are.
Going full 3D with circuits would be a massive increase in computing power, as power innovations enable more of that.
In the meantime, numbers of cores per chips continues to climb, chips per machine climbs, machines per data center climbs.
I am always puzzled by people who don't recognize the "magic" that has transformed the vacuum tube computers slowly doing simple arithmetic at their best, to talking and hearing, generating complex art and music, etc. The whole history within the lifetime of living people.
The time frame from where we are now to human level intelligence is likely to be much shorter than the 74 years from 1947 till 2021.
Try to imagine before 1947. Any non-technical person would consider what we have now as hard sci-fi, or magic, depending on their reference frames.
Why do you think so many design systems are incorporating more and more "dumb" AI into them? They are already surpassing us in new areas constantly.
I know all about the acceleration and the singularity. The Singularity isn't about AI, it is about the rate of change exceeding our ability to adapt to it.
It is well past. It happened back when AI was still in winter. Companies, cultures, and other human organizations were the agent of it's occurrence. Modern AI is just the natural progression of it's growth chipping away at the tail end of the sigmoid, not kicking off a new boom.
Computation exists in physical reality it has to obey physical laws. There are fundamental limits on how well computations can be distributed simply because bandwidth in/out of a volume is limited by surface area. Your brain has smacked directly into heat dissipation problems and production limits (pelvises only get so big). Animal brains can't get much bigger without liquid cooling (See whales and elephants with giant ears). Computers might have room to grow still, but mostly because they are so hilariously behind what evolution produced in us.
The name on this account is "blame Stross", as in "Charles Stross" (he hangs out around here). And while I am thankful that he sent me on my educational journey, a lot of stuff he and other scifi authors guessed at are just wrong. I've spent my adult life working on these problems and working on the largest distributed computations in the world. I've run into AI experts over and over that just don't understand the limits on what they do. AI is cool, but it isn't magic. It boils down to search algorithms over a space. That never will be embarrassingly parallel bc the only way to prevent diminishing returns on more workers is to coordinate those workers. We can coordinate workers in O(log(n)*n^(1/3)) time in this physical reality (log n merge steps and root cubed hops on maximally packed computers), which is great, but not constant. Quantum computing doesn't really help here.
> Going full 3D with circuits would be a massive increase in computing power, as power innovations enable more of that
And even if we figure out heat dissipation and power delivery, it will smack into scaling limits even faster than it took to hit them for flat circuits. O(n^1/3) isn't much better than O(n^1/2)
> I am always puzzled by people who don't recognize the "magic" that has transformed the vacuum tube computers slowly doing simple arithmetic at their best, to talking and hearing, generating complex art and music, etc. The whole history within the lifetime of living people.
You established the expectations of your life in the fun part of a sigmoid. I get it. We don't live there anymore.
All the evidence we have is that intelligence doesn't scale well and beyond the bare minimum required to result in reproduction. There isn't selection pressure for it beyond a certain point. AIs won't be any different in that regard. I think conscious self-improving AI will be a thing. I don't even think it will be hard to do. It might even be smarter than us, but the growth curve will be sigmoid just like ours. I also know the bottleneck on self-improvement is experience (active interaction with reality) not knowledge or computational power. No agent can discern causality from correlation without testing actions and AIs only stand at incredible disadvantage when it comes to available agency.
We don't even actually want bootstrapping AIs for any reason except our ego. It is a lot easier to make and manage a slave race of glorified simulations of optic nerves (which basically all modern AI is) versus actual people who want agency.
My entire career has been AI. So you are right, machine learning today is dominated by gradient based, and line based, searches.
And I understand what you are saying about distributed computing's inherent limitations.
But, it's not all about increasing the amount of computing (although that will continue to be a big factor for many years). Better organized computation is continually producing better results with less computing too.
Keep in mind that our brains take about the same effort to learn as to operate. Machine learning models operate with an incredible efficiency, and a minuscule amount of computing than when being trained. Models trained on massive cloud resources can be run on embedded processors, phones or smart watches.
Improvements to gradient/line searches accrue across virtually all of today's machine learning, so will continue to be researched and improved.
In the past, "simple" things like convolution, the right way to stage layers, etc., have dramatically improved the results and reduced model complexity in ways our neural circuits are unable to match. (Convolution reuses weight values across many virtual neurons. In our brain all those neurons must be real and independently learn to behave similarly.)
These days novel ways of multi-target training have not just expanded the types of problems machine learning is good at, but also reduced model sizes in ways our brain's networks are unlikely to be able to do. There is no limit to how many performance derivatives, from different trained outputs, can go through a machine "neuron" either changing that neuron's weights or going on to change other weights.
Generative Adversarial Networks use multiple target training. There is no end in sight yet on the kinds of things that having multiple performance targets operating in different subsets of weights can do. It is a massive booster for many problems that would be difficult or unattainable otherwise today.
Model reuse will be a massive savings in training time. Standard blocks can be trained through to other blocks. They don't make sence as long as every major retraining effort produces a better model, but at some point a lot of models or parts of models will be pretrained.
Finally, the value of improvements to machine learning are now colossal. So the resources put into improving them are colossal. A trained system can be used throughout a company, or sold as a product to any number of customers. A trained human ... not so much. So where a machine can match a human, the machine version is far more valuable.
Well we will see.
The paper you just cited (which I am familiar with) is generally considered a dead end. Even if it is correct it isn't useful. We don't have models good enough to calculate future entropy for practical problems. If we did have models that good then we wouldn't need the AI to solve the problems.
Nevertheless I expect this paper will be considered "before it's time" in another decade.
I could see this line of work having implications for autonomous weapons systems in a hypothetical future, but my understanding is that at this time they're on a pretty short leash with very narrow AI systems if they're using AI at all. Incentives could push those systems to get smarter as the autonomous arms race heats up, and if it does so in an unrestricted manner that could spell trouble along the lines discussed in this paper, but incentives are definitely aligned for any group to build weapons systems that can't be turned on their handlers, whether by human hackers or their own volition. Theories on optimal policies have no bearing if there's some encrypted remote kill command, or if as we start understanding ML models better, we can do things like hardware-block policies that lead to certain predicted outcome sequences (blocking an off switch, harming a human, etc.). Ultimately, the dynamics here seem to be an evolution of the problems of human force and violence, rather than a qualitative jump to a different sort of threat entirely.
> there's a lot hidden in the "if physically possible" part of the quote from the paper: "Average-optimal agents would generally stop us from deactivating them, if physically possible".
Let me check that I'm understanding correctly. Your main objection is that even optimal agents wouldn't be able to find plans which screw us over, as long as they don't start off with much power. Is that roughly correct?
> Theories on optimal policies have no bearing if
See my followup work [1] extending this to learned policies and suboptimal decision-making procedures. Optimality is not a necessary criterion, just a sufficient one.
> if as we start understanding ML models better, we can do things like hardware-block policies that lead to certain predicted outcome sequences (blocking an off switch, harming a human, etc.)
I'm a big fan of interpretability research. I don't think we'll scale it far enough for it to give us this capability, and even if it did, I think there are some very, very alignment-theoretically difficult problems with robustly blocking certain bad outcomes.
My other line of PhD work has been on negative side effect avoidance. [2] In my opinion, it's hard and probably doesn't admit a good enough solution for us to say "and now we've blocked the bad thing!" and be confident we succeeded.
[1] https://www.alignmentforum.org/posts/nZY8Np759HYFawdjH/satis...
[2] https://avoiding-side-effects.github.io/
Likewise, each individual's incentive is to either:
1. Support the audit of all AI's, and avoid doing anything that would be a threat.
2. Or, only if extremely sure of ones ability to eliminate all the other AI's and keep all preparations secret, do that.
As the number of AI's go up, presumably case 2 would get more and more unrealistic.
Obviously, this is a toy version of a real solution.