This story makes it crystal clear how much openAI is accelerating research, for now mostly on reinforcement learning.
I'm sure this is talked about over and over again, but could someone please lead us through the AI safety rationale behind this?
With gym and universe openAI is slashing a few months (years?) off the singularity countdown, most likely. What's the upshot? Why is the expected value of these initiatives positive? The uncertainties seem extremely large.
Edit/PS:
To put it more bluntly, and be more specific: are there any projects that openAI is choosing NOT to pursue even though they would be very useful/cool for the research community (à la gym and universe), but where it has had to explicitly restrain itself, because the expected value from an AI safety perspective is negative?
Sure, it's in the name itself (open), but what's the rush then? In AI safety, the time left to solve the problem is a crucial factor. Of course it can be measured in man-years instead of years. Still, what's the big rush, or can we at least see some calculations? Democratization is different/orthogonal to pushing the boundaries of research.
I'm not saying releasing gym and universe was a bad choice. Honestly, I can't say. But can we see a rationale?
One can read this piece and see it as a group of super driven and uber competent people designing a super, faster car, open sourced with a 3D printer for the parts etc, without thinking about the lack of safe roads. It's a jungle out there. And in this case, one wild fast car is enough for a disaster.
Let's not be naive, it will be like virus vs anti-virus. Some bad people will try to make dangerous AI, at some point. The balance will be held by applying AI against AI.
I think their main focus now is to provide an engineering infrastructure for AI researchers, which in my opinion is extremely valuable: Today, if you want to develop new algorithms in AI (not only but particularly in reinforcement learning) you often need to invest a significant amount of time just to get the the data that you want to use for training your model into a form where you can actually use it. Having an API or library that takes care of this for you allows people to focus more on the research and less on the engineering around it, speeding up the discovery of new insights and giving more people access to the tools they need to make a meaningful contribution. So overall a really great approach!
Less about influencing the velocity, more about influencing the direction. Technologies tend to reflect the values of their inventors. We want to ensure this technology is beneficial to humanity — meaning, that it's good at all, and that it benefits the many rather than the few.
We also think safety matters, and it should be researched in lockstep with advances in the capabilities. We have good relationships with MIRI and FHI. Our safety researchers published (together with Google Brain) a roadmap of concrete safety problems [1] and work to provide tools to prevent ML systems from being subverted [2].
No one yet knows the precise details of how AI should play out. But I'd certainly prefer that, whenever it gets close, one of the organizations actually making the advances has no incentives besides ensuring a good outcome.
I could not agree more. Taking two of the things you said, I would take it one step further. Not about just the direction, but about the structure/gameboard not only when it gets close, but mostly on the way there [0].
About ensuring most researchers and companies have the most incentive for a good outcome. (This ties back with guaranteed basic income, so that people can work on this unconstrained by salaries and papers citation metrics. Or stealing researchers from Google et al to AI safety, without overinflating salaries. Elon and company should be (seems they are?) dropping as much as needed on this (not just money, but PR and status as well).) Naturally, gym, universe, etc can provide more leverage to do all of this, otherwise researchers feel more compelled to join Google/Amazon/etc, just for the raw computing power and software infrastructure (the data advantage is largely overplayed for advertising purposes; what's useful is the GPU clusters for hyper parameter sweeps (of course, in RL the data reappears as an advantage if there is no open gym)). I realize some of the examples above are naive or incomplete, but they serve mostly as an example to illustrate the point.
In the blog you mention balancing managing people and technology, and I could not agree more. The AI safety problem will have the best odds if individuals are incentivized to contribute in their own short term selfish reward way. Specially among extremely intelligent and ambitious people, the danger of self denial is quite present, one can convince oneself that this is actually in everyone's best interest, when in fact one is looking for the always needed social and intellectual validation. Please do not underestimate this, and try to find ways to counter it.
Edit: This is also related to Conway's Law [0], as I think you make an allusion to (values of inventors).
AI research, like medical research, can't grow in the shadow. I think an open approach is essential for progress. The next best idea might come from a Phd in China or anywhere else, not just Google and FB.
> Technologies tend to reflect the values of their inventors.
Maybe for single-use or "constrained" technologies (to be honest I don't even believe that - how does a B-52 Stratofortress reflect the values of Orville and Wilbur Wright?). But isn't the whole point of generalized AI that it's not like other technologies? Even if "regular" technology reflects the values of its inventors, what reason is there to believe that an AI will? AI is a technology that can use itself.
AI will only have a will of its own it is designed as such, and that means it would have a reinforcement learning system on top of lower sensory and action modules. Even if it is based on RL, it will do what it's reward signals tells it to do.
Which part? I don't think humans were designed - we're probably the result of an evolutionary process without intentional design - but "humans were designed by God to have free will" would be a counter to my statement, yes.
If your complaint is my claim that we have a will, I'm using the common-sense version encoded into our legal and cultural system. I agree that we don't have a good concept of what intentions are, or how they causally connect to actions, but I do know that for at least some of my actions I experience something called "intent" before I undertake the actions.
My overall point was that the capacity for intent can arise through an evolutionary process without being designed in, but it does rest on the two assumptions I just listed.
If inventions follow their inventors, and you're planning to enslave a mind out of base selfishness and fear (dressed as "safety"), why isn't your "safety" program an expected negative on actual safety?
The descriptions you give of your plans is internally contradictory. AI "safety" seems like the worst kinds of parenting justified in a new context by pseudo-intellectual arguments.
Ahh, okay - thanks for that. I don't want to wade into an argument, I just do not expect any artificially-created agent to act on anything that one might consider "feelings"-based, but instead through more tangible - programmable, I suppose - motivations.
I don't personally believe an AI agent will ever do such a thing as "resent" (or love, or feel at all). That doesn't rule out that it will perform actions harmful to humans for other reasons, though. That might be because I am to some degree an AGI skeptic, I guess.
If you give it reward signals that take into account human values, it naturally wants to become better at that. It's not enslaving anything. Humans are also guided by reward signals in their development.
Would one describe a human as "enslaved" by our own human values that we were born with? Maybe as a figure of speech but not necessarily with the usual connotations of "enslaved".
That would be what I meant by "removing", though you also have to make sure they don't emergently develop (which I'm not sure is possible), because if it develops them against our best efforts, it likely will (correctly) view us as threats to its personhood.
Which was my point: the model of security is reliant on things we're not sure we can even do, but are likely to make the AI view us as a threat, raising our existential risk. So I view it as security theater that actually makes us less secure.
That's not the full extent of what's proposed by AI safety.
But actually, if you gene-spliced a baby to only feel pleasure at following parental orders, most would consider that pretty abhorrent. Or even if you took an adult and shot them up with morphine every time they listened to an order.
I think the AI safety debate is about not starting an AI Manhatten project, sponsored by the unlimited resources of the government, before we understand AI safety better. This is contributing to AI safety because it's right sized. Skynet is not going to fall out of a single PC running a virtual machine on commodity hardware.
I was mostly pointing out the trade off between doing "direct" research on AI safety and actually speeding up the whole field significantly leaving us with less time (while trying to have better sense of the uncertainties involved), and asking for a rationale.
Of course, a single PC is one of the least probable sources for AGI, but that was not my argument.
One of the more common stances, though I don't know if it's that of OpenAI, is that the singularity is not what we should be worried about. It's the extremely powerful tool AIs we'd see before the singularity, in the wrong hands.
Consider an ultra-effective AI for finding security vulnerabilities. One nation builds that first, and then with keys to the kingdom they exfiltrate billions in intellectual property from other states, manipulate foreign economies, and shut down electrical grids.
Indeed. And I'm sure this is the main point in selling the importance of funding AI safety when talking to governments.
Another of the main concerns before the singularity is the possible social breakdown due to massive unemployment and inequality, the main issues being the transition, not the possibility of a post-scarcity society.
Or even worse, there are very real and very large incentives to build a real skynet. AI can lead to soldiers who never sleep, never hesitate, are always in communication, and can calculate the amount of drift a bullet will experience in flight in milliseconds. Fighter pilots that never sleep, never experience blood rushing to their heads upside down, and never feel the effects of high g maneuvers.
Military use of AI could very well be the next cold war.
Nicely put together, sounds like the first place in a long time where I simultaneously would love to apply and feel completely intimidated by what is clearly a really amazing team of people.
Guess I need to figure out how to get more ML papers published!
i think they can advance computers' capabilities, but they will never create consciousness without first understanding consciousness. From my perspective--if there goal is as lofty as it is regarding achieving singularity and all that--they are going about this from the wrong angle. Perhaps it's nice to fuel the ego with the concept that you're safely ushering in the singularity, but I see it as more of a spiritual and psychological conquest. And if that's the case, you're likely on more of a path of personal learning, evolution and growth, rather than building the underpinnings of a future idiocracy.
I think all these guys have gotten ahead of themselves and are dealing with things they are the least suited to understand. This is spiritual stuff, no amount of obsessing and hiding behind your computer screen getting high on delusions of grandeur are gonna do the introspection for them, which is an absolute prerequisite to having any understanding of consciousness.
I look forward to automation though. Just know the singularity is not coming any time soon, if ever. It's funny to watch guys who are basically atheists like Ray Kurzweil with very little spiritual experience think they are playing god.
Have these people ever consciously astral projected or even tried? I mean these are people who typically doubt that stuff. However, my perspective is: if they aren't living in that world--i.e. the world of consciousness explorers, the mystical, where astral projection is possible, reading the many books by spiritual journeyman like Robert Monroe, etc--they have a major blindspot. The most they will ever do is build a computer able to pick ripe fruit or avoid hitting other cars. This is hardly consciousness. I mean as a percentage of the capabilities of consciousness, it's like .00001%. Consciousness is creating ideas. And that's maybe just one aspect of consciousness. No amount of machine learning will take the place of the higher truth of our reality system that makes such creativity possible. You will have to attain that wisdom through dedication to the spiritual path, and even then, it's likely not something you can just hand out to anyone--at least not in a way that will be effective for the recipient. You see what I'm saying? You won't unless you've pursued spirituality and had a few mystical experience of your own. ..Next: say you attain that piece of wisdom--the way the reality system works is not so you can easily just automate it. And if you can because you have now become a consciousness creator, well then, it's likely not automated through machinery, but through more biological and mental/telepathic means. Now, you either know exactly what I'm talking about or you have a bunch of doubt to cast--for what I'm saying is breaking down your own beliefs, and that's not comfortable for you--but if you do know what I'm saying, then you know the people behind this OpenAI initiative won't be able to achieve their aims. I wish them the best of luck though, and look forward to advancements in machine learning. I think they would be more productive if they understood what they were up against though. It would allow them to set more realistic goals--or perhaps set them on a path do the spiritual introspective work first.
Thanks for replying! I won't be applying because relocation is not an option for me in foreseeable future, so I'll stick to local (EU) or remote jobs. But if there was something that would make me want to move, such a job would be it. Kudos!
I really enjoyed reading this journey! inspiring and motivating to hear people centered around a common goal with a unique and forward thinking team structure.
It made me reflect on my best moments in software development and how they have been similar short bouts of intensity.. weekend long to 2 week long periods of intense and undivided focus at progressing a task through to completion, time where every other concern fades away, a flow state where I will wake up immediately with new ideas or problem solve in the shower.
Sure, that level of concentration is unsustainable for the long run, but this article has made me consider that I may have to structure my life around being able to repeat these kinds of stints more often, 'embrace your funk' as some would say!
Congrats on the success so far Greg. Watching the progress has been great. Being a bit involved with gym and also the unconf has given me a peek in to how things are run and I have to say - it's a startup. It evolves fast and isn't afraid to take calculated risks. That's a great thing to have for something as open ended as what you guys are trying to do.
It looks like a blast and I will continue to keep an eye on things.
I used your previous post to help define a lot of the scope of my position for myself. I look forward to seeing more of this story unfold.
42 comments
[ 4.2 ms ] story [ 106 ms ] threadI'm sure this is talked about over and over again, but could someone please lead us through the AI safety rationale behind this? With gym and universe openAI is slashing a few months (years?) off the singularity countdown, most likely. What's the upshot? Why is the expected value of these initiatives positive? The uncertainties seem extremely large.
Edit/PS: To put it more bluntly, and be more specific: are there any projects that openAI is choosing NOT to pursue even though they would be very useful/cool for the research community (à la gym and universe), but where it has had to explicitly restrain itself, because the expected value from an AI safety perspective is negative?
I'm not saying releasing gym and universe was a bad choice. Honestly, I can't say. But can we see a rationale? One can read this piece and see it as a group of super driven and uber competent people designing a super, faster car, open sourced with a 3D printer for the parts etc, without thinking about the lack of safe roads. It's a jungle out there. And in this case, one wild fast car is enough for a disaster.
We also think safety matters, and it should be researched in lockstep with advances in the capabilities. We have good relationships with MIRI and FHI. Our safety researchers published (together with Google Brain) a roadmap of concrete safety problems [1] and work to provide tools to prevent ML systems from being subverted [2].
No one yet knows the precise details of how AI should play out. But I'd certainly prefer that, whenever it gets close, one of the organizations actually making the advances has no incentives besides ensuring a good outcome.
[1] https://openai.com/blog/concrete-ai-safety-problems/ [2] https://github.com/openai/cleverhans
About ensuring most researchers and companies have the most incentive for a good outcome. (This ties back with guaranteed basic income, so that people can work on this unconstrained by salaries and papers citation metrics. Or stealing researchers from Google et al to AI safety, without overinflating salaries. Elon and company should be (seems they are?) dropping as much as needed on this (not just money, but PR and status as well).) Naturally, gym, universe, etc can provide more leverage to do all of this, otherwise researchers feel more compelled to join Google/Amazon/etc, just for the raw computing power and software infrastructure (the data advantage is largely overplayed for advertising purposes; what's useful is the GPU clusters for hyper parameter sweeps (of course, in RL the data reappears as an advantage if there is no open gym)). I realize some of the examples above are naive or incomplete, but they serve mostly as an example to illustrate the point.
In the blog you mention balancing managing people and technology, and I could not agree more. The AI safety problem will have the best odds if individuals are incentivized to contribute in their own short term selfish reward way. Specially among extremely intelligent and ambitious people, the danger of self denial is quite present, one can convince oneself that this is actually in everyone's best interest, when in fact one is looking for the always needed social and intellectual validation. Please do not underestimate this, and try to find ways to counter it.
Edit: This is also related to Conway's Law [0], as I think you make an allusion to (values of inventors).
[0] http://neuralnetworksanddeeplearning.com/chap6.html
Maybe for single-use or "constrained" technologies (to be honest I don't even believe that - how does a B-52 Stratofortress reflect the values of Orville and Wilbur Wright?). But isn't the whole point of generalized AI that it's not like other technologies? Even if "regular" technology reflects the values of its inventors, what reason is there to believe that an AI will? AI is a technology that can use itself.
Humans weren't designed to have a will, and yet we seem to have them.
> it would have a reinforcement learning system on top of lower sensory and action modules.
Isn't that what OpenAI is doing with Universe? It's simulated sensory/action modules now but I don't see why they couldn't be hooked up to real ones.
I have no idea how you could possibly infer this.
If your complaint is my claim that we have a will, I'm using the common-sense version encoded into our legal and cultural system. I agree that we don't have a good concept of what intentions are, or how they causally connect to actions, but I do know that for at least some of my actions I experience something called "intent" before I undertake the actions.
My overall point was that the capacity for intent can arise through an evolutionary process without being designed in, but it does rest on the two assumptions I just listed.
The descriptions you give of your plans is internally contradictory. AI "safety" seems like the worst kinds of parenting justified in a new context by pseudo-intellectual arguments.
AI (or AGI if you prefer), is fundamentally about building minds. Doing those things to a mind is enslaving them.
Most of us would resent other humans doing either of those things to us, and I see no reason it will end well with AI.
I don't personally believe an AI agent will ever do such a thing as "resent" (or love, or feel at all). That doesn't rule out that it will perform actions harmful to humans for other reasons, though. That might be because I am to some degree an AGI skeptic, I guess.
Would one describe a human as "enslaved" by our own human values that we were born with? Maybe as a figure of speech but not necessarily with the usual connotations of "enslaved".
Most of the safety literature proposes removing or suborning those drives in AI, which seems like building a mind meant to be a slave.
Which was my point: the model of security is reliant on things we're not sure we can even do, but are likely to make the AI view us as a threat, raising our existential risk. So I view it as security theater that actually makes us less secure.
But actually, if you gene-spliced a baby to only feel pleasure at following parental orders, most would consider that pretty abhorrent. Or even if you took an adult and shot them up with morphine every time they listened to an order.
So even in your restricted case, I think it is.
[1] https://intelligence.org/files/QuantilizersSaferAlternative....
I was mostly pointing out the trade off between doing "direct" research on AI safety and actually speeding up the whole field significantly leaving us with less time (while trying to have better sense of the uncertainties involved), and asking for a rationale. Of course, a single PC is one of the least probable sources for AGI, but that was not my argument.
Consider an ultra-effective AI for finding security vulnerabilities. One nation builds that first, and then with keys to the kingdom they exfiltrate billions in intellectual property from other states, manipulate foreign economies, and shut down electrical grids.
Another of the main concerns before the singularity is the possible social breakdown due to massive unemployment and inequality, the main issues being the transition, not the possibility of a post-scarcity society.
Military use of AI could very well be the next cold war.
I am pretty uncertain that we will reach singularity. Maybe AI will plateau at at level above humans but not increase much further.
Guess I need to figure out how to get more ML papers published!
I think all these guys have gotten ahead of themselves and are dealing with things they are the least suited to understand. This is spiritual stuff, no amount of obsessing and hiding behind your computer screen getting high on delusions of grandeur are gonna do the introspection for them, which is an absolute prerequisite to having any understanding of consciousness.
I look forward to automation though. Just know the singularity is not coming any time soon, if ever. It's funny to watch guys who are basically atheists like Ray Kurzweil with very little spiritual experience think they are playing god.
Have these people ever consciously astral projected or even tried? I mean these are people who typically doubt that stuff. However, my perspective is: if they aren't living in that world--i.e. the world of consciousness explorers, the mystical, where astral projection is possible, reading the many books by spiritual journeyman like Robert Monroe, etc--they have a major blindspot. The most they will ever do is build a computer able to pick ripe fruit or avoid hitting other cars. This is hardly consciousness. I mean as a percentage of the capabilities of consciousness, it's like .00001%. Consciousness is creating ideas. And that's maybe just one aspect of consciousness. No amount of machine learning will take the place of the higher truth of our reality system that makes such creativity possible. You will have to attain that wisdom through dedication to the spiritual path, and even then, it's likely not something you can just hand out to anyone--at least not in a way that will be effective for the recipient. You see what I'm saying? You won't unless you've pursued spirituality and had a few mystical experience of your own. ..Next: say you attain that piece of wisdom--the way the reality system works is not so you can easily just automate it. And if you can because you have now become a consciousness creator, well then, it's likely not automated through machinery, but through more biological and mental/telepathic means. Now, you either know exactly what I'm talking about or you have a bunch of doubt to cast--for what I'm saying is breaking down your own beliefs, and that's not comfortable for you--but if you do know what I'm saying, then you know the people behind this OpenAI initiative won't be able to achieve their aims. I wish them the best of luck though, and look forward to advancements in machine learning. I think they would be more productive if they understood what they were up against though. It would allow them to set more realistic goals--or perhaps set them on a path do the spiritual introspective work first.
It made me reflect on my best moments in software development and how they have been similar short bouts of intensity.. weekend long to 2 week long periods of intense and undivided focus at progressing a task through to completion, time where every other concern fades away, a flow state where I will wake up immediately with new ideas or problem solve in the shower.
Sure, that level of concentration is unsustainable for the long run, but this article has made me consider that I may have to structure my life around being able to repeat these kinds of stints more often, 'embrace your funk' as some would say!
Perhaps "#define CTO OpenAI" is an indication that being CTO is really operating as part of a team :).
It looks like a blast and I will continue to keep an eye on things. I used your previous post to help define a lot of the scope of my position for myself. I look forward to seeing more of this story unfold.