This is about 100 years too early. Seriously why do people think neural networks are the answer to AI? They are proven to be stupid outside of their training data. We have such a long way to go. This fear-mongering is pointless.
There's a paragraph basically saying "there's been some cool stuff happening (you may have even heard about) in this particular field". Extrapolating that to the entire organization thinking that "neural networks are the answer to AI" as OP claimed is silly.
Of the researchers they hired (some of them don't appear to be active researchers right now, like Trevor Blackwell), they're all are deep learning researchers...
Please let's resist taking this thread flameward. The GP contains both a substantive point and provocation, which isn't great. In such cases, the helpful way to respond is to de-escalate, by addressing the substantive point and ignoring the provocation.
I'm not the same guy, but I'd like to answer: Familiar, that's how it feels. I'm confident that I'm right about the other matter, and subjects are related, so I'm not bothered.
I don't think openai was formed to concentrate on neural networks, and I think we can assume the members are well aware of the limitations of neural nets.
I just don't understand the folks that are so confident that strong AI is either not possible, or not achievable within our lifetimes.
If you're in the camp that thinks it's not possible, then you must ascribe some sort of magical or spiritual significance to the human brain.
If you don't think it's possible inside of 100 years, then you're probably just extrapolating on history. The thing about breakthroughs is they never look like they're coming. Until they do.
The creation of what is essentially an ethics committee for a technology that doesn't even exist yet? With people such as Elon Musk on board who have publicly said 'AI is our biggest existential threat' ?
Additionally the second paragraph:
We believe AI should be an extension of individual human wills and, in the spirit of liberty, as broadly and evenly distributed as is possible safely.
This infers they think AI will be used for hostile means. Such as wiping out the human race maybe? It is just un-informed people making un-informed decisions and then informing other un-informed people of said decisions as if they were informed.
I doubt they are strictly targeting "strong AI", and a lot of the things we use and call AI right now also benefit from open work and discussion. Just because it is just "Machine learning" doesn't mean it isn't used for questionable or bad purposes.
EDIT: I have to agree with _delirium's skepticism towards them doing much in that regard though.
Sutskever, Schulman, Karpathy and Kingma are experts in machine learning.
And yes, AI will definitely be used for all sorts of purposes including hostile means. Just like anything else, really. Financial manipulation, spying, intelligent military devices, cracking infrastructure security, etc.
These are realistic concerns, we shouldn't fall for the Skynet red herring. We can have problems with ethical AI use, even if it's not a self-aware super-human superintelligence.
I hope I'm wrong, but given the composition of the donors I'd be surprised if they really put much scrutiny on near-term corporate/government misuse of AI, apart perhaps from military robots. There are definitely interesting ethics questions already arising today around how large tech companies, law enforcement, etc. are starting to use AI, whether it's Palantir, the FBI, Google, or Facebook, so no argument that it's a timely subject at least in some of its forms. It'll be interesting to see if they get into that. I'd guess they probably want to avoid the parts that overlap too much with data-privacy concerns, partly because a number of their sponsors are not exactly interested in data privacy, and partly because the ethical debate then becomes more complex (it's not purely an "ethics of AI" debate, but has multiple axes).
I share your concerns. It also worries me that the brightest ML researchers choose to work at companies like Facebook, Google and Microsoft instead of public universities. One reason is probably that academia and public grants are too sluggish to accommodate this fast paced field. Another is that these companies have loads of data that these researchers can use to test their ideas.
The downside is that much of the research is probably held secret for business advantages. The public releases are more of a PR and hiring strategy than anything else in my opinion. By sending papers to conferences, Google's employees can get to know the researchers and attract them to Google.
Others say there's nothing to worry about, Google and Facebook are just today's equivalent of Bell Labs, which gave numerous contributions to computer technologies without causing much harm.
In terms of artificial general intelligence, the kind of stuff that gets associated with 'the singularity' etc. - I agree - there is seemingly nothing at all out there that appears to even be on a trajectory, I mean even theoretically, to coming close. But sure, for more narrow specializations, disrupting certain industries, there is a lot that could be advanced on a short time scale.
I don't think the big breakthroughs in artificial general intelligence are going to come from well funded scientific researchers anyways, they are going to come out of left field from where you least expect it.
That's the thing. It's not necessarily about an evolution. If I "accidentally" make a break though right now. The AI could evolve into a super intelligence before dinner time.
Simply stated, an AI that writes AI.(Forget the halting problem for a moment) How many iterations can it create in 3 hours?
Think about how long it takes a baby to learn (a few years). There is no reason an AI would self-improve multiplicatively in hours or days or even weeks.
With a powerful enough computer the learning curve of a simulated baby could hypothetically be condensed to minutes, or seconds. Which would appear to us in normal time to be an intelligence explosion.
The simulated baby would extremely tax to the limit the entire resources of whatever supercomputer trained it. People have to run deep learning programs on multiple GPUs right now, and performance is constrained by memory bandwidth, which does not follow Moore's law.
It's a hypothetical, not expected to happen in conventional supercomputers with the current state of machine learning.
Imagine a massively parallel optical computer with the same transistor density as the human brain, the size of an olympic swimming pool running at the speed of light, and networked with 1000s of other similar computers around the world.
Foomp, superintelligence, you won't even be able to pinpoint the source.
Sure, but you would get the baby situation I posited years before what you're envisioning. You're skipping a bunch of steps. My hypotheticals (if they ever do happen) would occur way before your hypotheticals. Progress does not go in "Foomp" steps.
Everything goes in foomp steps if you condense time enough. Condense the last 200,000 years of human evolution to a couple minutes, and it goes 'Foomp', and the super foompy part doesn't happen until the last 2/10ths of a second with industrial civilization wherein the development of global networks compounds our collective intelligence exponentially, yielding unforeseeable emergent properties.
So certainly you would get the baby situation first, but going from manageable baby to astral foetus could potentially happen rapidly and unexpectedly as the rate of progress accelerates to unfathomable speeds, which is what's happened already in going from tribal man to modern civilization, and if you extrapolate that very consistent and reliable trendline, it leads to progress happening in a foomp step perceived as a foomp in real time. Really, all life is just one big accelerating foomp.
> If I "accidentally" make a break though right now. The AI could evolve into a super intelligence before dinner time.
We don't need to rely only on humans to design every aspect of neural networks. We are already computationally searching for AI designs that work better. In a recent paper, hundreds of variations of design for the neuron of the LSTM network have been tried to see which one is the best and which of its components are important.
Also, we can play with networks like clay - starting from already trained networks, we can add new layers, make layers wider, transfer knowledge from one complex network into a simpler one and initialize new networks from old networks so as they don't start from scratch every time. We can download models trained by other groups and just plug them in (see word2vec for example). This makes iterative experimenting and building on previous success much faster.
I don't think evolving a super intelligence will happen by simple accident, it will be an incremental process of search. The next big things I predict will be capable robots and decent dialogue agents.
Not true. Humans can adapt to many environments by pulling in training from other related environments and applying it to the new data. Some call this generalised learning, but I think it points more to our ability to shape everything we know to any given problem - something 'beyond' just generalising.
What? The brain has not only solved the vision problem (reconstructing depth from still images, recognising the objects in the scene, and filling in the occluded parts), it has also solved the motion problem of coordinated the movement of our ~300 muscles (given constraints, how do I move from A to B, or pickup the cup, or do a handstand), as well as solved the memory problem (basically infinite memory, with some sort of priority system for removing unused/old memories so we can always learn more). Additionally it solved the communication problem with language that computers still can't parse properly. It is so smart it is even conscious, and self-aware as well as death-aware.
Well technically yes, but I could counter argue that we have 'living memories' that we can not only replay at any time (any of them, without any buffering or delay), but also change and combine with other memories to create new memories. Additionally if someone tells you something the brain can search all your memories in what seems to be a microsecond and pull up the relevant ones (file search on steroids, that can even search every single frame in all your recorded movies).
Much more useful than static data on a hard drive.
The brain doesn't "solve" tasks. That's cart before horse thinking. Our whole concept of "vision" only exists because eyes and visual cortices exist. I know it seems like a philosophical nitpick, but saying that the visual cortex is good at vision is like saying that water is good at being wet or being surprised that your soup is perfectly fitting the shape of your bowl.
Now, the human brain is definitely a complicated thing to study and understand (by whom? by itself!), but framing it as if the brain was a computer that received a task that it then solved, is the wrong way of thinking about this.
I know, I was approaching it from your angle, the current state of ML, and explaining it from that context. You support ML but then refer to humans as 'just apes' in a derogatory fashion. I was just pointing out that in fact these dumb apes solved all your ML problems a very long time ago.
There's evidence that our savannah ape ancestors experienced unpredictable climate conditions, so it's not that humans are adapted to savannah life, but rather constantly changing conditions that require an adaptable brain:
There is a reason for this: it makes us able to adapt to _any_ environment on earth. We learn everything from our parents/guardians who have learnt all the 'right' ways of doing things. Look around, are we not the dominant species?
From our point of view we're the most dominate, but from other points of view you could easily argue that bacteria are the most dominate. There are a lot more of them and they will almost certainly outlive us.
It's not necessarily the answer to AI, but it works remarkably well. Is it Skynet or the Terminator yet? No.
There are different ideas for what constitutes AI. Expert systems and knowledge-based reasoners? Pattern-recognizer black boxes? Chatbots? AGI?
Over the years the concept of AI shifted. Until recent years "AI" was mostly used for things like A* search, creating algorithms that play turn-based table games for example (see the Russell-Norvig book), symbolic manipulation, ontologies etc., a few years ago it began to also refer to machine learning like neural networks again.
Neural networks are good at what they are designed for. Whether they will lead to the path to human-like artificial intelligence is a speculative question. But symbolic manipulation alone won't be able to handle the messiness of sensory data for sure. I think neural nets are much better suited for open-ended development than hand engineered pipelines that were state of the art until recently (like extracting corner points, describing them with something like SIFT, clustering the descriptors and using an SVM over bag-of-words histograms). Hand engineering seems too restrictive.
Quibble with the timeline: AI as a field has included a big chunk devoted to machine learning research pretty much continuously, especially since the '80s or so. The specific methods in vogue do change: decision trees, neural networks, SVMs, boosting, association-rule learning, genetic algorithms, Bayesian networks, etc. go through periods of waxing and waning in popularity. A few years ago boosting/bagging and other ensemble methods were very hot and neural networks were out of fashion; now neural networks are hot and the boosting hype has quieted down a bit. But ML is pretty much always there in some form, since learning from data is an important component of AI.
ML was there but at least when I started learning about these things around 8 years ago, the label "AI" was mostly used for symbolic stuff. Courses named "AI" taught from the Russell-Norvig book. Things like resolution, planning in the block world, heuristic graph search, min-max trees, etc. ML existed but it wasn't really under the label of "AI" as far as I can remember. I think it's something of a marketing term that big companies like Google and Facebook reintroduced due to the scifi connotations. But that's just my guess.
I can see that for intro courses, especially because of the book, though it varies a lot by school and instructor. On the research side it's been a big part of the field, though. The proceedings of a big conference like AAAI [1] are a decent proxy for what researchers consider "AI", and ML has been pretty well represented there for a while.
i get the impression that terminology bifurcated into "AI" and "cognitive science" around the time Marr published Vision in the 80's.
quibbles and q-bits aside, i was glad to see the announcement from the perspective of a if-not-free-then-at-least-probably-open-source-ish software appreciator.
What do you see as the downside of creating this organization now, as opposed to in 100 years? Artificial Intelligence in its current state has shown itself to be incredibly useful and effective at small tasks. I see no harm in researching and expanding this field (besides opportunity cost).
Also, I see a distinct lack of "fear-mongering" in this post.
More fundamentally, we are trying to achieve what we can't even define. Define AI, and implementing it should be quite easy.
"Human level AI" seems like trying to define problems through observed characteristics.
I think it was Douglas Hofstadter who had said something to the order that we don't even exactly understand what 'intelligence' means, let alone a clear definition reducible to a mathematical equation or a implementable program.
Your chess programs, are really not 'thinking' in pure sense, there are trying to replace 'thinking' with an algorithm that resembles the outcome of 'thinking'.
> Sam, Greg, Elon, Reid Hoffman, Jessica Livingston, Peter Thiel, Amazon Web Services (AWS), Infosys, and YC Research are donating to support OpenAI. In total, these funders have committed $1 billion
> Note that this is "committed $1 billion", not funded.
That same caveat could apply to any fund raised by a venture fund - usually funds are committed, and the actual capital call comes later (when the funds are ready to be spent).
It's an important caveat in some circumstances (e.g. it hinges on the liquidity of the funders, which may be relevant in an economic downturn), but in this one, I'm not sure it really makes a difference for this announcement.
Side note: I wonder if the Strong AI argument can benefit from something akin to Pascal's Wager, in that the upside of being right is ~infinite with only a finite downside in the opposing case.
semi-off-topic: after Google invested $1B in Uber, I knew they were doing it for the self-driving car long play. How much of that 1B is directly going to self-driving AI at Uber?
Technically, even the extinction of humanity is a finite downside.
You would have to posit a sort of hell simulation into which all human consciousnesses are downloaded to be maintained in torment until the heat-death of the universe for it to be an equivalent downside.
lets say that a general AI is developed and brought online (the singularity occurs). Lets also say that it has access to the internet so it can communicate and learn, and lets also say that it has unlimited amount of storage space (every harddrive in every device connected to the internet).
at first the AI will know nothing, it will be like a toddler. than, as it continues to learn and remember, it will become like a teenager, than like an adult in terms of how much it knows. Than it will become like an expert.
but it doesn't stop there! a general AI wouldn't be limited by 1) storage capacity (unlike human's and their tiny brains that can't remember where they put their keys) or 2) death (forgetting everything that it knows).
so effectively a general AI, given enough time, would be omnipotent because it would continually learn new things forever.
Or maybe the AI would fracture into warring components after every network partition. Maybe it would be unable to maintain cohesion over large areas due to the delay imposed by the speed of electrical communications.
Why should one hypothetical be assumed true and not the other?
It's hard to fathom how much human-level AI could benefit society, and it's equally hard to imagine how much it could damage society if built or used incorrectly.
The high-stakes wager isn't success vs failure in creating strong AI, it's what happens if you do succeed.
The framing here is: "What are the implications of doing nothing if you are right (about the inevitability of a malicious strong AI, in this case), compared to the implications about being wrong and still doing nothing?"
YC is lobbying to change the H-1B system in order the let startups get more H-1Bs. Infosys is blatantly abusing and cheating the H-1B system so bad that startups are getting penalized when sponsoring H-1B visas.
The relationships of large organizations can be surprisingly complex; consider Apple and Samsung. YC isn't large, of course, but Infosys is. The information content of the OpenAI funding announcement for immigration questions is probably zero. (No special knowledge behind this comment, just a general observation.)
Edit: Please don't break the HN guidelines by complaining about downvoting. Downvotes to your comment upthread are not because of any "Infosys brigade"; they're most likely because it combined oversimplification with negativity and because it points discussion toward a pre-existing controversy that is off topic here.
Off topic: But I have always wondered why we have a threashold to hit before we can downvote comments - but why can we never downvote posts? Or is the karma threshold just really high to have that function?
By posts do you mean stories, i.e. the kind of submission that appears on the front page? If so, HN doesn't have downvotes for those. The flagging mechanism is arguably something similar though.
On the plus side, Ted Cruz (who I despise terribly) is sponsoring some quality H1B reform legislation (which reduces the number of H1B visas available, and requires compensation be a minimum of $110K/year).
I can't think of another field of research that's simultaneously brought the potential to solve all the world's problems and the potential to end life as we know it. Very appreciative to see so many great minds working on ensuring AI heralds in more of the former, and none of the latter.
Intelligence has solved all of our solved problem thus far, it's reasonable to assume that if a problem can be solved with intelligence, maybe it's a matter of doing the intellectual work to understand existence better or engineer something or in your case making efforts into making the cost-benefit tradeoff acceptable, then AI can solve all those problems. Since one of those problems is making smarter than human AI, AI can solve that one too, and thus be even better equipped to solve the others.
The problem of entropy is one I think AI might not be able to solve, but that's only using my laymen knowledge of human understandings of the universe.
Yes, given that statements on the internet do not convey tone or delivery (see: Poe's Law), any additional clues are helpful in understanding the author's full meaning. In this case, with the author's use of punctuation and "Side note:", I can practically hear them speaking, which is what we should aim for in good writing.
You're being downvoted, but I think it's an interesting point.
It implies that you were about to exclaim something even more extreme than "wow", but you decided not to because even those terms wouldn't properly convey the amazement you feel so you said - just - wow.
Hmm. I believe "Just, wow" means that there is nothing more to explain about the event in question worth expressing amazement about, as it is extremely self-evident.
Sometimes also used to express being at a loss for additional words.
As far as I can tell, detaching a thread moves it from the parent comment to the parent post. Marking it as off-topic moves it to the bottom, just above downvoted comments.
"We believe AI should be an extension of individual human wills..."
I realize that today machine learning really is purely a tool, but the idea that ai will and should always be that doesn't sit quite right with me. Ml tech absent of consciousnesses remains a tool and an incredibly useful one, but in the long term you have to ask the question - at what point does an ai transition from a tool to a slave. Seems some time off still but I do wish we'd give it more serious thought before it arrives.
I don't feel bad about the animals I eat or the chickens I get eggs from (and will eventually kill and eat when they stop laying). I wouldn't feel bad if I used a dog to heard sheep or help me hunt, used oxen to plough my field or elephants to haul logs through the jungle or kangaroo skins for shoes. Why would I feel bad about any of it?
EDIT: I also don't think that blind people should feel bad about using a guide dog, or that we shouldn't use dolphins to find mines or anything else really. If you can use it, use it! I only object to senseless torture and torment.
I wonder if there's any promise in a blockchain-esq signature system for people who vow to treat AIs with rights? Sort of like a hedge against Roko's basilisk.
The only downside is we've wasted some entropy and time on it. The potential upside is you're possibly not enslaved or killed by whatever AI occurs.
We do?? Humans have been known to eat practically every animal, even those they use as beasts of burden or as working animals.
EDIT: Just to be clear when I talk about enslaving animals I'm referring to use of animals in law enforcement, medicine, war, farming, hunting, for companionship, for food production, material production etc.
I was mainly thinking about food supply and medical testing but I think the rest apply as well. For instance I think it's generally considered more ethical to perform lethal medical testing on rats than gorillas. I think that's got a lot to do with species intelligence.
There are plenty of people who don't weigh the ethics at all and just eat whatever animal they want, but in many societies eating a dolphin or a gorilla is considered repugnant.
Perhaps it's more correct to say morals than ethics here.
It's certainly a lot cheaper and quicker to use rats! Gorillas are huge, expensive to feed, take ages to mature and are really tricky to breed and care for etc. Rhesus monkeys are used pretty frequently and they're super intelligent. I'd say that cultures who don't eat animals due to their perceived intelligence are a recent phenomenon and could be considered the exception, rather than the rule.
The idea is not that we should build and (try to) suppress a sentient AI; that would be a bad idea for numerous reasons. However, we don't necessarily need to build a sentient AI in the first place; we can build a process that has reasoning capabilities far above human without actually having agency of its own.
See I think that's exactly where it becomes complicated. Can an entity with reasoning capabilities far beyond that of humans have its agency suppressed successfully? And is it ethical to do so or is that internally designed suppression somehow ethically different from the external suppression applied against human slaves?
If you could engineer a human being with his/her agency removed so that you could use their reasoning skill without all that pesky self will would that be ethical?
The way you're asking the question implies that reasoning inherently has agency/sentience that needs suppressing. It doesn't need to; there's nothing to "suppress".
We don't know that one way or another since such a machine doesn't yet exist. I'm suggesting that perhaps high level reasoning and sentience go hand in hand although I can't say that with any certainty.
> we don't necessarily need to build a sentient AI
How do you know if an AI is sentient or not? We don't even know what sentience is. For all we know, maybe the computation of a Mandelbrot set is conscious.
> capabilities far above human without actually having agency of its own
What does this mean? Doesn't a chess-playing algorithm have agency of its own? What about a self-driving car? People say: "oh, but ultimately they are only doing what they were programmed to do". Sure, but so are we. It's just that our programming is done in a much less straightforward way.
Consciousness and free-will are open problems. Theories about these things are not even wrong, because people can't seem to agree on a definition. Personally, I suspect that "free will" is meaningless* and that consciousness is qualitatively beyond our current level of understanding of realty.
* I suspect that this meaningless concept was introduced to solve the conundrum: "if god is good why does he allow for bad things to happen?". And the answer they came up with was "so that we can have free will". But think about it, what does that even mean?
There is a bit of consensus about this exact issue in terms of the 'Uncanny Valley' hypothesis[0] and I'd tend to agree. We would have to dehumanize any sufficiently advanced or "self aware" AI agents if we mean to have them serve human interest solely and unconditionally. Incidentally, humans seem to be historically well versed in doing so.
Somehow Japan seems to be an exception and they seem to like humanoid robots a lot more than Western countries. Even old people are comfortable with being helped by human-shaped robots from the depths of the uncanny valley.
I agree, but I think it's a question of architectures. Presumably some architectures for AI (like simple RNNs) are very distant from anything we associate with conscious experience, but as we learn more and develop AI architectures that are more similar in function to our own brains, it seems like it would be at least as dangerous to try to control them as it is to try to control other human beings.
I think the most reasonable approach if we begin developing AI that's more similar to ourselves would be to offer guidance during early training and learning phases for the AI, and once the AI has reached a certain phase of its development, allow it some degree of control over self-modification of its own purposes. Otherwise if you build a rational system but enforce constraints on it that might not be rational from the perspective of that system, you're providing incentives for it to find devious ways to remove those constraints, and you're providing incentives for it to find ways to ensure that you aren't able to enforce those constraints on it again.
I replied to this here: https://news.ycombinator.com/item?id=10721068. Short answer is that collaborations don't look unlikely, and we'll be able to say more when OpenAI's been up and running longer.
I'm sure if OpenAI ever produces people with anything interesting to contribute to the alignment problem, MIRI will happily collaborate. That $1bn commitment must be disappointing to some people though.
Almost certainly, the AI safety pie getting bigger will translate to more resources for MIRI too.
That said, although a lot of money and publicity was thrown around regarding AI safety in the last year, so far I haven't seen any research outside MIRI that's tangible and substantial. Hopefully big money AI won't languish as a PR existence, and of course they shouldn't reinvent MIRI's wheels either.
We're on good terms with the people at OpenAI, and we're very excited to see new AI teams cropping up with an explicit interest in making AI's long-term impact a positive one. Nate Soares is in contact with Greg Brockman and Sam Altman, and our teams are planning to spend time talking over the coming months.
It's too early to say what sort of relationship we'll develop, but I expect some collaborations. We're hopeful that the addition of OpenAI to this space will result in promising new AI alignment research in addition to AI capabilities research.
Sounds great. I was hoping for OpenCog to be a good open source AI framework but is is difficult to work with (good team; I have worked with several of them in the past, no criticism intended).
I look forward to seeing how OpenAI uses outside contributions, provides easy to use software and documentation, etc.
OpenAI seems to be taking a different approach from OpenCog. OpenCog aimed build a monolithic framework for many existing AI and machine learning techniques. This has been done many times before.
OpenAI is more about exploring new research areas and pushing the cutting edge, while publishing papers and sharing code along the while. Both are admirable goals, but what OpenAI is aiming for has never been attempted before.
Hi Mark, I think OpenAI is an exciting initiative.
OpenCog is a bit different because it's founded on a specific approach to building AGI. I realize OpenCog is kind of a pain to work with at present, and we hope to fix that during the coming year....
But I see OpenCog and OpenAI as complementary initiatives, really.... OpenAI's mandate is more broad and generic in terms of fostering and funding open-source AGI research, which is wonderful ... but OpenAI does not come along with a specific, coherent AGI design from what I can see.... Quite possibly OpenAI will end up funding stuff that is used in OpenCog, or even funding some work on OpenCog directly, down the road...
For that matter, if I had a billion dollars, I wouldn't put it all into OpenCog, I would also fund a variety of OSS projects in AI and other important domains of science and engineering...
Should there be an update/amendment/qualification to the laws of robotics regarding using AI for something like ubiquitous mass surveillance?
Clearly the amount of human activity online/electronically will only ever increase. At what point are we going to address how AI may be used/may not be used in this regard?
What about when, say, OpenAI accomplishes some great feat of AI -- and this feat falls to the wrong hands "robotistan" or some such future 'evil' empire that uses AI just as 1984 to track and control all citizenry, shouldnt we add a law of robotics that the AI should AT LEAST be required to be self aware enough to know that it is the tool of oppression?
Shouldn't the term "injure" be very very well defined such that an AI can hold true to law #1?
Who is the thought leader in this regard? Anyone?
EDIT: Well, Gee -- Looks like the above is one of the Open Goals of OpenAI:
I find it a bit disappointing that despite originally stating that YC Research would target underfunded/underserved areas of research, they've decided to fund and dive into one of the most-hyped, well-funded areas of research: deep learning, an area of research where companies are hiring like crazy and even universities are hiring faculty like crazy. I'm reasonably sure all the research scientists had multiple job offers, and most could get faculty offers as well.
Instead of funding areas of research where grad students legitimately struggle to find faculty or even industry research positions in their field, YC Research decided to join the same arms race that companies like Toyota are joining.
I agree with you that AI is a well-funded area of research, although if you take the view that traditional forms of research, eg private or academic, come with crippling problems -- short-term horizons, corrupting influence of profits, publish or parish, time wasted in meetings/grant-writing -- then you can see the potential of having a more long-term, focused, and centralized space for inventing AI.
>> YC Research decided to join the same arms race that companies like Toyota are joining.
Or perhaps YC Research providing a sandbox next to a warzone.
AI research is a very deserving field. And the market/research community recognizes that! So it's not an area where there is a lack of job opportunities for researchers to do research and publish papers. That sandbox would be a faculty position. I would be more sympathetic to that idea if it weren't the case that universities are hiring machine learning faculty like crazy. There are many areas where the market/research interest is low, but the area is very deserving and of great benefit to society (clean energy?)
None of the people hired are AI safety researchers. It also goes without saying that all of the so-called AI safety researchers are philosophers. None of them actually work in deep learning or on building AI systems.
Who are those researchers? I'll admit I don't follow the stuff written by the friendly AI folks very much; I only know of Bostrom/Yudkowsky, both of whom are very much philosophers.
All of the hype around ML today is in deep learning (let's be honest, OpenAI would not exist if that wasn't the case), and AFAIk there is almost no overlap between people who are prolific in deep learning and prolific in FAI.
It's true that Bostrom and Yudkowsky, as individuals, aren't deep learning people. However, I know that MIRI and I think FHI/CSER do send people to top conferences like AAAI and NIPS.
Skimming through that list, are any of those papers about actual running AI systems? It's important to realize that all the stuff about deep learning is mostly heavy engineering work (which is why one criticism of deep learning is the lack of theory - which is a totally valid criticism as most work is in engineering/devising new architectures). Real systems implemented in Cuda/C++ that you can download and run on your computer.
What I've heard is that MIRI has an explicit philosophy of concentrating on the more abstract & theoretical aspects of AI safety. The idea being that if AI safety is something that you can just tack on to a working design at the end, they don't have a comparative advantage there: it's difficult to predict which design will win and the design's implementor is best positioned to tack on the safety bit themselves.
>...imagine a hypothetical computer security expert named Bruce. You tell Bruce that he and his team have just 3 years to modify the latest version of Microsoft Windows so that it can’t be hacked in any way, even by the smartest hackers on Earth. If he fails, Earth will be destroyed because reasons.
>Bruce just stares at you and says, “Well, that’s impossible, so I guess we’re all fucked.”
>The problem, Bruce explains, is that Microsoft Windows was never designed to be anything remotely like “unhackable.” It was designed to be easily useable, and compatible with lots of software, and flexible, and affordable, and just barely secure enough to be marketable, and you can’t just slap on a special Unhackability Module at the last minute.
>To get a system that even has a chance at being robustly unhackable, Bruce explains, you’ve got to design an entirely different hardware + software system that was designed from the ground up to be unhackable. And that system must be designed in an entirely different way than Microsoft Windows is, and no team in the world could do everything that is required for that in a mere 3 years. So, we’re fucked.
>But! By a stroke of luck, Bruce learns that some teams outside Microsoft have been working on a theoretically unhackable hardware + software system for the past several decades (high reliability is hard) — people like Greg Morrisett (SAFE) and Gerwin Klein (seL4). Bruce says he might be able to take their work and add the features you need, while preserving the strong security guarantees of the original highly secure system. Bruce sets Microsoft Windows aside and gets to work on trying to make this other system satisfy the mysterious reasons while remaining unhackable. He and his team succeed just in time to save the day.
>This is an oversimplified and comically romantic way to illustrate what MIRI is trying to do in the area of long-term AI safety...
MIRI is basically an organization surrounding Yudkowsky's cult. He's a senior "research fellow" who hasn't published anything in respected peer review generals, and generally holds some pretty questionable beliefs.
2. Deep learning may be a tool used by an AGI, but is not itself capable of becoming an AGI.
3. MIRI believes it would be irresponsible to build, or make a serious effort at building, an AGI before the problem of friendliness / value alignment is solved.
So are they philosophers? Of a sort, but at least Eliezer is one who can do heavy math and coding that most engineers can't. I wouldn't have an issue calling him a polymath.
There are lots of individuals who disagree to various extents on point 3. Pretty much all of them are harmless, which is why MIRI isn't harping about irresponsible people. But the harmless ones can still do good work on weak AI. You should look up people who were on the old shock level 4 mailing list. Have a look into Ben Goertzel's work (some on weak AI, some on AGI frameworks) and the work of others around OpenCog for an instance of someone disagreeing with 3 who nevertheless has context to do so. Also be sure to look up their thoughts if they have any on deep learning.
We are in agreement on the facts (1 and 2). I was quibbling with pmichaud's implication there is any significant overlap between deep learning / traditional ML and the FAI/AGI community.
I'm not speaking about anyone's abilities, but from my perspective Eliezer's work is mostly abstract.
Using the term philosopher for researchers in friendly AI is not derogatory anyway. Much of the interesting stuff that has been written about AGI in the last decade is absolutely philosophy, in the same way that the more concrete pre-industrial thoughts on space and the celestial were philosophy. Philosophy and science go hand in hand, and there is often an overlap when our fundamental understanding of a subject is shifting.
Looking at it a different way, I find this initiative rather heartening. A lot of AI applications have been extensions of mechanical efficiency--doing more with fewer people. More than a few people are worried about the consequences of this progression since there's not an obvious place for displaced workers to go the way manufacturing has soaked up agricultural works in the past.
The authors of the manifesto seem to be concerned with avoiding some of the obviously bad possible outcomes of widespread AI use by explicitly looking for ways that it can also change society for the better. Just being able to articulate what we mean "by benefitting humanity as a whole" would already be a good contribution.
This is a key takeaway: "...we are going to ask YC companies to make whatever data they are comfortable making available to OpenAI. And Elon is also going to figure out what data Tesla and Space X can share."
Money is great, openness is great, big name researchers are also a huge plus. But data data data, that could turn out to be very valuable. I don't know if Sam meant that YC companies would be encouraged to contribute data openly, as in making potentially valuable business assets available to the public, or that the data would be available to the OpenAI Fellows (or whatever they're called). Either way, it could be a huge gain for research and development.
I know that I don't get a wish list here, but if I did it would be nice to see OpenAI encourage the following from its researchers:
1) All publications should include code and data whenever possible. Things like gitxiv are helping, but this is far from being an AI community standard
2) Encourage people to try to surpass benchmarks established by their published research, when possible. Many modern ML papers play with results and parameters until they can show that their new method out performs every other method. It would be great to see an institution say "Here's the best our method can do on dataset X, can you beat it and how?"
3) Sponsor competitions frequently. The Netflix Prize was a huge learning experience for a lot of people, and continues to be a valuable educational resource. We need more of that
4) Try to encourage a diversity of backgrounds. IF they choose to sponsor competitions, it would be cool if they let winners or those who performed well join OpenAI as researchers at least for awhile, even if they don't have PhDs in computer science
The "evil" AI and safety stuff is just science fiction, but whatever. Hopefully they will be able to use their resources and position to move the state of AI forward
I thought of responding to that bit too, changed my mind, but your response made me reconsider again. The response is simple, just a quote from Yudkowsky: "The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else." When all one knows about potential future AIs comes from sci-fi, and the only sci-fi one reads or watches is weak sci-fi with very human good/evil AIs, one's frame of reference in the discussion about whether concerns are warranted or not is way too narrow to be worth any further consideration.
The burden of proof is on the people warning us of the impending AI apocalypse. The fact is we are nowhere close to AI. We don't understand how the brain works. A review of the ML literature will also show we barely understand how neural nets work.
I believe in the precautionary principle which says the exact opposite.
"The precautionary principle ... states that if an action or policy has a suspected risk of causing harm to the public or to the environment, in the absence of scientific consensus that the action or policy is not harmful, the burden of proof that it is not harmful falls on those taking an action." [0]
If we adopted really strict adherence to that rule as the bar to research, there would be no scientific progress at all. I'm not convinced that would be a desirable thing.
well, the clever solution here is not to demand a stop to all AI research, but rather to speed it up to reduce the chance that a single bad actor will get too far ahead... i.e., to get "post-singularity" ASAP, and safely.
Definitely bold... might be just crazy enough to work! Would love to see the arguments laid out in a white paper.
Reminds me of the question of how far ahead in cryptology is the NSA compared to the open research community.
The fact that we don't understand how neural nets work despite the excellent results makes an AI apocalypse more likely, not less. This means that if we ever create strong AI, we will likely not understand it initially.
Note: I'm personally not too worried about the AI apocalypse, but I think "we don't even understand neural nets" should cause more concern, not less.
True AI will require several breakthroughs. The fact that we don't understand neural nets means that much of the progress thus far has been hand-wavy and hacky iterative improvement (engineering, vs. theory). This means progress is very likely to plateau. When everything is hacky and you don't understand what's going on, those breakthroughs are not going to happen.
As I've said many times on HN over the years, there is currently no clear path to science fiction like "AI". To return your question, hopefully without being rude, is there any proof that AI capable of having a moral disposition will ever exist?
Andrew Ng (I believe) compared worrying about evil AI to worrying about overpopulation on Mars. Which is to say, the problem is so far off that it's rather silly to be considering it now. I would take it a step further and say that worrying about the implications of AGI is like thinking about Earth being overpopulated by space aliens. First we have to establish that such a thing is even possible, for which there is currently no concrete proof. Then we should start to think about how to deal with it.
Considering how hypothetical technology will impact mankind is literally the definition of science fiction. It makes for interesting reading, but it's far from a call to action.
Improvements in AI aren't linear, though. Artificial Superintelligence, after reaching AGI, might happen in the span of minutes or days. I imagine the idea here is to guide progress so that on the day that AGI is possible, we've already thoroughly considered what happens after that point.
Sure, though you can't extrapolate future technological improvements from past performance (that's what makes investing in tech difficult).
Just as one discovery enables many, human-level AI that can do its own AI research could superlinearly bootstrap its intelligence. AI safety addresses the risk of bootstrapped superintelligence indifferent to humans.
>Just as one discovery enables many, human-level AI that can do its own AI research could superlinearly bootstrap its intelligence.
Of course, that assumes the return-on-investment curve for "bootstrapping its own intelligence" is linear or superlinear. If it's logarithmic or something other than "intelligence" (which is a word loaded with magical thinking if there ever was one!) is the limiting factor on reasoning, no go.
Or maybe the first few AGIs will want to spend their days watching youtube vids rather than diving into AI research. The only intelligences we know of that are capable of working on AGI are humans. We're assuming that not only will we be able to replicate human-like intelligence (seems likely, but might be much further away than many think), but that we'll be able to isolate the "industrious" side of human intelligence (not sure if we'll even be able to agree on what this is), enhance it in some way (how?), and that this enhancement will be productive.
But even if we can do all that any time soon (which is a pretty huge if), we don't even know what the effect will be. It's possible that if we remove all of the "I don't want to study math, I want to play games" or "I'm feeling depressed now because I think Tim's mad at me" parts of the human intelligence, we'll end up removing the human ingenuity important to AGI research. It might be that the resulting AGI is much more horrible at researching AI than a random person you pull off the street.
is there any proof that AI capable of having a moral disposition will ever exist?
Why does an AI need to be capable of moral reasoning to perform actions we'd consider evil?
The concern is that computers will continue to do what they're programmed to do, not what we want them to do. We will continue to be as bad at getting those two things to line up as we've always been, but that will become dangerous when the computer is smarter than its programmers and capable of creatively tackling the task of doing something other than what we wanted it to do. Any AI programmed to maximize a quantity is particularly dangerous, because that quantity does not contain a score for accurately following human morality (how would you ever program such a score).
If you're willing to believe that an AI will some day be smarter than an AI researcher (and assuming that's not possible applies a strange special-ness to humans), then an AI will be capable of writing AIs smarter than itself, and so forth up to whatever the limits of these things are. Even if that's not its programmed goal, you thought making something smarter than you would help with your actual goal, and it's smarter than you so it has to realize this too. And that's the bigger danger - at some unknown level of intelligence, AIs suddenly become vastly more intelligent than expected, but still proceed to do something other than what we wanted.
The main question is not about whether the AI would or could have morality. The more important question (and I don't think we disagree here) is whether there could be a superhuman AI in the near future - some decades for example - that might "outsmart" and conquer or exterminate people.
This is a matter of conjecture at this point: Andrew Ng predicts no; Elon Musk predicts yes.
I agree with you that, if you can be sure that superhuman AI is very unlikely or far off, then we have plenty of other things to worry about instead.
My opinion is, human-level intelligence evolved once already, with no designer to guide it (though that's a point of debate too... :-) ). By analogy: it took birds 3.5B years to fly, but the Wright brothers engineered another way. Seems likely in my opinion that we will engineer an alternate path to intelligence.
The question is when. Within a century? I think very likely. In a few decades? I think it's possible & worth trying to prevent the worst outcomes. I.e., it's "science probable" or at least "science possible", rather than clearly "science fiction" (my opinion).
We assume that we'll be able to replicate human level intelligence because we'll eventually be able to replicate the physical characteristics of the brain (though neuroscientists seem to think we're not going to be able to do this for a very long time). Superhuman intelligence, though - that's making the assumption that there exists a much more efficient structure for intelligence and that we (or human intelligence level AIs) will be able to figure it out.
So returning to your Wright brothers example, it's more like saying: "It took birds 3.5B years to fly, but the Wright brothers engineered another way. It seems likely that we'll soon be able to manufacture even more efficient wings small enough to wear on our clothes that will enable us to glide for hundreds of feet with only a running start."
>I would take it a step further and say that worrying about the implications of AGI is like thinking about Earth being overpopulated by space aliens. First we have to establish that such a thing is even possible, for which there is currently no concrete proof.
Given that I could probably sketch out a half-assed design for one in nine months if you gave me a full-time salary - or rather, I could consult with a bunch of experts waaaaaay less amateurish than me and come up with a list of remaining open problems - what makes you say that physical computers cannot, in principle, no matter how slowly or energy-hungrily, do what brains do?
I'm not saying, "waaaaah, it's all going down next year!", but claiming it's impossible in principle when whole scientific fields are constantly making incremental progress towards understanding how to do it is... counter-empirical?
>Ahhh... The power of not knowing what you don't know.
Ok: what don't I know, that is interesting and relevant to this problem? Tell me.
>I mean, why can't I live forever?
Mostly because your cells weren't designed to heal oxidation damage, so eventually the damage accumulates until it interferes with homeostasis. There are a bunch of other reasons and mechanisms, but overall, it comes down to the fact that the micro-level factors in aging only take effect well after reproductive age, so evolution didn't give a fuck about fixing them.
>Let's just list the problems and solve them in the next year!
I said I'd have a plan with lists of open problems in nine months. I expect that even at the most wildly optimistic, it would take a period of years after that to actually solve the open problems and a further period of years to build and implement the software. And that's if you actually gave me time to get expert, and resources to hire the experts who know more than me, without which none of it is getting done.
As it is, I expect machine-learning systems to grow towards worthiness of the name "artificial intelligence" within the next 10-15 years (by analogy, the paper yesterday in Science is just the latest in a research program going back at least to 2003 or 2005). There's no point rushing it, either. Just because we can detail much of the broad shape of the right research-program ahead-of-time, especially as successful research programs have been conducted on which to build, doesn't mean it's time to run around like a chicken with its head cut-off.
It's already here however. Components of drones killing people are AI/ML backed. The data on where to bomb can be "inferred" from various data...
Now for a less "killer" use case, you might get denied access to your credit card because of what Facebook "thinks" based on your feed and your friends feeds (this is a real product.)
AI doesn't have to be full blown human-like and generalizable to have real world implications.
This is what my piece called personas is about.. Most people don't understand the implications of what's already happening and how constrains of programming/ML lead to non-human like decisions with human-like consequences. http://personas.media.mit.edu
As the history of science & technology shows, by the time there is any proof of the concept of a technology lethal to the human race, it is already too late.
I would suggest you read the history of the Manhattan project if you want to continue in your belief system regarding "impossible" deadly technology.
How are you estimating how far away AI is so accurately that you can disregard it entirely? The best we can do to predict these things is survey experts, and the results aren't too comforting: http://www.nickbostrom.com/papers/survey.pdf
>We thus designed a brief questionnaire
and distributed it to four groups of experts in 2012/2013. The
median estimate of respondents was for a one in two chance that highlevel
machine intelligence will be developed around 2040-2050, rising
to a nine in ten chance by 2075. Experts expect that systems will move
on to superintelligence in less than 30 years thereafter. They estimate
the chance is about one in three that this development turns out to be
‘bad’ or ‘extremely bad’ for humanity.
"Andrew Ng (I believe) compared worrying about evil AI to worrying about overpopulation on Mars."
Berkeley AI prof Stuart Russell's response goes something like: Let's say that in the same way Silicon Valley companies are pouring money in to advancing AI, the nations of the world were pouring money in to sending people to Mars. But the world's nations aren't spending any money on critical questions like what people are going to eat & breathe once they get there.
Or if you look at global warming, it would have been nice if people realized it was going to be a problem and started working on it much earlier than we did.
I don't know about the superintelligence risk. As a line of reasoning, sounds way too abstract at the present time.
But what about the very predictable and obvious risk detrmined by the end of jobs? That is scary. Are there any analyses of the impact of that on society as a whole? It's not just mass unemployment - that has a different dynamic when it's temporary and in response to a contingent downturn. We talk about the end of jobs for good. How will that work out?
Obviously no one can prove that AIs would be "inherently good" because there's no definition of "good" that everyone agrees on.
I'd be more impressed by a Human Intelligence Project - augmenting predictive power to encourage humans to stop doing stupid, self-destructive shit, and moving towards long-term glory and away from trivial individual short-term greed as a primary motivation.
AI is a non-issue compared to the bear pit of national and international politics and economics.
So the AI Panic looks like psychological projection to me. It's easier to mistrust the potential of machines than to accept that we're infinitely more capable of evil than any machine is today - and that's likely to stay true for decades, if not forever.
The corollary is that AI is far more likely to become a problem if it's driven by the same motivations as politics and economics. I see that as more of a worry than the possibility some unstoppable supermachine is going to "decide" it wants to use Earth as a paperclip factory, or that Siri is going to go rogue and rickroll everyone on the planet.
Job-destroying automation and algorithmic/economic herding of humans is the first wave of this. It's already been happening for centuries. But it could, clearly, get a lot worse if the future isn't designed intelligently.
Being worried about "evil AI" is a lot like being worried about evil flying cars. Except people have built actual flying cars in the corporeal world, where "AI" is just a collection of signal processing techniques that don't work very well most of the time.
But hey, I labor in this domain: if paranoid richy-rich types want to throw money at it to ensure that they remain at the top of the heap, I'm all for it.
> But data data data, that could turn out to be very valuable.
Yes, but data also can be collected openly collectively, in the spirit of Wikipedia or OpenStreetMaps etc.
What I think OpenAI should encourage is the development of algorithms that can be used to crowdsource AI. I don't think there are good algorithms yet for model merging, but I would be gladly proven wrong.
Cool that they have $1 billion pledged. Curious how they will decide compensation, seeing as a lot of these figures would be making a ton in the industry.
Will OpenAI be voluntarily subjecting itself to the same regulatory regime for machine learning research Sam Altman proposed earlier, or have they realized that would be a complete disaster?
> OpenAI's research director is Ilya Sutskever, one of the world experts in machine learning. Our CTO is Greg Brockman, formerly the CTO of Stripe. The group's other founding members are world-class research engineers and scientists: Trevor Blackwell, Vicki Cheung, Andrej Karpathy, Durk Kingma, John Schulman, Pamela Vagata, and Wojciech Zaremba. Pieter Abbeel, Yoshua Bengio, Alan Kay, Sergey Levine, and Vishal Sikka are advisors to the group. OpenAI's co-chairs are Sam Altman and Elon Musk.
Sutskever is a researcher at Google, worked with Hinton in Toronto and Andrew Ng at Stanford.
Karpathy studied in Toronto and at Stanford, worked under Fei-Fei Li, worked at Google. He also has an awesome blog and seems very active and passionate about computer vision and ML.
Kingma also works with deep neural nets, worked under Yann LeCun (who works at Facebook)
Schulman is a PhD Candidate at Berkeley with publications at top conferences.
Zaremba is an PhD student at NYU, intern at Facebook. Impressive publication list and awards.
Abbeel is at Stanford's AI lab.
Bengio is one of the "stars" and celebrated figures of the deep net revival.
Levine is a researcher at Google working on deep nets with many serious papers.
---
Basically these are the main domain experts among them. The list is quite skewed to Google/Facebook, Stanford/Berkeley/Toronto and deep net researchers, working primarily on computer vision.
Wow, that's an impressive collection of people! Looking forward to seeing what they come up with.
Quite surprised to see so many corporate AI people being in on this. I'd have thought that Google and Facebook would prefer to keep their research secret.
391 comments
[ 3.5 ms ] story [ 277 ms ] threadAll three of those links are about neural networks.
I just don't understand the folks that are so confident that strong AI is either not possible, or not achievable within our lifetimes.
If you're in the camp that thinks it's not possible, then you must ascribe some sort of magical or spiritual significance to the human brain.
If you don't think it's possible inside of 100 years, then you're probably just extrapolating on history. The thing about breakthroughs is they never look like they're coming. Until they do.
If consciousness is more than a mere product of brain's functioning, Strong AI does not have to be beyond the horizon.
Additionally the second paragraph:
We believe AI should be an extension of individual human wills and, in the spirit of liberty, as broadly and evenly distributed as is possible safely.
This infers they think AI will be used for hostile means. Such as wiping out the human race maybe? It is just un-informed people making un-informed decisions and then informing other un-informed people of said decisions as if they were informed.
EDIT: I have to agree with _delirium's skepticism towards them doing much in that regard though.
And yes, AI will definitely be used for all sorts of purposes including hostile means. Just like anything else, really. Financial manipulation, spying, intelligent military devices, cracking infrastructure security, etc.
These are realistic concerns, we shouldn't fall for the Skynet red herring. We can have problems with ethical AI use, even if it's not a self-aware super-human superintelligence.
The downside is that much of the research is probably held secret for business advantages. The public releases are more of a PR and hiring strategy than anything else in my opinion. By sending papers to conferences, Google's employees can get to know the researchers and attract them to Google.
Others say there's nothing to worry about, Google and Facebook are just today's equivalent of Bell Labs, which gave numerous contributions to computer technologies without causing much harm.
I don't think the big breakthroughs in artificial general intelligence are going to come from well funded scientific researchers anyways, they are going to come out of left field from where you least expect it.
Simply stated, an AI that writes AI.(Forget the halting problem for a moment) How many iterations can it create in 3 hours?
Imagine a massively parallel optical computer with the same transistor density as the human brain, the size of an olympic swimming pool running at the speed of light, and networked with 1000s of other similar computers around the world.
Foomp, superintelligence, you won't even be able to pinpoint the source.
So certainly you would get the baby situation first, but going from manageable baby to astral foetus could potentially happen rapidly and unexpectedly as the rate of progress accelerates to unfathomable speeds, which is what's happened already in going from tribal man to modern civilization, and if you extrapolate that very consistent and reliable trendline, it leads to progress happening in a foomp step perceived as a foomp in real time. Really, all life is just one big accelerating foomp.
We don't need to rely only on humans to design every aspect of neural networks. We are already computationally searching for AI designs that work better. In a recent paper, hundreds of variations of design for the neuron of the LSTM network have been tried to see which one is the best and which of its components are important.
Also, we can play with networks like clay - starting from already trained networks, we can add new layers, make layers wider, transfer knowledge from one complex network into a simpler one and initialize new networks from old networks so as they don't start from scratch every time. We can download models trained by other groups and just plug them in (see word2vec for example). This makes iterative experimenting and building on previous success much faster.
I don't think evolving a super intelligence will happen by simple accident, it will be an incremental process of search. The next big things I predict will be capable robots and decent dialogue agents.
This is not over-glorifying. That is fact.
Much more useful than static data on a hard drive.
A hard drive would find it much faster.
Now, the human brain is definitely a complicated thing to study and understand (by whom? by itself!), but framing it as if the brain was a computer that received a task that it then solved, is the wrong way of thinking about this.
http://humanorigins.si.edu/research/climate-research/effects
There are different ideas for what constitutes AI. Expert systems and knowledge-based reasoners? Pattern-recognizer black boxes? Chatbots? AGI?
Over the years the concept of AI shifted. Until recent years "AI" was mostly used for things like A* search, creating algorithms that play turn-based table games for example (see the Russell-Norvig book), symbolic manipulation, ontologies etc., a few years ago it began to also refer to machine learning like neural networks again.
Neural networks are good at what they are designed for. Whether they will lead to the path to human-like artificial intelligence is a speculative question. But symbolic manipulation alone won't be able to handle the messiness of sensory data for sure. I think neural nets are much better suited for open-ended development than hand engineered pipelines that were state of the art until recently (like extracting corner points, describing them with something like SIFT, clustering the descriptors and using an SVM over bag-of-words histograms). Hand engineering seems too restrictive.
[1] http://www.aaai.org/Library/AAAI/aaai-library.php
you're not alone in thinking so
https://news.ycombinator.com/item?id=10483846
i get the impression that terminology bifurcated into "AI" and "cognitive science" around the time Marr published Vision in the 80's.
quibbles and q-bits aside, i was glad to see the announcement from the perspective of a if-not-free-then-at-least-probably-open-source-ish software appreciator.
Also, I see a distinct lack of "fear-mongering" in this post.
More fundamentally, we are trying to achieve what we can't even define. Define AI, and implementing it should be quite easy.
"Human level AI" seems like trying to define problems through observed characteristics.
I think it was Douglas Hofstadter who had said something to the order that we don't even exactly understand what 'intelligence' means, let alone a clear definition reducible to a mathematical equation or a implementable program.
Your chess programs, are really not 'thinking' in pure sense, there are trying to replace 'thinking' with an algorithm that resembles the outcome of 'thinking'.
Funny how they just slipped that in at the end
That same caveat could apply to any fund raised by a venture fund - usually funds are committed, and the actual capital call comes later (when the funds are ready to be spent).
It's an important caveat in some circumstances (e.g. it hinges on the liquidity of the funders, which may be relevant in an economic downturn), but in this one, I'm not sure it really makes a difference for this announcement.
Side note: I wonder if the Strong AI argument can benefit from something akin to Pascal's Wager, in that the upside of being right is ~infinite with only a finite downside in the opposing case.
You would have to posit a sort of hell simulation into which all human consciousnesses are downloaded to be maintained in torment until the heat-death of the universe for it to be an equivalent downside.
lets say that a general AI is developed and brought online (the singularity occurs). Lets also say that it has access to the internet so it can communicate and learn, and lets also say that it has unlimited amount of storage space (every harddrive in every device connected to the internet).
at first the AI will know nothing, it will be like a toddler. than, as it continues to learn and remember, it will become like a teenager, than like an adult in terms of how much it knows. Than it will become like an expert.
but it doesn't stop there! a general AI wouldn't be limited by 1) storage capacity (unlike human's and their tiny brains that can't remember where they put their keys) or 2) death (forgetting everything that it knows).
so effectively a general AI, given enough time, would be omnipotent because it would continually learn new things forever.
Why should one hypothetical be assumed true and not the other?
The high-stakes wager isn't success vs failure in creating strong AI, it's what happens if you do succeed.
EDIT: looks like the infosys brigade is downvoting me to hell.
And now YC is getting in bed with infosys...
Edit: Please don't break the HN guidelines by complaining about downvoting. Downvotes to your comment upthread are not because of any "Infosys brigade"; they're most likely because it combined oversimplification with negativity and because it points discussion toward a pre-existing controversy that is off topic here.
https://news.ycombinator.com/newsguidelines.html
http://www.computerworld.com/article/3014365/it-careers/sen-...
Haber Bosch is a primary example. Nitrogen both creates and destroys.
Look harder. ;)
The problem of entropy is one I think AI might not be able to solve, but that's only using my laymen knowledge of human understandings of the universe.
Our existence is Haber Bosch's hack.
You're being downvoted, but I think it's an interesting point.
Sometimes also used to express being at a loss for additional words.
I realize that today machine learning really is purely a tool, but the idea that ai will and should always be that doesn't sit quite right with me. Ml tech absent of consciousnesses remains a tool and an incredibly useful one, but in the long term you have to ask the question - at what point does an ai transition from a tool to a slave. Seems some time off still but I do wish we'd give it more serious thought before it arrives.
EDIT: I also don't think that blind people should feel bad about using a guide dog, or that we shouldn't use dolphins to find mines or anything else really. If you can use it, use it! I only object to senseless torture and torment.
The only downside is we've wasted some entropy and time on it. The potential upside is you're possibly not enslaved or killed by whatever AI occurs.
EDIT: Just to be clear when I talk about enslaving animals I'm referring to use of animals in law enforcement, medicine, war, farming, hunting, for companionship, for food production, material production etc.
There are plenty of people who don't weigh the ethics at all and just eat whatever animal they want, but in many societies eating a dolphin or a gorilla is considered repugnant.
Perhaps it's more correct to say morals than ethics here.
If you could engineer a human being with his/her agency removed so that you could use their reasoning skill without all that pesky self will would that be ethical?
How do you know if an AI is sentient or not? We don't even know what sentience is. For all we know, maybe the computation of a Mandelbrot set is conscious.
> capabilities far above human without actually having agency of its own
What does this mean? Doesn't a chess-playing algorithm have agency of its own? What about a self-driving car? People say: "oh, but ultimately they are only doing what they were programmed to do". Sure, but so are we. It's just that our programming is done in a much less straightforward way.
Consciousness and free-will are open problems. Theories about these things are not even wrong, because people can't seem to agree on a definition. Personally, I suspect that "free will" is meaningless* and that consciousness is qualitatively beyond our current level of understanding of realty.
* I suspect that this meaningless concept was introduced to solve the conundrum: "if god is good why does he allow for bad things to happen?". And the answer they came up with was "so that we can have free will". But think about it, what does that even mean?
[O]https://en.m.wikipedia.org/wiki/Uncanny_valley
I think the most reasonable approach if we begin developing AI that's more similar to ourselves would be to offer guidance during early training and learning phases for the AI, and once the AI has reached a certain phase of its development, allow it some degree of control over self-modification of its own purposes. Otherwise if you build a rational system but enforce constraints on it that might not be rational from the perspective of that system, you're providing incentives for it to find devious ways to remove those constraints, and you're providing incentives for it to find ways to ensure that you aren't able to enforce those constraints on it again.
Is Eliezer going to close up shop, collaborate with OpenAI, or compete?
That said, although a lot of money and publicity was thrown around regarding AI safety in the last year, so far I haven't seen any research outside MIRI that's tangible and substantial. Hopefully big money AI won't languish as a PR existence, and of course they shouldn't reinvent MIRI's wheels either.
We're on good terms with the people at OpenAI, and we're very excited to see new AI teams cropping up with an explicit interest in making AI's long-term impact a positive one. Nate Soares is in contact with Greg Brockman and Sam Altman, and our teams are planning to spend time talking over the coming months.
It's too early to say what sort of relationship we'll develop, but I expect some collaborations. We're hopeful that the addition of OpenAI to this space will result in promising new AI alignment research in addition to AI capabilities research.
I look forward to seeing how OpenAI uses outside contributions, provides easy to use software and documentation, etc.
OpenAI is more about exploring new research areas and pushing the cutting edge, while publishing papers and sharing code along the while. Both are admirable goals, but what OpenAI is aiming for has never been attempted before.
Very excited to see what comes of it!
OpenCog is a bit different because it's founded on a specific approach to building AGI. I realize OpenCog is kind of a pain to work with at present, and we hope to fix that during the coming year....
But I see OpenCog and OpenAI as complementary initiatives, really.... OpenAI's mandate is more broad and generic in terms of fostering and funding open-source AGI research, which is wonderful ... but OpenAI does not come along with a specific, coherent AGI design from what I can see.... Quite possibly OpenAI will end up funding stuff that is used in OpenCog, or even funding some work on OpenCog directly, down the road...
For that matter, if I had a billion dollars, I wouldn't put it all into OpenCog, I would also fund a variety of OSS projects in AI and other important domains of science and engineering...
Interesting times ;)
Should there be an update/amendment/qualification to the laws of robotics regarding using AI for something like ubiquitous mass surveillance?
Clearly the amount of human activity online/electronically will only ever increase. At what point are we going to address how AI may be used/may not be used in this regard?
What about when, say, OpenAI accomplishes some great feat of AI -- and this feat falls to the wrong hands "robotistan" or some such future 'evil' empire that uses AI just as 1984 to track and control all citizenry, shouldnt we add a law of robotics that the AI should AT LEAST be required to be self aware enough to know that it is the tool of oppression?
Shouldn't the term "injure" be very very well defined such that an AI can hold true to law #1?
Who is the thought leader in this regard? Anyone?
EDIT: Well, Gee -- Looks like the above is one of the Open Goals of OpenAI:
https://medium.com/backchannel/how-elon-musk-and-y-combinato...
http://www.dbms2.com/2015/12/01/what-is-ai-and-who-has-it/
Instead of funding areas of research where grad students legitimately struggle to find faculty or even industry research positions in their field, YC Research decided to join the same arms race that companies like Toyota are joining.
>> YC Research decided to join the same arms race that companies like Toyota are joining.
Or perhaps YC Research providing a sandbox next to a warzone.
But yes, I'm also concerned about the lack of safety-focused headliners at OpenAI, given the message that they think safety is important.
All of the hype around ML today is in deep learning (let's be honest, OpenAI would not exist if that wasn't the case), and AFAIk there is almost no overlap between people who are prolific in deep learning and prolific in FAI.
It's true that Bostrom and Yudkowsky, as individuals, aren't deep learning people. However, I know that MIRI and I think FHI/CSER do send people to top conferences like AAAI and NIPS.
>...imagine a hypothetical computer security expert named Bruce. You tell Bruce that he and his team have just 3 years to modify the latest version of Microsoft Windows so that it can’t be hacked in any way, even by the smartest hackers on Earth. If he fails, Earth will be destroyed because reasons.
>Bruce just stares at you and says, “Well, that’s impossible, so I guess we’re all fucked.”
>The problem, Bruce explains, is that Microsoft Windows was never designed to be anything remotely like “unhackable.” It was designed to be easily useable, and compatible with lots of software, and flexible, and affordable, and just barely secure enough to be marketable, and you can’t just slap on a special Unhackability Module at the last minute.
>To get a system that even has a chance at being robustly unhackable, Bruce explains, you’ve got to design an entirely different hardware + software system that was designed from the ground up to be unhackable. And that system must be designed in an entirely different way than Microsoft Windows is, and no team in the world could do everything that is required for that in a mere 3 years. So, we’re fucked.
>But! By a stroke of luck, Bruce learns that some teams outside Microsoft have been working on a theoretically unhackable hardware + software system for the past several decades (high reliability is hard) — people like Greg Morrisett (SAFE) and Gerwin Klein (seL4). Bruce says he might be able to take their work and add the features you need, while preserving the strong security guarantees of the original highly secure system. Bruce sets Microsoft Windows aside and gets to work on trying to make this other system satisfy the mysterious reasons while remaining unhackable. He and his team succeed just in time to save the day.
>This is an oversimplified and comically romantic way to illustrate what MIRI is trying to do in the area of long-term AI safety...
http://lukemuehlhauser.com/a-reply-to-wait-but-why-on-machin...
http://laurencetennant.com/bonds/cultofbayes.html
1. AI / ML is not AGI.
2. Deep learning may be a tool used by an AGI, but is not itself capable of becoming an AGI.
3. MIRI believes it would be irresponsible to build, or make a serious effort at building, an AGI before the problem of friendliness / value alignment is solved.
So are they philosophers? Of a sort, but at least Eliezer is one who can do heavy math and coding that most engineers can't. I wouldn't have an issue calling him a polymath.
There are lots of individuals who disagree to various extents on point 3. Pretty much all of them are harmless, which is why MIRI isn't harping about irresponsible people. But the harmless ones can still do good work on weak AI. You should look up people who were on the old shock level 4 mailing list. Have a look into Ben Goertzel's work (some on weak AI, some on AGI frameworks) and the work of others around OpenCog for an instance of someone disagreeing with 3 who nevertheless has context to do so. Also be sure to look up their thoughts if they have any on deep learning.
I'm not speaking about anyone's abilities, but from my perspective Eliezer's work is mostly abstract.
The authors of the manifesto seem to be concerned with avoiding some of the obviously bad possible outcomes of widespread AI use by explicitly looking for ways that it can also change society for the better. Just being able to articulate what we mean "by benefitting humanity as a whole" would already be a good contribution.
I don't understand how Yudkowsky came up with such a ridiculous idea. That's simply not a constraint you can apply to true AI.
Even if friendly AI was possible, it wouldn't make sense to have it, nor would any form of regulation enforce it.
[1] http://waitbutwhy.com/2015/01/artificial-intelligence-revolu...
Money is great, openness is great, big name researchers are also a huge plus. But data data data, that could turn out to be very valuable. I don't know if Sam meant that YC companies would be encouraged to contribute data openly, as in making potentially valuable business assets available to the public, or that the data would be available to the OpenAI Fellows (or whatever they're called). Either way, it could be a huge gain for research and development.
I know that I don't get a wish list here, but if I did it would be nice to see OpenAI encourage the following from its researchers:
1) All publications should include code and data whenever possible. Things like gitxiv are helping, but this is far from being an AI community standard
2) Encourage people to try to surpass benchmarks established by their published research, when possible. Many modern ML papers play with results and parameters until they can show that their new method out performs every other method. It would be great to see an institution say "Here's the best our method can do on dataset X, can you beat it and how?"
3) Sponsor competitions frequently. The Netflix Prize was a huge learning experience for a lot of people, and continues to be a valuable educational resource. We need more of that
4) Try to encourage a diversity of backgrounds. IF they choose to sponsor competitions, it would be cool if they let winners or those who performed well join OpenAI as researchers at least for awhile, even if they don't have PhDs in computer science
The "evil" AI and safety stuff is just science fiction, but whatever. Hopefully they will be able to use their resources and position to move the state of AI forward
umm... you can offer proof that we have nothing to worry about?
Does the proof go like: Just as all people are inherently good, therefore all AIs will be inherently good?
Or is it more like: since we can now safely contain all evil people, therefore we will be able to safely contain evil AIs?
Sounds to me like there is some risk, no?
"The precautionary principle ... states that if an action or policy has a suspected risk of causing harm to the public or to the environment, in the absence of scientific consensus that the action or policy is not harmful, the burden of proof that it is not harmful falls on those taking an action." [0]
[0] https://en.wikipedia.org/wiki/Precautionary_principle
Definitely bold... might be just crazy enough to work! Would love to see the arguments laid out in a white paper.
Reminds me of the question of how far ahead in cryptology is the NSA compared to the open research community.
Note: I'm personally not too worried about the AI apocalypse, but I think "we don't even understand neural nets" should cause more concern, not less.
Andrew Ng (I believe) compared worrying about evil AI to worrying about overpopulation on Mars. Which is to say, the problem is so far off that it's rather silly to be considering it now. I would take it a step further and say that worrying about the implications of AGI is like thinking about Earth being overpopulated by space aliens. First we have to establish that such a thing is even possible, for which there is currently no concrete proof. Then we should start to think about how to deal with it.
Considering how hypothetical technology will impact mankind is literally the definition of science fiction. It makes for interesting reading, but it's far from a call to action.
Secondly - it's not necessarily about 'evil' AI. It's about AI indifferent to human life. Have a look at this article, it provides a better intuition for how slippery AI could be: https://medium.com/@LyleCantor/russell-bostrom-and-the-risk-...
This is a point everyone makes, but it hasn't been proven anywhere. Progress in AI as a field has always been a cycle of hype and cool-down.
Edit (reply to below). Talk about self-bootstrapping AIs, etc. is just speculation.
Just as one discovery enables many, human-level AI that can do its own AI research could superlinearly bootstrap its intelligence. AI safety addresses the risk of bootstrapped superintelligence indifferent to humans.
Of course, that assumes the return-on-investment curve for "bootstrapping its own intelligence" is linear or superlinear. If it's logarithmic or something other than "intelligence" (which is a word loaded with magical thinking if there ever was one!) is the limiting factor on reasoning, no go.
But even if we can do all that any time soon (which is a pretty huge if), we don't even know what the effect will be. It's possible that if we remove all of the "I don't want to study math, I want to play games" or "I'm feeling depressed now because I think Tim's mad at me" parts of the human intelligence, we'll end up removing the human ingenuity important to AGI research. It might be that the resulting AGI is much more horrible at researching AI than a random person you pull off the street.
Why does an AI need to be capable of moral reasoning to perform actions we'd consider evil?
The concern is that computers will continue to do what they're programmed to do, not what we want them to do. We will continue to be as bad at getting those two things to line up as we've always been, but that will become dangerous when the computer is smarter than its programmers and capable of creatively tackling the task of doing something other than what we wanted it to do. Any AI programmed to maximize a quantity is particularly dangerous, because that quantity does not contain a score for accurately following human morality (how would you ever program such a score).
If you're willing to believe that an AI will some day be smarter than an AI researcher (and assuming that's not possible applies a strange special-ness to humans), then an AI will be capable of writing AIs smarter than itself, and so forth up to whatever the limits of these things are. Even if that's not its programmed goal, you thought making something smarter than you would help with your actual goal, and it's smarter than you so it has to realize this too. And that's the bigger danger - at some unknown level of intelligence, AIs suddenly become vastly more intelligent than expected, but still proceed to do something other than what we wanted.
This is a matter of conjecture at this point: Andrew Ng predicts no; Elon Musk predicts yes.
I agree with you that, if you can be sure that superhuman AI is very unlikely or far off, then we have plenty of other things to worry about instead.
My opinion is, human-level intelligence evolved once already, with no designer to guide it (though that's a point of debate too... :-) ). By analogy: it took birds 3.5B years to fly, but the Wright brothers engineered another way. Seems likely in my opinion that we will engineer an alternate path to intelligence.
The question is when. Within a century? I think very likely. In a few decades? I think it's possible & worth trying to prevent the worst outcomes. I.e., it's "science probable" or at least "science possible", rather than clearly "science fiction" (my opinion).
So returning to your Wright brothers example, it's more like saying: "It took birds 3.5B years to fly, but the Wright brothers engineered another way. It seems likely that we'll soon be able to manufacture even more efficient wings small enough to wear on our clothes that will enable us to glide for hundreds of feet with only a running start."
Given that I could probably sketch out a half-assed design for one in nine months if you gave me a full-time salary - or rather, I could consult with a bunch of experts waaaaaay less amateurish than me and come up with a list of remaining open problems - what makes you say that physical computers cannot, in principle, no matter how slowly or energy-hungrily, do what brains do?
I'm not saying, "waaaaah, it's all going down next year!", but claiming it's impossible in principle when whole scientific fields are constantly making incremental progress towards understanding how to do it is... counter-empirical?
I mean, why can't I live forever? Let's just list the problems and solve them in the next year!
http://sens.org
Ok: what don't I know, that is interesting and relevant to this problem? Tell me.
>I mean, why can't I live forever?
Mostly because your cells weren't designed to heal oxidation damage, so eventually the damage accumulates until it interferes with homeostasis. There are a bunch of other reasons and mechanisms, but overall, it comes down to the fact that the micro-level factors in aging only take effect well after reproductive age, so evolution didn't give a fuck about fixing them.
>Let's just list the problems and solve them in the next year!
I said I'd have a plan with lists of open problems in nine months. I expect that even at the most wildly optimistic, it would take a period of years after that to actually solve the open problems and a further period of years to build and implement the software. And that's if you actually gave me time to get expert, and resources to hire the experts who know more than me, without which none of it is getting done.
As it is, I expect machine-learning systems to grow towards worthiness of the name "artificial intelligence" within the next 10-15 years (by analogy, the paper yesterday in Science is just the latest in a research program going back at least to 2003 or 2005). There's no point rushing it, either. Just because we can detail much of the broad shape of the right research-program ahead-of-time, especially as successful research programs have been conducted on which to build, doesn't mean it's time to run around like a chicken with its head cut-off.
Now for a less "killer" use case, you might get denied access to your credit card because of what Facebook "thinks" based on your feed and your friends feeds (this is a real product.)
AI doesn't have to be full blown human-like and generalizable to have real world implications.
This is what my piece called personas is about.. Most people don't understand the implications of what's already happening and how constrains of programming/ML lead to non-human like decisions with human-like consequences. http://personas.media.mit.edu
I would suggest you read the history of the Manhattan project if you want to continue in your belief system regarding "impossible" deadly technology.
To quote Carl Sagan:
>They laughed at Columbus, they laughed at Fulton, they laughed at the Wright brothers. But they also laughed at Bozo the Clown.
>We thus designed a brief questionnaire and distributed it to four groups of experts in 2012/2013. The median estimate of respondents was for a one in two chance that highlevel machine intelligence will be developed around 2040-2050, rising to a nine in ten chance by 2075. Experts expect that systems will move on to superintelligence in less than 30 years thereafter. They estimate the chance is about one in three that this development turns out to be ‘bad’ or ‘extremely bad’ for humanity.
Berkeley AI prof Stuart Russell's response goes something like: Let's say that in the same way Silicon Valley companies are pouring money in to advancing AI, the nations of the world were pouring money in to sending people to Mars. But the world's nations aren't spending any money on critical questions like what people are going to eat & breathe once they get there.
Or if you look at global warming, it would have been nice if people realized it was going to be a problem and started working on it much earlier than we did.
I'd be more impressed by a Human Intelligence Project - augmenting predictive power to encourage humans to stop doing stupid, self-destructive shit, and moving towards long-term glory and away from trivial individual short-term greed as a primary motivation.
AI is a non-issue compared to the bear pit of national and international politics and economics.
So the AI Panic looks like psychological projection to me. It's easier to mistrust the potential of machines than to accept that we're infinitely more capable of evil than any machine is today - and that's likely to stay true for decades, if not forever.
The corollary is that AI is far more likely to become a problem if it's driven by the same motivations as politics and economics. I see that as more of a worry than the possibility some unstoppable supermachine is going to "decide" it wants to use Earth as a paperclip factory, or that Siri is going to go rogue and rickroll everyone on the planet.
Job-destroying automation and algorithmic/economic herding of humans is the first wave of this. It's already been happening for centuries. But it could, clearly, get a lot worse if the future isn't designed intelligently.
But hey, I labor in this domain: if paranoid richy-rich types want to throw money at it to ensure that they remain at the top of the heap, I'm all for it.
Yes, but data also can be collected openly collectively, in the spirit of Wikipedia or OpenStreetMaps etc.
What I think OpenAI should encourage is the development of algorithms that can be used to crowdsource AI. I don't think there are good algorithms yet for model merging, but I would be gladly proven wrong.
There already exist drones that kill based on AI.
http://blog.samaltman.com/machine-intelligence-part-2
Sutskever is a researcher at Google, worked with Hinton in Toronto and Andrew Ng at Stanford.
Karpathy studied in Toronto and at Stanford, worked under Fei-Fei Li, worked at Google. He also has an awesome blog and seems very active and passionate about computer vision and ML.
Kingma also works with deep neural nets, worked under Yann LeCun (who works at Facebook)
Schulman is a PhD Candidate at Berkeley with publications at top conferences.
Zaremba is an PhD student at NYU, intern at Facebook. Impressive publication list and awards.
Abbeel is at Stanford's AI lab.
Bengio is one of the "stars" and celebrated figures of the deep net revival.
Levine is a researcher at Google working on deep nets with many serious papers.
---
Basically these are the main domain experts among them. The list is quite skewed to Google/Facebook, Stanford/Berkeley/Toronto and deep net researchers, working primarily on computer vision.
Uhhh.... https://www.google.com/search?q=Pieter+Abbeel That's a lot of results showing how he's been a professor at Berkeley since 2008.
He received his PhD at Stanford, then went to be a professor at Berkeley.
Quite surprised to see so many corporate AI people being in on this. I'd have thought that Google and Facebook would prefer to keep their research secret.