Summary of the paper for those who don't want to read it:
So basically there are two categories of "learning" involved in this sort of research, supervised and unsupervised. In supervised learning, someone gives the computer a long list of concepts and their attributes ("frog", "green frog", "jumping frog") and a set of pictures to go with each item, and feeds them into a visual-recognition algorithm. In unsupervised learning, the computer is given a concept like "frog" but then has to discover all the variations itself and get its own visual data to match.
The claim in this paper is that they have made the unsupervised learning as strong as the supervised learning. That is, they give the computer a concept ("frog"), it goes and searches through Google Books for common variations ("green frog", "jumping frog") and then uses Google image search to fetch images for each of those queries. They can then remove the obvious false positives (they test to see which images seem to screw up their learning algorithm and leave those out), and the result they get is on par with the supervised learning methods.
----------------------
In my opinion, this is only mildly interesting because Google Image Search functions based on human input anyway -- Google knows the difference between a "frog" and a "jumping frog" or even a "camel" simply because people on the internet caption such images and Google can make associations between images and their captions. Essentially, what the researchers have managed to do is outsource the work of some grad student to millions of people around the world through Google.
Of course, it could be argued that there is some sort of parallel with what humans actually do (we know what things are called because we hear other people call them that), but even if I didn't know the name of an animal I could still tell you when the same animal is in different pictures, and I can also tell you when it's jumping and what colour it is. I don't need to have someone caption the image for me to understand the broad range of situations to which the caption "jump" applies.
>I don't need to have someone caption the image for me to understand the broad range of situations to which the caption "jump" applies.
I wonder if this has anything to do with the fact that we can jump too. That we can translate the frog's position into something we do as well.
of course one can argue that we can do the same for non-anthro-moprhic things as well. What i think is that we dont directly relate pictures, as the software is taught. What we do is translate that 2D picture into something we'd see in the 3d world. And that 3d "vision" isn't just another image. It represents an object in our world. something that has shape, existence etc. something which we can observe from other senses as well. For us a picture doesn't always represent an abstract thing, an arbitary pattern of colours. It usually represents something concrete. Something about which we have tons of other pieces of knowledge as well.
So we relate pictures by checking if they map to the same real-world object. And here that "object" is a sort of nexus of many pieces of information we have on it which is a product of many direct and indirect human experiences.
So i don't really think that we are in a position to teach a computer to do anything like that.
> I don't need to have someone caption the image for me to understand the broad range of situations to which the caption "jump" applies.
I'm more interested in metaphor and analogy.
My 3.5 year old son said "look at the rain! It is bouncing like hopping frogs!"
I don't know if he created that. It's not in any of his books. I guess he jumps like a hopping frog at nursery and transferred that to rain.
I'm not so interested in a computer that is trained on frogs, and which sees a hopping frog and describes it as such. If it saw a hopping cat and said this thing is hopping but I don't know what it is, then I'd be interested.
The problem with all approaches to machine learning I see today is that they only focus on grouping and separating concepts based on certain characteristics. They seem to be all fundamentally statistical.
None of them seem to work towards a fundamental understanding of what the concepts mean. I'm not sure how that could be accomplished though.
Is there even a meaningful distinction to be made between beeing able to identify a concept and understanding what a concept actually means?
This is in fact a classic criticism of modern AI. Classic approaches tried to replicate human reasoning mechanisms. The work was very cool, but far harder to understand and scale, compared to modern statistical algorithms. To some extent, you are faced with a choice: a small palette of mediocre techniques whose workings are beautiful theories of the human mind; or, a broadly-applicable, incredibly practical set of tools with little motivation or spirit.
I understand that other being hard to scale in a machine efficiency sense, the old techniques were also rather limited in applicable domain. They were often based on unjustified models of whatever the author decided was a good model of thought/reasoning, whether "frames", predicate calculus, constraint propagation or whatever. It seems to me that although you say statistical methods have little motivation, the motivation of the alternative - heuristics - is very questionable. Can anyone correct me on this?
That's actually what I meant by scale—that they were slow isn't really a problem if you're interested in AI for the raw fun of it. While justifications for using logic or constraints or probability theory were often lacking from classical techniques, do understand that logic itself was originally developed as a crystallization of proper human thought. And many of the classic AI folks did care deeply about understanding how humans achieved certain results. For example, take a look at Marvin Minsky's work. To the contrary, modern techniques aren't interested in replicating human thought; they are simply interested in replicating human results.
Whatever your opinions on what "intelligence" really is and the role of "imagination" in producing it, I can assure you that many computer scientists would like to replicate human creativity, heuristics, and problem solving techniques. Many of the early AI pioneers were as interested in understanding human thought and understanding machine thought (source: conversations with some of them). Modern techniques simply have a different aim.
I don't think there is a meaningful distinction. Do chess computers have a "fundamental understanding" of chess, which humans traditionally considered a benchmark of human intelligence/strategy?
Analogous to the philosophical zombie thought experiment, I think that "real intelligence/understanding" is indistinguishable from simply being able to perform actions to accomplish the same tasks that humans traditionally consider to require intelligence.
Of course, that's one of the PR problems that AI has always had: once a computer can outperform a human at some task, that task is no longer considered to be something that requires "true intelligence." Most people would consider someone who can multiply numbers together to be intelligent, but when computers do that (incomprehensibly faster and more reliably), few people consider even for a moment that it's AI. Same with more advanced mathematics, like computer algebra and automated theorem proving. Same with facial and voice recognition. And I'm sure it will be the same with self-driving cars.
If fictional media is anything to go by, the single defining aspect of human intelligence is love. This is the last bastion of human understanding that is incomprehensible to evil, machines and aliens.
That's the place to start; the Behemoth trilogy isn't bad, per se, but it is a ramshackle thing by comparison, and I think only partly because it has a larger story to tell.
But love is indistinguishable from a set of actions that we traditionally associate with love, any of which a computer system (especially one connected to a realistic android) could conceivably perform.
Until I can just talk to a computer and it always understands what I'm asking, there's no AI is there? It's a fuzzy concept to exactly define, but it's pretty obvious what we all mean when we talk about it.
Brute forcing billions of moves of chess isn't going to achieve that, so it's nothing more than a cute trick that lay people don't understand is a trick so we have to tell them it is.
David Copperfield can "fly" but he can't just fly on demand, he can only fly in very special circumstances in a tightly controlled environment.
He's no more a flying man than deep blue is a thinking computer.
But that's the thing. We solve a problem and then the solution is just called a "cute trick". The goalpost gets moved further. Natural language processing is getting really advanced. Soon enough you will be able to talk to a computer in a conversation. And then you will just call that a cute trick.
I appreciate your opinion, because its food for thought, which is always great. Actually, it's really hard to refute your point after giving it some thought. But then again, I am pretty sure that there is a difference. I think it lies in the "universality" of our knowledge creation capabilities. In the end, we (humans) are the ones who fathered (mothered?) the algorithms driving the chess computer. I.e. we discovered an alternative way to reason about problems arising in a play of chess. It turns out, that the alternative solution we discovered, is actually better suited to make a successful move than the one humans use when reasoning about chess problems. This is because there is no "fundamental principle" of a successful chess move. It's all just predictions down different branches of possibilities. This is something a computer is naturally much better at than a human brain. Compare this to problems in different domains, those that cannot be modeled with our mathematical tool set: creativity, morality, consciousness, etc. Those a random concepts that may exist on different levels of emergence, but they are nonetheless real, or at least i'd say there is a consensus among most people that these phenomena have a bearing in reality.
The assumption that our brains are nothing but glorified (bio-) computers naturally leads to the conclusion that there is not a single thing humans can do, that cannot be done by any other equally capable computing platform, be it implemented in silicon-based or dna-based hardware. While I agree with this assumption, you shouldn't be so quick as to assume that we are already at the point where we can effectively re-implement the software which defines a human on any platform. There are a still "hard" problems of which we have no clue at all how to model them, even conceptually. The fact that we can do statistical analysis on an ordinary computer superior to a human doesn't mean much.
The fundamental problem humans will have with strong AI is that it won't be able to properly rationalize it's actions. Sure you'll have a statistical model for why it did what it did but that means about as much to anybody as a numerical reification of the quantum states of every particle of a person's mind at the time they made a decision.
Who says they won't be able to? In fact, rationalizing may be the easiest part - the rules of logic are few, and computers are very good at using them fast.
Building new concepts from raw inputs is the difficult part at which humans are better, but now computers are showing that they may be able to do that too.
When you say "rationalize its actions," are you referring to some mechanism in the inner workings of the brain/software? Or are you just talking about saying things that help humans understand why an action was chosen? I maintain that there is no distinction between the two, and it's quite conceivable that an AI could take an action then say something that helps humans understand why that action was chosen.
If an AI can speak it can tell humans why it did what it did. But it's impossible to look through it's brain and see for yourself. To see if it's telling to truth or exaggerating or rationalizing.
I don't think that anyone can answer that categorically. It's a question for philosophy or metaphysics. Almost merely a matter of opinion. It's hard to criticize software for not working towards a "fundamental understanding of what the concepts mean" if we admit that we don't really even agree on whether that is a valid sentence or what it might mean.
Machine Learning is more of an engineering discipline than a philosophical one. The aim (mostly, with exceptions) is to make machines that learn, not to research the nature of "understanding" or "concepts".
There are people who are looking at broader ideas of "what is intelligence", sometimes even the same people doing the more statistical research, but it's an open ended problem.
The main reason science/math/statistics doesn't spend time worrying about philosophy is because you can't really specify the problem cleanly - and having a clear idea of the problem is 90% of the way to finding the solution.
[edit]: If you want brain-inspired ML stuff, it's worth looking at what neuroscientists and cognitive scientists are doing. From what I've seen of some of their work they're developing ML algos as minimal tests for understanding how parts of the brain or consciousness works.
This reminds me of one of my favorite Dijkstra quotes. "The question of whether machines can think... is about as relevant as the question of whether submarines can swim"
Do you happen to natively speak Russian? I've heard that in Russian the word used for people swimming is the same word used for ships sailing.
In English, swimming specifically refers to the motions living organisms use to move in water. Ships instead are said to "sail", even though most do not use sails for power.
I'm of the impression that concepts only have fundamental meaning to us because we can relate them to sensory input. For example, initially a child is taught what a boat is by seeing pictures, it eventually learns that boats, cars and bicycles share common characteristics and can be grouped together as vehicles. These common characteristics are all understood and identified by seeing, hearing, touching or smelling.
It would be impossible completely describe a concept such as "horse" to a person with no knowledge of animal physiology, without recourse to pointing at a a horse or impersonating a horse noise. Resorting to analogies with other concepts - "a horse is like a cow" - doesn't work because the person has no knowledge of the other concepts. This person could be a young child or an untrained computer, and they can be taught fundamental meanings of concepts by feeding them information in the form of sights, sounds, smells etc.
I haven't thought about this very deeply but these are the thoughts that always come into my mind when this topic comes up. Concepts have no meaning without relating them to sensory input. Humans learn by connecting the two. Why can the same not be applied to computers?
There is no question that additional data would be potentially useful to an AI. But it's a common fallacy that it's the only way to get "meaning". Data from a digital camera isn't fundamentally different than data from text. The only benefit is that visual data is far less compressed, more redundant. It's easier to find patterns in it. Natural language is a highly compressed description of things in the real world.
That said, it doesn't at all mean you can't learn anything meaningful from it. For example, you build a machine learning system to predict the missing words in a sentence. "A horse is similar to other farm animals like ____." Machines are getting better at this kind of thing, though still far from human level. Google's word2vec for example can take a word like "horse" and list the words that it is most similar to. You can subtract the representation for "man" from "king", add "woman" and it outputs "queen".
(Disclaimer: I am not an native english speaker)
The title of the article and of the algorithm (learn everything about anything) is a bit misleading. You might believe that it learns everything, period. Actually it's more about finding every variation of a "concept", I quote their website:
"a fully automated method that given any concept, e.g., horse, discovers an exhaustive vocabulary for it that explains all variations (i.e., actions, interactions, attributes, etc) that modify its appearance."
I wonder. If they have an exhaustive vocabulary would it be possible to generate a picture of what the system believes an object to look like? I know that there is something called generative models in machine learning and my guess is that it could be applied here.
Well you'd have to build a probabilistic model for each concepts, whether on pixels or on features, and you could use it to generate images randomly. It might show up some good shapes.
It's possible, but generally generative models have to be trained in a specific way. If not, you could do something like for every layer of the neural net, you train another NN which can "predict" the layer below it, it's input. Then you can work your way down each layer to try to find an input which would produce that output.
Another way is to use some kind of optimization to find an input which produces that pattern (e.g. backprop to the pixels themselves.) This will give you the image that most strongly triggers that output. Not necessarily a typical example.
This strongly reminds me of the first chapter of Greg Egan's Diaspora [1] (which I can't recommend enough) in which newly-formed AIs bootstrap their way to consciousness in part by connecting randomly to an online library and using the various data streams to build up an associative model of the world.
Thirded. Before Diaspora, I first read "Wang's Carpets"[1] which is a short story of his. Then found out this story had later been incorporated as a chapter into the book. I remember basically immediately ordering said book that night.
fwiw, that "Webly-Supervised Visual Concept Learning" reminds me of the stuff that Hinton et al. do re: unsupervised (concept, etc.) learning (using restricted Boltzmann machines, and so on.) Good talk on the subject (of deep learning, etc.): https://www.youtube.com/watch?v=AyzOUbkUf3M
Umm.. fourthed? I just couldn't help but jump in and
also recommend Greg Egan's "Permutation City". That book is just wonderful... think simulation, cellular automata as a model for computation, artificial life and all that other good stuff :).
Also, about the LEVAN thing... given the amount of data available online, both in various structured formats and unstructured formats, don't be surprised if deep learning will yield better and better results moving forward. To me though, they mostly seem evolutionary rather than revolutionary. I mean if you look back at the AI field, during the days before the "AI winter" came, huge amounts of data is one thing researchers back then didn't have available. This is not to say that there haven't been advances in learning algorithms at all recently. ..
As well as adding my own strong recommedatios for Egan's "Permutation City" and "Diaspora" I would also recommend "Quarantine" - which has a rather splendid idea for mobile apps - "neural mods" that actually augment the brains own congnitive capabilities (including augmenting sensory data for the ultimate in VR).
And there is what one group chooses to do with a very special neural mod...
This is happening now in deep learning. Deep autoencoders[1] are allowing for computer representations of "similar" concepts. I recently gave a talk on this very concept to assist in QA systems.
>
This is an implementation of the "Learning Everything about Anything" system. The system
is implemented in MATLAB, with various helper functions written in Shell, Python, MEX C++
for efficiency reasons. For details about the method, please see [1].
This readme contains instructions on using the code, as well as accessing/using already
trained models for various concepts.
The software has been tested on Linux using MATLAB versions R2011a. There may be compatibility
issues with older versions of MATLAB. At least 4GB of memory (plus an additional 0.75GB for each
parallel matlab worker) is assumed.
Judging by the amount of "deep learning" submissions to HN bad times for AI are already here although maybe they never left.
In defense of the LEVAN thing, though, I didn't see any claims that this is science at all, more like an exploratory application illustrating an algorithm.
I think the increase in deep-learning links in this particular community is partly because Google recently dropped a huge pile of money on a deep-learning startup (Deep Mind), which gave it more visibility in the startup scene. Though in that case, are we talking about a separate AI frenzy, or is it just an appendage of the current startup/acquisitions frenzy?
Disclaimer: I have vested interest in Deep Learning having built a distributed deep learning framework[1] and building a business around it.
Deep Learning is actually worth the hype though. It has 2 main merits that are interesting.
1. Auto Trend Discovery
2. Plays very well with parallelism
The main problem, which I'm hoping to fix, is feasibility and ease of use. Neural nets to the untrained eye can be a black box that takes a really long time to train with little to no reward.
The hype isn't all for naught either. I'll elaborate if asked, but won't bore you guys otherwise.
The results coming out from different tasks are currently blowing away many of the old school algorithms in tasks like sentiment analysis, speech to text, object recognition, among others.
Yes. I'll have a cleaner website with that going up next week. I just got done running a 20 node cluster on amazon training a few different datasets, it hummed ;).
The overall idea is a support, services, training model.
Support: Install/Infrastructure
Services: Help with data etl/onboarding, tuning
Training: How to use deep learning, how to think about it, and how to run/setup clusters. This will be run through the machine learning academy I teach at[1]
Sure! You're absolutely right. There are just a lot of myths about deep learning that I like clearing up, that being: it's a real world algorithm with actual merits, not just some marketing hype.
Marketing hype and machine learning really does make things convoluted. That being said, what DOESN'T the press exaggerate?
The track record for AI has definitely been over promise and under deliver. I think the hardware is getting there where we can start making real progress though.
In this case, google is putting its money where its mouth is.
Thanks for the wishes, it's been a great ride so far. My overall goal with this is to bring deep learning to everyone else. I see real merits in auto discovery of trends to automate some of the worst parts of machine learning and would like others to see these benefits as well. Seeing cool apps built with it is another side goal of this as well.
> The track record for AI has definitely been over promise and under deliver.
I thought that this was an early, encouraging sign that the models being used were fundamentally valid. It's entirely consistent with the track record for natural intelligence.
Still needs a bit of work in terms of linearity, but running it on a 20 node cluster was easy. Not going to benchmark it more than that right now, but it's really starting to come to life.
Seem to be getting some disagreeing with me. Could you elaborate as to why? I'm not pretending to be an all knowing person who can predict the future here. I'd especially love to hear from others in the field.
I think it's worth clarifying a few points here - this is a fairly naive view of deep learning being put forward.
In particular -
1) what does 'automatic trend discovery' mean? We've been able to do change-point detection, linear regression, etc. for hundreds of years. If you're talking about automatically learning a feature representation, then there are other algorithms that can do this, in a much simpler way. If you're arguing that it produces better representations, then make that argument.
2) This is almost _completely_ false, and indicates a substantial lack of experience of ML beyond deep learning. Other machine learning algorithms (SVMs, LR, even some decision tree algorithms) are _much_ easier to train in parallel - this is (partially) because your objective function has certain nice properties that allow you to combine partial solutions together that are produced in parallel (convexity, separability). When you're using gradient-based methods on an incredibly ugly non-convex function from a multi-layer neural network, you're in a completely different world.
Granted, there have been techniques coming out for training in multiple address spaces, but these are _hacks_ to get around the ugly structure of the problem, not the principled approaches that exist for other algorithms.
I don't have any perspective on your deeplearning4j library, but I'm skeptical of its utility given the existence of existing well-tested deep learning libraries written/contributed to by renowned experts in the field (e.g. cuda-convnet, caffe, torch). This stuff is a) very tricky to get right, b) very tricky to debug, and c) very performance sensitive. Just a quick pass through shows zero references to CUDA/GPGPU programming, so I'd suspect performance is going to be significantly worse than the aforementioned libraries.
1. In this case, we are talking about the pretrain part of neural nets. Automatic trend discovery comes down to doing feature extraction for the user. That being said, if I was inexperienced I wouldn't be teaching this stuff[1]. Am I the best machine learning practitioner out there? No. A lot of us aren't. I am all about making other people's jobs practical though.
Yes, I am talking about learning better representations. See hinton's deep autoencoder work as a prime example of this comparing PCA to RBM based methods for topic detection[2].
2. Google and people way smarter than I am seem to be doing just fine with this[3]. That being said, I didn't say that random forest (with whole companies built on this parallelism[3]) or any of the algorithms WEREN'T friendly.
I would say one of the main appeals for deep learning is the scale of data with which it can benefit from.
Feel free to be skeptical all you want, if the researchers want to take the time to write a full stack distributed framework, I welcome others in to the game. The problem with the packages out there right now, (being matlab, python) are training times, and integrating in to an actual ecosystem. I'm addressing this this year at 2 different talks[5][6].
Replying to your last point, I use blas underneath for all of the matrix calculations, I will be adding GPUs later this year, and yes you're right,this stuff is hard to make. I also wouldn't be publicizing it if I wasn't already using it in production applications. Frankly right now though, I use cpu matrices right now, because I can fire this up on AWS (without the limit of GPU RAM), and it's practical for hadoop deployments. Honestly whether we like it or not, GPUs take a lot to get right. NVIDIA[7] and AMD[8] are going to make my job pretty easy though.
To end, if I was afraid of every little obstacle, why do anything in the first place? While you're hiding behind a throw away account, I'm actually trying to put this in the hands of people who don't have the time to learn every little thing about neural networks. At the end of the day, I follow the papers very closely and enjoy what I do. I also work on all sorts of different techniques for different problems combining different machine learning algorithms for different tasks (just like anyone else would). This framework is my way of getting this out to everyone else. If you have a deep learning framework, I'd love to see it, maybe I could learn a thing or 2.
Please post that as a Show HN when you feel you're ready. (Email hn@ycombinator.com if you want clarification of what that means.)
Edit: In case that seems foreboding at all, what I mean is: this looks really cool and will probably make an awesome Show HN, whenever you think it's ready for a post of its own.
I think what he is trying to say is that if you read about it, it would influence you. And you may end up building something similar (possibly remotely) to it. And then you can be sued for it.
I am not saying that i agree with him, i am just trying to clarify his point
FWIW, the paper is much more modest and honest than the gigaom article. It's certainly a hot topic and getting wide coverage, but most scientists are being fairly conservative about their claims.*
* Grant applications and future work sections excluded :)
Nah, I think you're overreacting. It seems like every pop-sci article on CS research reads like this one (e.g. "breakthrough in cryptography", "scientists solve quantum computing", etc.).
Usually it's just weak journalism rather than researchers trying to overinflate.
It's easy to demonstrate that this is a huge advance in AI. All these guys need to do is start entering Kaggle competitions and win, and win, and win, and win...
That said, given that Google et al. are scraping every corner of the bottom of the barrel in search of advances in search, I think the good times for AI peeps will continue for some time...
This doesn't really help the symbol grounding problem: it uses pre-human-sorted data (who use their own ability to match symbols and meaning) to form its associative network. So, it's using human consciousness as part of the input to form its own consciousness. You could argue that humans use other humans' consciousness to develop its own, but now you have the infinite regress which seem to be the fate of all symbol grounding contemplations.
Surely there has to be a starting point, some "axioms" you initially accept about the world to start the process. Maybe these are embedded in our DNA and have evolved to be a practical start (eg: sharp sensory input from nerves on your skin is automatically linked with pain, which we automatically avoid).
If we define 'learning' in an appropriate way, then
any good research library already knows "everything
about anything".
Okay, let the thing 'learn' about the Kuhn-Tucker conditions by searching on Google and reading, say, Wikipedia or some books at Google or Amazon. Then have the thing show that for problems in functional form the
Zangwill and Kuhn-Tucker constraint qualifications are independent. Do that and I will start to believe that the
terminology 'deep learning' is appropriate. I'm not holding
my breath.
Yes, it may be that in some rough sense the kind of 'learning' it is doing is roughly like some of the learning of a child of, say, 2 as it is starting to learn about language and things. Yes, it may be that such 'learning' is a significant part of the intelligence of, say, a child of 3-5. Maybe. Big, huge maybe.
When I was working in AI, I noticed the terminology had
been cooked up to imply much more than was being
accomplished. Now, as I understand it, there is a
specific definition for the current AI term 'deep
learning' and has to do with the 'depth' of where
adjust parameters in a neural network, not how 'deep'
the 'learning' is about the subject in question.
Cute terminology.
80 comments
[ 4.8 ms ] story [ 163 ms ] threadhttp://levan.cs.washington.edu/ngrams/objectNgrams_cvpr14.pd...
So basically there are two categories of "learning" involved in this sort of research, supervised and unsupervised. In supervised learning, someone gives the computer a long list of concepts and their attributes ("frog", "green frog", "jumping frog") and a set of pictures to go with each item, and feeds them into a visual-recognition algorithm. In unsupervised learning, the computer is given a concept like "frog" but then has to discover all the variations itself and get its own visual data to match.
The claim in this paper is that they have made the unsupervised learning as strong as the supervised learning. That is, they give the computer a concept ("frog"), it goes and searches through Google Books for common variations ("green frog", "jumping frog") and then uses Google image search to fetch images for each of those queries. They can then remove the obvious false positives (they test to see which images seem to screw up their learning algorithm and leave those out), and the result they get is on par with the supervised learning methods.
----------------------
In my opinion, this is only mildly interesting because Google Image Search functions based on human input anyway -- Google knows the difference between a "frog" and a "jumping frog" or even a "camel" simply because people on the internet caption such images and Google can make associations between images and their captions. Essentially, what the researchers have managed to do is outsource the work of some grad student to millions of people around the world through Google.
Of course, it could be argued that there is some sort of parallel with what humans actually do (we know what things are called because we hear other people call them that), but even if I didn't know the name of an animal I could still tell you when the same animal is in different pictures, and I can also tell you when it's jumping and what colour it is. I don't need to have someone caption the image for me to understand the broad range of situations to which the caption "jump" applies.
I wonder if this has anything to do with the fact that we can jump too. That we can translate the frog's position into something we do as well.
of course one can argue that we can do the same for non-anthro-moprhic things as well. What i think is that we dont directly relate pictures, as the software is taught. What we do is translate that 2D picture into something we'd see in the 3d world. And that 3d "vision" isn't just another image. It represents an object in our world. something that has shape, existence etc. something which we can observe from other senses as well. For us a picture doesn't always represent an abstract thing, an arbitary pattern of colours. It usually represents something concrete. Something about which we have tons of other pieces of knowledge as well.
So we relate pictures by checking if they map to the same real-world object. And here that "object" is a sort of nexus of many pieces of information we have on it which is a product of many direct and indirect human experiences.
So i don't really think that we are in a position to teach a computer to do anything like that.
I'm more interested in metaphor and analogy.
My 3.5 year old son said "look at the rain! It is bouncing like hopping frogs!"
I don't know if he created that. It's not in any of his books. I guess he jumps like a hopping frog at nursery and transferred that to rain.
I'm not so interested in a computer that is trained on frogs, and which sees a hopping frog and describes it as such. If it saw a hopping cat and said this thing is hopping but I don't know what it is, then I'd be interested.
Am I being too harsh on the robots?
None of them seem to work towards a fundamental understanding of what the concepts mean. I'm not sure how that could be accomplished though.
Is there even a meaningful distinction to be made between beeing able to identify a concept and understanding what a concept actually means?
IMO "Imagination" itself isn't intelligence but an input to the last "intelligent" activity.
Computers can be taught to exhibit different "intelligent" activities and that is AI & Machine learning is all about.
Human imagination works without any knowledge skill or learning (it is just my understanding, and i might be wrong).
Producing rough and wild-west imagination isn't problem of AI.
Analogous to the philosophical zombie thought experiment, I think that "real intelligence/understanding" is indistinguishable from simply being able to perform actions to accomplish the same tasks that humans traditionally consider to require intelligence.
Of course, that's one of the PR problems that AI has always had: once a computer can outperform a human at some task, that task is no longer considered to be something that requires "true intelligence." Most people would consider someone who can multiply numbers together to be intelligent, but when computers do that (incomprehensibly faster and more reliably), few people consider even for a moment that it's AI. Same with more advanced mathematics, like computer algebra and automated theorem proving. Same with facial and voice recognition. And I'm sure it will be the same with self-driving cars.
Until I can just talk to a computer and it always understands what I'm asking, there's no AI is there? It's a fuzzy concept to exactly define, but it's pretty obvious what we all mean when we talk about it.
Brute forcing billions of moves of chess isn't going to achieve that, so it's nothing more than a cute trick that lay people don't understand is a trick so we have to tell them it is.
David Copperfield can "fly" but he can't just fly on demand, he can only fly in very special circumstances in a tightly controlled environment.
He's no more a flying man than deep blue is a thinking computer.
The assumption that our brains are nothing but glorified (bio-) computers naturally leads to the conclusion that there is not a single thing humans can do, that cannot be done by any other equally capable computing platform, be it implemented in silicon-based or dna-based hardware. While I agree with this assumption, you shouldn't be so quick as to assume that we are already at the point where we can effectively re-implement the software which defines a human on any platform. There are a still "hard" problems of which we have no clue at all how to model them, even conceptually. The fact that we can do statistical analysis on an ordinary computer superior to a human doesn't mean much.
Building new concepts from raw inputs is the difficult part at which humans are better, but now computers are showing that they may be able to do that too.
There are people who are looking at broader ideas of "what is intelligence", sometimes even the same people doing the more statistical research, but it's an open ended problem.
The main reason science/math/statistics doesn't spend time worrying about philosophy is because you can't really specify the problem cleanly - and having a clear idea of the problem is 90% of the way to finding the solution.
[edit]: If you want brain-inspired ML stuff, it's worth looking at what neuroscientists and cognitive scientists are doing. From what I've seen of some of their work they're developing ML algos as minimal tests for understanding how parts of the brain or consciousness works.
In English, swimming specifically refers to the motions living organisms use to move in water. Ships instead are said to "sail", even though most do not use sails for power.
It would be impossible completely describe a concept such as "horse" to a person with no knowledge of animal physiology, without recourse to pointing at a a horse or impersonating a horse noise. Resorting to analogies with other concepts - "a horse is like a cow" - doesn't work because the person has no knowledge of the other concepts. This person could be a young child or an untrained computer, and they can be taught fundamental meanings of concepts by feeding them information in the form of sights, sounds, smells etc.
I haven't thought about this very deeply but these are the thoughts that always come into my mind when this topic comes up. Concepts have no meaning without relating them to sensory input. Humans learn by connecting the two. Why can the same not be applied to computers?
That said, it doesn't at all mean you can't learn anything meaningful from it. For example, you build a machine learning system to predict the missing words in a sentence. "A horse is similar to other farm animals like ____." Machines are getting better at this kind of thing, though still far from human level. Google's word2vec for example can take a word like "horse" and list the words that it is most similar to. You can subtract the representation for "man" from "king", add "woman" and it outputs "queen".
How would you define “understanding” and “actually means” in your question?
See also: http://en.wikipedia.org/wiki/Chinese_room
Another way is to use some kind of optimization to find an input which produces that pattern (e.g. backprop to the pixels themselves.) This will give you the image that most strongly triggers that output. Not necessarily a typical example.
[1] http://gregegan.customer.netspace.net.au/DIASPORA/01/Orphano...
fwiw, that "Webly-Supervised Visual Concept Learning" reminds me of the stuff that Hinton et al. do re: unsupervised (concept, etc.) learning (using restricted Boltzmann machines, and so on.) Good talk on the subject (of deep learning, etc.): https://www.youtube.com/watch?v=AyzOUbkUf3M
[1]: read online here: http://bookre.org/reader?file=222997
Also, about the LEVAN thing... given the amount of data available online, both in various structured formats and unstructured formats, don't be surprised if deep learning will yield better and better results moving forward. To me though, they mostly seem evolutionary rather than revolutionary. I mean if you look back at the AI field, during the days before the "AI winter" came, huge amounts of data is one thing researchers back then didn't have available. This is not to say that there haven't been advances in learning algorithms at all recently. ..
And there is what one group chooses to do with a very special neural mod...
[1] http://www.slideshare.net/agibsonccc/ir-34811120
I quote the readme:
> This is an implementation of the "Learning Everything about Anything" system. The system is implemented in MATLAB, with various helper functions written in Shell, Python, MEX C++ for efficiency reasons. For details about the method, please see [1].
This readme contains instructions on using the code, as well as accessing/using already trained models for various concepts.
For questions concerning the code please contact Santosh Divvala (http://homes.cs.washington.edu/~santosh) at santosh@cs.washington.edu.
The software has been tested on Linux using MATLAB versions R2011a. There may be compatibility issues with older versions of MATLAB. At least 4GB of memory (plus an additional 0.75GB for each parallel matlab worker) is assumed.
In defense of the LEVAN thing, though, I didn't see any claims that this is science at all, more like an exploratory application illustrating an algorithm.
Deep Learning is actually worth the hype though. It has 2 main merits that are interesting.
1. Auto Trend Discovery
2. Plays very well with parallelism
The main problem, which I'm hoping to fix, is feasibility and ease of use. Neural nets to the untrained eye can be a black box that takes a really long time to train with little to no reward.
The hype isn't all for naught either. I'll elaborate if asked, but won't bore you guys otherwise.
The results coming out from different tasks are currently blowing away many of the old school algorithms in tasks like sentiment analysis, speech to text, object recognition, among others.
[1] http://deeplearning4j.org/
The overall idea is a support, services, training model.
Support: Install/Infrastructure
Services: Help with data etl/onboarding, tuning
Training: How to use deep learning, how to think about it, and how to run/setup clusters. This will be run through the machine learning academy I teach at[1]
[1]: http://zipfianacademy.com/
My point is that mixing business goals and scientific truth is dangerous if not handled carefully. That said all the best with both your goals.
Marketing hype and machine learning really does make things convoluted. That being said, what DOESN'T the press exaggerate?
The track record for AI has definitely been over promise and under deliver. I think the hardware is getting there where we can start making real progress though.
In this case, google is putting its money where its mouth is.
Thanks for the wishes, it's been a great ride so far. My overall goal with this is to bring deep learning to everyone else. I see real merits in auto discovery of trends to automate some of the worst parts of machine learning and would like others to see these benefits as well. Seeing cool apps built with it is another side goal of this as well.
I thought that this was an early, encouraging sign that the models being used were fundamentally valid. It's entirely consistent with the track record for natural intelligence.
Even the emergence of neural nets in the past few years has been due to hardware increases.
Still needs a bit of work in terms of linearity, but running it on a 20 node cluster was easy. Not going to benchmark it more than that right now, but it's really starting to come to life.
In particular -
1) what does 'automatic trend discovery' mean? We've been able to do change-point detection, linear regression, etc. for hundreds of years. If you're talking about automatically learning a feature representation, then there are other algorithms that can do this, in a much simpler way. If you're arguing that it produces better representations, then make that argument.
2) This is almost _completely_ false, and indicates a substantial lack of experience of ML beyond deep learning. Other machine learning algorithms (SVMs, LR, even some decision tree algorithms) are _much_ easier to train in parallel - this is (partially) because your objective function has certain nice properties that allow you to combine partial solutions together that are produced in parallel (convexity, separability). When you're using gradient-based methods on an incredibly ugly non-convex function from a multi-layer neural network, you're in a completely different world.
Granted, there have been techniques coming out for training in multiple address spaces, but these are _hacks_ to get around the ugly structure of the problem, not the principled approaches that exist for other algorithms.
I don't have any perspective on your deeplearning4j library, but I'm skeptical of its utility given the existence of existing well-tested deep learning libraries written/contributed to by renowned experts in the field (e.g. cuda-convnet, caffe, torch). This stuff is a) very tricky to get right, b) very tricky to debug, and c) very performance sensitive. Just a quick pass through shows zero references to CUDA/GPGPU programming, so I'd suspect performance is going to be significantly worse than the aforementioned libraries.
Yes, I am talking about learning better representations. See hinton's deep autoencoder work as a prime example of this comparing PCA to RBM based methods for topic detection[2].
2. Google and people way smarter than I am seem to be doing just fine with this[3]. That being said, I didn't say that random forest (with whole companies built on this parallelism[3]) or any of the algorithms WEREN'T friendly. I would say one of the main appeals for deep learning is the scale of data with which it can benefit from.
Feel free to be skeptical all you want, if the researchers want to take the time to write a full stack distributed framework, I welcome others in to the game. The problem with the packages out there right now, (being matlab, python) are training times, and integrating in to an actual ecosystem. I'm addressing this this year at 2 different talks[5][6].
Replying to your last point, I use blas underneath for all of the matrix calculations, I will be adding GPUs later this year, and yes you're right,this stuff is hard to make. I also wouldn't be publicizing it if I wasn't already using it in production applications. Frankly right now though, I use cpu matrices right now, because I can fire this up on AWS (without the limit of GPU RAM), and it's practical for hadoop deployments. Honestly whether we like it or not, GPUs take a lot to get right. NVIDIA[7] and AMD[8] are going to make my job pretty easy though.
To end, if I was afraid of every little obstacle, why do anything in the first place? While you're hiding behind a throw away account, I'm actually trying to put this in the hands of people who don't have the time to learn every little thing about neural networks. At the end of the day, I follow the papers very closely and enjoy what I do. I also work on all sorts of different techniques for different problems combining different machine learning algorithms for different tasks (just like anyone else would). This framework is my way of getting this out to everyone else. If you have a deep learning framework, I'd love to see it, maybe I could learn a thing or 2.
[1]: http://zipfianacademy.com/
[2]: http://www.cs.toronto.edu/~fritz/absps/esann-deep-final.pdf
[3]: https://bigml.com/
[4]: http://static.googleusercontent.com/media/research.google.co...
[5]: http://hadoopsummit.org/san-jose/schedule/
[6]: http://www.oscon.com/oscon2014/public/schedule/detail/33709
[7]: http://www.jcuda.org/jcuda/jcublas/JCublas.html [8]: http://developer.amd.com/tools-and-sdks/heterogeneous-comput...
Edit: In case that seems foreboding at all, what I mean is: this looks really cool and will probably make an awesome Show HN, whenever you think it's ready for a post of its own.
I am not saying that i agree with him, i am just trying to clarify his point
* Grant applications and future work sections excluded :)
Usually it's just weak journalism rather than researchers trying to overinflate.
That said, given that Google et al. are scraping every corner of the bottom of the barrel in search of advances in search, I think the good times for AI peeps will continue for some time...
1. http://jveness.info/software/default.html
Okay, let the thing 'learn' about the Kuhn-Tucker conditions by searching on Google and reading, say, Wikipedia or some books at Google or Amazon. Then have the thing show that for problems in functional form the Zangwill and Kuhn-Tucker constraint qualifications are independent. Do that and I will start to believe that the terminology 'deep learning' is appropriate. I'm not holding my breath.
Yes, it may be that in some rough sense the kind of 'learning' it is doing is roughly like some of the learning of a child of, say, 2 as it is starting to learn about language and things. Yes, it may be that such 'learning' is a significant part of the intelligence of, say, a child of 3-5. Maybe. Big, huge maybe.
When I was working in AI, I noticed the terminology had been cooked up to imply much more than was being accomplished. Now, as I understand it, there is a specific definition for the current AI term 'deep learning' and has to do with the 'depth' of where adjust parameters in a neural network, not how 'deep' the 'learning' is about the subject in question. Cute terminology.