The network guessed some of the "correct" answers far to quickly. For example, with just an L shape (|_) it guessed suitcase. Feels like the model is suffering from overfitting.
For me a bit less extreme, but it guessed 'police car' when all i drew was a poor car doodle. there was nothing about that car the would make it a police car.
I wonder if it has a very small pool of potential drawings. Eg if it only has "police car" and no other cars it might be able to jump straight to police car after seeing a car.
It also seems to constantly have a "best guess" to some degree and if that happens to be correct it confirms pretty quickly.
I was asked to draw a "phone". Drew a telephone (the old ones with a rotary dialler, because that was quickest to draw), and it guessed "telephone", but it was incorrect :-(
EDIT: Drew a "cake", it guessed "birthday cake". Wrong answer apparently.
It's possible, but I'm not entirely convinced that this is honest, or at least not really accurate. It's far too eager to try to match it to something than it is to figure out what it has.
It could figure out most of my drawings but it would get them well in advance of me completing anything substantial (like others, I would be asked to draw a leg and draw just a curved line and it would guess leg before I finished).
Trying to draw what it asked for but with some unusual features (like lines or dot patterns around what it asked for before drawing it) and it gets extremely confused; it doesn't really seem to be good at filtering out any noise: http://imgur.com/a/oE1j2 (gallery of results and what it thought it saw.
Drawing things it didn't ask for just to see what it was guessing resulted in some really strange responses and fits. The answer set it has is extremely limited, so something like a hand giving the horns (\m/) was last guess a duck. A moose was a scorpion, then a duck, then a hand. Godzilla (or a bipedal dinosaur if you prefer) was a vase, then a scorpion, then a boat. My loaf of bread was a washing machine, an anvil, then a postcard. The Deathstar was a bandage, a helicopter, and a lighthouse. And a chainsaw was considered an aircraft.
Between the disruptive patterns and drawing things outside of it's vocabulary, the system seems really confused. Looking at the comparison results, I can see how when drawing some things it got it real fast. (Tennis Rackets were mostly defined by a crosshatch pattern, Harps by a series of parallel vertical lines). This makes sense. For other things, not as much.
It might be a more convincing presentation to give the user a list of items the machine knows (the full list) and tell the user to try to draw some, and then the computer could check it off as it gets them. That seems like a better way of presenting this than "Draw a box. hey! you drew a box! Isn't that cool?"
Huh, I had no clue what you meant until I looked at the picture then back at my scribbles and saw the accidental drawing of the face I made.
To me this is another interesting distinction on the NN recognition versus a human recognition - QuickDraw having a limited "vocabulary" to refer to really highlights this, as does my own lack of knowledge of Roobarb. Some of these things can really blindside us, and I suspect that it's going to require a lot of human hand-holding for awhile for the machines to get a strong vocabulary.
For some time I've been pondering how far you could take a machine's tabula rasa learning, for something like language, and how closely it would mimic a child's learning. (Language, color, math, etc).
Yeah, but it's cheating itself. If it's telling me what it recognizes before I'm done, it severely reduces the usefulness of my input for further training. Anything submitted will just reinforce existing patterns, and do comparatively little to improve recognition.
They just want you to provide annotated training samples. Their guess is visible to you, to gamify this for you and make it more interesting, it might also be used to force you to draw some distinctive features once it shows you some other category.
Yup, the minimalist Picassoesque renditions I was able to do on my laptop trackpad were still recognised as long as I gave a vague impression of the major shapes I figured other people would draw. The result was the kind of image where a human would frown at it until you said “kangaroo”.
It's because it gets the most plausible answer, even if it's only a small chance of being correct. If there are not that many words, there are only a few objects that could be draw with a L shape like that. I mean, even if you draw something else starting with a L shape, it will probably be a slight different L shape (with a curve, for example). I think that's it, doesn't really mean it's overfitting, it would mean that if the probability was really high.
The ai keep throwing guesses in 20 seconds until he got the right one.
So if you draw a simple shape it start to go trough the list of things he recognize and end the game there.
It works great for this game because he can have very fast answer but only work win the cases when you actually have a feedback that eliminate all the wrong guesses.
Basically if it simply went trought the whole english dictionary fast enough he could get the same result without even looking at the pictures.
Of course it just takes one juvenile prankster to say in a crowded room "You won't believe what it recognises phalluses, poo emoji, and bottoms as" and it's going to be Microsoft Tay all over again.
So nobody do that, right.
Seriously, don't even think about it.
(More seriously, I think I might have been de-trainging it by trying to draw the correct drawing using a touch-pad... rather too many of my phones seem to look like tornados by the time the dang thing's got stuck in draw mode trying to click-and-drag to draw)
I think GP has raised a fair point. The landing page says:
"Can a neural network learn to recognize doodles" but the game doesn't demonstrate that the ANN is able to recognize arbitrary objects.
It merely shows that it's able to pick the correct name from a finite list of possible names that could very well be small (e.g. 10 or 20 object names).
A better way to demonstrate that the ANN is truly able to recognize objects with good accuracy would be to ask a person to draw an unspecified object and guess its name.
If the goal is to acquire training data, they could still ask for a name in case of failure.
This game seems to essentially be win lose or draw the gameshow; the computer is throwing out some random variations like contestants guessing. It makes it easy for the computer to try 'car' the getting a no it tries 'police car'. I wonder how many things it understands...
We are so far from general AI. It's so strange how our brains can refine existance down so easily... how many CPUs is the human brain worth as far as we know?
it guessed phone when asked to draw phone but obviously wasn't strong enough a match because it moved on without getting it right before coming back to it twice. Finally stayed on blackberry after doing cellphone. Requires some weighting to get there?
Their 'target market' is whoever uses the site. The program has no preconceived notion of what a "toilet" is so if enough people draw a hole in the ground for a toilet then it will start to recognize that as a toilet.
Not really. Most kids walk or bike to school. In places where public transport is built out kids that need to take a bus to school just use that. In the places that offer dedicated buses to take kids to school the county just charters buses from the local bus company. The idea of a school bus as a visually distinct concept doesn't exist.
It would be an interesting exercise to collect these doodles as a dataset and publish them. It could give insight into what are the key identifying features we look for as humans in relation to the world around us.
People really can't draw hospitals either. Not one of them looks like a real hospital (because what does a hospital look like. You know one if you see one but it's the people not the buildings).
The drawings at best look like churches (a building with a cross on top), but most just have giant crosses on the side.
Yeah, I got the same problem. I drew an aircraft carrier similar to yours. Google guessed "cruise ship" and "submarine" because it was comparing my drawing to drawings of airplanes.
After I did add the hole, it looked nothing like a steak to me. The only thing I could think of was to put a pan around it. As soon as I started drawing the pan it said, "Oh I know, steak". Makes zero sense to me.
I was pretty shocked it didn't figure out my pencil. In the last few seconds (I was already done) I started coloring in the eraser but it still didn't get it.
----------------------
EDIT:
Just did another one. Shocked it didn't get this!!:
(third from bottom and second from bottom rows.) Obviuosly mine looks a LOT more like a traffic light than these other people's, because I contextualize it!
Researchers are trying to collect data for training. I wonder how many examples they receive and how many are bogus.
The microsoft chat experiment from a few months ago shows that people can hack these systems to train them into racist slur-spewers.
Similarly, i wonder how this project aims to get accuracy given that many people are going to draw the porno objects no matter what the prompt was to mess with it.
It didn't recognize my phone drawing, but it was interesting to see other's people drawings of phones, I drew what it called a cell phone. Some drawings were smartphones, other were corded phone with a rotary dial and everything in between.
I don't know how it sounds exactly for a native English speaker, but saying "adult neural networks" seems to give the intelligence trait to some piece of software.
I was going to link the book because I thought it was on the Public Domain, but apparently not in every country [1]. Also, the site you reference seems to be purposely misguiding visitors by omission about the book's licensing status [2].
> The narrator explains that, as a young boy, he once drew a picture of a boa constrictor digesting an elephant in its stomach; however, every adult who saw the picture would mistakenly interpret it as a drawing of a hat. Whenever the narrator would try to correct this confusion, he was ultimately advised to set aside drawing and take up a more practical or mature hobby. The narrator laments the crass materialism of contemporary society and the lack of creative understanding displayed by adults. As noted by the narrator, he could have had a great career as a painter, but this opportunity was crushed by the misunderstanding of the adults.
In case you don't know the cultural reference, the joke is that kids can easily see that the hat is obviously a snake that has eaten an elephant, while adults only see a hat.
It's told in the first chapter of the famous childrens book, "the little prince"
>saying "adult neural networks" seems to give the intelligence trait to some piece of software
Ignoring the cultural reference everyone here is talking about:
How do you get that perception? We apply the adjective adult to animals to whom we (rightfully or wrongfully) don't attribute intelligence, and even to other lifeforms which we usually categorize as non-sentient (e.g. adult trees).
I think it is perfectly legitimate to assign the adjective adult to a neural network that has either left the training phase, or that is only undergoing marginal changes in further training. This seems to be mostly in line with how the word adult is used in other contexts.
Not that I'm opposed to calling software intelligent, in fact I think it would be weird if we couldn't call something intelligent just because it's silicon-based instead of being based on organic neurons. I just find it odd that you associate "adult" with intelligence at all.
Well, you're right. But somehow, my mind had wired "adult" to this kind of definition (wikipedia) : "Biologically, an adult is a human being or other organism that has reached sexual maturity". And therefore, I associated that to "organism" (I didn't think about animals, but I'd venture to say that when compared to computers, many familiar animals like a dog are more intelligent, but that's a debate, I understand that).
So in this sense, I found it surprising to associate a word that I use for "living"/biological things to a digital thing; specially in the context of A.I.
Maybe the term adult is, for me at least, very loaded with lots of meanings that go far behind the simple notion of "maturity"
I will try it in French, I've read it in Spanish, English, Italian and Japanese so far (the latter barely) so I think French would be a great addition and I probably will understand it being a Spaniard and all. Thanks for the recommendation
It seems like you can draw garbage (e.g. faces/text/swastikas) on the picture in the moments after it correctly identifies the object and before it starts the next object, and it seems to "save" it (though I am not sure if it is persisting it in their training data).
Really fun. The algorithm generates new nouns based on my drawing on the fly and add to the original pool. It stops and say 'Oh I know, it is ___', when there exist a guess that is the same as the answer. Might seem like it is really clever, or, it could have like 1000 guesses for every stroke and one of it is the correct answer.
My bathtub looks too ridiculous for it to be able to guess it with confidence.
It says draw a leg, I draw a straight line and before i even finish drawing the line it guesses "leg". Not rigged. And the "toe" I drew could have easily been a finger or thumb but it seemed to be sure it was a toe.
Perhaps it was never trained to see fingers or thumbs. Perhaps it is trained to recognize a couple dozen things. If it never heard of a finger or a thumb, how could it have used it as a guess.
I got "finger" as a prompt at least twice, so that's one of the things. It didn't recognize the ones I drew though. It expects it to be flat and pointing up with the nail, not bent.
This is the best "visualization" / "explanation" of the possibilities and limits of AI that I've seen.
I can show this to someone and say:
1. The software can recognize a feather, as long as it looks similar to what it thinks a feather looks like.
2. The software can't recognize a feather if it's never seen a feather like that. It's not a sentient being.
This is good, because most examples focus on point #1 and -- if enough marketing is involved -- don't go enough into point #2.
People read news articles like "X can recognize cats in a picture with Y certainty!" and are quick to assume that this "AI" can make sense of a picture and understand it, when all it does is apply certain methods for a certain use case.
This does a much better job by letting people write (or draw) their own test cases and figure out the limits intuitively.
Humans usually can't do your 2. either. In some cases, people may be able to recognize things based on descriptions alone, but those are typically simple combinations of known entities.
For recognizing relatively simple entities, are there advantages humans still have over neural nets (assuming the same scope of knowledge)?
Humans are great at learning abstraction from concrete examples. That's also what deep learning does and the big reason for its success as well. I'd guess that some neural nets architecture can do the same with your cat example (perhaps with adaptation). Can any expert weigh in?
An idea: We can also run several cat photos through image processing algorithms to filter out details. The output would be outlines similar to the drawings in the Google Quickdraw app. We put those through the app to generalize (perhaps the app needs some training with a few categories of objects, not necessarily animals). Voila! Software can now recognize drawings based on photo examples.
Of course, there's severe bias here, in the sense that what we consider abstraction is by definition "human shaped" abstraction
If multiple humans try to "abstract" a cat, the overlap in underlying processes will be pretty big, making it more likely that we can recognise each other's abstractions.
Of course, there's severe bias here, in the sense that what we consider abstraction is by definition "human shaped" abstraction
I can read the words here, but I don't understand the meaning.
We abstract to find a common set of features in things that are supposed to be the same but that are not present in things that are not supposed to be the same. Grouping these features then produces higher level abstractions, and so on.
Where would the bias be?
Even if the features differ, the process is the same.
And even the features are often the same. If you reverse a DCNN to see what it uses to classify things as "cats", expect to see whiskers and fur.
You implicitly (and I think without realising) presume objectivity + complete knowledge in the observer.
Human perception is heavily biased towards features that had evolutionary advantages, and limited by whatever technical flaws our eyes/brains/etc have. That's a selection bias in our perception of information, in our processing of said information, and therefore in the abstractions that result from it.
I agree with what you say, but it doesn't support your earlier statements.
I presume it's possible that the limitations of our visual system means we may miss powerful features and hence the ability to build some more powerful abstractions. (I didn't even argue this, just pointed out the process is the same even if features differ)
But I don't see how this supports your original claim of bias, which was: "If multiple humans try to "abstract" a cat, the overlap in underlying processes will be pretty big, making it more likely that we can recognize each other's abstractions."
If humans are good at recognizing each others' abstraction, that's a validation that low-pass (for lack of a better term) filtering the features due to human's physical design still creates very good abstractions and classifiers. That is to say, if anything you're confirming that humans are designed in a way that makes the abstractions they can make maximally useful.
That's exactly what I and others have been arguing. Now to be clear: it's not that these classifications are wrong, just that out of all possible classifications we could have found, we will most likely find the ones that fit the human perspective of the world.
Think of the Turing test and its criticisms; it's kind of has the same issues.
PS: I've upvoted every comment of yours; asking questions like this should be encouraged :)
> still creates very good abstractions and classifiers.
My point is that "good" and "bad" are not objective here, but depend on human use-cases.
Now to be clear: I'm not disagreeing with you! These are good abstractions, for humans. It lets us communicate concepts easily, which is great! But it might not be the best abstraction in every circumstance.
For example, I recall reading an article that said that AI is better at spotting breast cancer from photos (which is essentially interpreting abstract blobs as cancer or not). The main reason seems to be that it is not held back by the human biases in perception.
Think of Bugs Bunny. He looks nothing like a real rabbit, yet humans recognise him as a rabbit (presumably) because we look at the characteristics that separate him from a normal human, then compare those characteristics with our list of things with those characteristics (long ears, big feat, eats carrots) and get a rabbit.
If he'd been made to look like a rabbit-octopus hybrid instead of rabbit-human, we may have struggled more.
Computers don't look at things from a human perspective; they're still good at abstraction, just different to human abstraction. i.e. there's a human bias in there.
That's OK though; the objective is to make a computer that sees things the way people do; so it's a bias we want.
However the issue isn't that the computer's not a sentient being and therefore can't abstract things it's never seen before; only that the algorithm hasn't been written to sufficiently take account of human bias.
I think the word you're looking for is "familiarity", insofar as it describes a particularly efficient means of recognition. E.g. humans have become pretty good at identifying cats.
I don't see a fundamental difference between biological and electronic neural nets; so please take the following with a physicalist grain of salt. Imho, precisely because NNs will be fed with nothing else than the reality (physical or virtual) we live in, it should gradually develop the same familiarity as humans have; i.e. nothing more and nothing less than elements of our lives/civs. Visually lots of cats, lots of cars, mountains and coasts; functionally all the tasks we accomplish daily, like driving or cooking or cleaning.
I don't really think you can hard-code "human bias" as it's an emergent property of our biology: too complex (we don't really understand much of it, imho you're bound to miss the mark and induce subjective biases), and somewhat contradictory to how NNs are supposed to evolve (thinking long term here). Basically, I don't think it would be practical nor cost efficient to induce too much perturbations in deep learning, better work on refining the process itself. Think of plants: you can tweak the growing all you want, but the root deciding factors lie in genetics (their potential, and in understanding how to maximize it).
I realize another wording is that we should apply sound evolutionary (Darwin etc.) principles in "growing" AI at large. Because AI and humans share the same environment, we should see converging "intelligence" (skills, familiarity, etc). It's a quite fascinating time from an ontological perspective.
It's interesting to think about what the limits of an AI that doesn't have a full human experience are. I think you're probably right that machine vision will be competitive with human vision. It's already much better in specialized areas.
General purpose machine translation is harder, for instance. Brute force algorithms have gotten decent, but aren't in the same ballpark as humans (though professional translation services now often work by correcting a machine translation). However, MT systems trained on a specific domain do much better (medical or legal docs, etc).
What would be the hardest task for machines that's trivial for humans? Maybe deciding if a joke is funny or not?
Perhaps not the hardest, but one where there's tons of room for improvement: the Story Cloze Test [1] is a test involving very simple, five-sentence stories, where you pick the ending that makes sense out of two endings.
A literate human scores 100% on this test. No computer system so far scores better than 60%. (And remember that random guessing gets 50%.)
Interesting study; whilst it's possible to guess which ending is expected as correct, the alternate could be easily argued. For example, in the case of Jim's getting a new credit card, I recall during my uni days many students took that exact approach to debt...
Good point; I'd not considered whether the human imprint would be down to familiarity (individual's) or in-built through evolution (inherited familiarity); likely a combination of both.
In fact, I recently read that chimpanzees raised by humans are believed to identify as human rather than chimp; so individual familiarity does seem stronger.
The book, "We are all Completely Beside Ourselves" is fiction, but refers to findings from real studies.
Cats are probably a particularly unfortunate example to use in comparing abstraction forming cabilities, as given our history it's highly likely that we come supplied with some dedicated cat recognition circuitry.
Humans have a bit of an advantage on two levels here. First, we know what a cat looks like. Not a video or a picture or a drawing, but an actual cat. That gives us a solid frame of reference. "That is definitely a cat. That drawing looks kind of like what I know a cat to look like, so it's a drawing of a cat." The closest a computer can get is "This drawing has quite a bit in common with these other drawings, and apparently these other drawings are cats. So this is probably a cat too."
Second, when we look at a picture of a cat, we're looking at a human's interpretation of what a cat looks like. If we asked a computer to draw a cat, it might look nothing like a cat to us, but another computer could look at it and go "Oh sure, that's a cat." I seem to recall Google did a thing with this a while ago, where they effectively created a feedback loop in a neural net - feeding its own drawing back into itself. As I recall, the result looked like the computer had done way too much LSD.
Can you sketch an example of such a drawing? I'm having a hard time imagining something that looks enough like a cat to be recognized as such but unlike any cat a three-year-old has ever seen before.
I'd say that misses both my criteria: it looks just like lots of cat drawings any three-year-old has been exposed to, and it also seems like an image Google would have no trouble recognizing as a cat.
Again, I think first-world children over the age of 3 have been exposed to plenty of drawings like that, and also, Google can recognize it as a cat anyway -- in fact, it even knows which cat; do an Image Search and you'll see, "Best guess for this image: garfield meme"
I think we do. We see a building we've never seen before and we know it's a building because it has certain features that we use to classify it as a building. The examples aren't scarce.
I also think a good indicator of us doing it is the use of "y" and "ish" and "sort".
As for sthlm's point 2:
>2. The software can't recognize a feather if it's never seen a feather like that. It's not a sentient being.
When it comes to abstraction from a simple rendering – no shading, no sense of depth, no discernible dimensions – it's hard to extrapolate features.
I feel there is an immense difference between recognizing simple sketches and deriving what an object is based on extended characteristics.
The video you linked furthers that by showing that ASIMO was using three-dimensional observation to calculate certain features and ascertain what that object was.
Familiarity with teammates may factor into that as well, partially from having unspoken frames of reference to infer from.
It is unmistakable how much the difficulty level ramps up when you're paired with those of an unlike-nature to you. Sometimes that level of abstraction is taken way outside of generic context clues.
It is but we've also had decades of practice. What scares the most about AI isn't how advanced computers can become but how slow we are to learn in comparison.
Actually in mind when I was mentioning that was playing a game I coined "foot pictionary" (we've also played "blind pictionary") with kids ages ~6 to 10yo.
We use very generic "words" (eg egg, tree, bike, cloud, plate).
When you're using your foot to draw you really have to distill down to the essence of the item. Yes there is a deal of guessing but in some way the image (however unlike the object) has to have some element of the Platonic nature, if you will, of the object being drawn.
You're just wrong on this one. Humans can recognise a lot of things that aren't in the form that they're used to. It's seen a lot of research in psychology.
As for advantages over neural nets, one of the primary ones is that humans can recognise things from unusual angles much more easily. When I tried QuickDraw and doodled things from non-stereotyped angles (like a three-quarter view of a car rather than the usual 2D side view), it had no idea.
The dalmation optical illusion[1] is another example of human ability to pick out patterns and assign them to belong to certain objects. Neural nets have different abilities, and are sometimes better at picking out different sorts of patterns than humans.
Sure, you can fool a human. But there are things AI is missing that would be embarrassing if a human made the same mistake. It's hard to say, based on anecdotes like this, how big that gap is, but it's there.
> 1. The software can recognize a feather, as long as it looks similar to what it thinks a feather looks like.
I was prompted to draw a hurricane. I drew something that looked like the typical hurricane doodle used on news reports.
The software didn't recognize it.
When the game was over and I was able to look at all of the doodles that were used to train the software to recognize a hurricane ... the majority of them instead looked like tornadoes!
So maybe we should more precisely say:
1. The software can recognize a feather, as long as it looks similar to what the humans who contributed its training set think a feather looks like.
My hurricane was just terrible. I ended up with a scribbled mess because I got that in the first set or two, didn't really have a plan and drew components of a hurricane as I remembered them.
I'm also ashamed to admit I drew some less than ideal stuff due to forgetting details on things and then panicking because of the timer. Like the spots on a panda's face for some odd reason.
When the game was over and I was able to look at all of the doodles that were used to train the software to recognize a hurricane ... the majority of them instead looked like tornadoes!
Idiocracy was prophetic -- except it missed the aspect that "Idiocracy" would first manifest on the Internet.
Alas, if I had to stretch. Basically, there's rampant prejudice and anti-intellectualism from all points on the political spectrum. The response to the enabling of trolling by anonymity is an upsurge of authoritarianism by (of all people) many on the Left. The Right? Not much better.
If I had a dollar for every time someone pattern matched me or a phrase I wrote, then jumped to conclusions about my ideas of internal emotional state then even insisted I am lying when I tried to disabuse them of the notion -- I'd have a whole lot of dollars. (Hint, if you start sniffing around trying to justify that they're right, I haven't left you sufficient evidence and you're probably also doing that.)
Apparently most players of this game didn't see the "carrier" part in "aircraft carrier" and just drew airplanes. Probably because of the time constraint.
Which is actually a pretty big win. After all you could also say this:
1. The person can recognize a feather as long as it looks similar to what the other people who contributed to it's learning think a feather looks like.
Not exactly: if you've never seen a particular kind of feather before, you may not recognize it at first sight, but most certainly you'll sit, examine it and eventually acknowledge it's a feather -- the neural networks we're using aren't prepared to do this kind of analysis yet.
No, not at all. If you only showed it a bunch of pickup trucks in various colors, it would be really good at identifying pickup trucks. But if you then showed it a Prius, or a motorcycle, it would have no idea that it was looking at a vehicle. A human brain wouldn't have much trouble with that, though, because it associates more information with the vehicle idea than just statistical similarity to previously seen shapes, and can extrapolate without having direct previous experience with the object being seen.
Neural networks can learn new categories of things like that with about 5 examples. They are already outperforming humans on some tests. https://news.ycombinator.com/item?id=11737640
If you showed a small child 10 pictures of pickup trucks and told them "These are cars" then showed them a motorcycle and said "What is this?" what do you expect to happen?
Remember, this child has never been on the road, never driven a car, never had the mechanics of locomotion taught to them. All they know is that objects that are longer than they are tall with a flat bed on one side and wheels on the bottom are classified as cars.
Once the child (or machine) has more information to associate with the 'vehicle idea' it can call on this information when it sees shapes that are also associated with the 'vehicle idea' in order to extrapolate without having direct previous experience with that object being seen.
Trucks are generally not classified as cars, nor are motorcycles. These are all types of vehicles, per my original terminology. I actually did a similar experiment with my friend's daughter (3 years old) and she was able to figure it out just fine. Humans are generally able to extrapolate that things with wheels move, and if they have a seat, it's meant for someone to sit on, while it's moving. Hence a vehicle. It's this level of conceptual understanding and "how would this thing work" thinking that ML lacks in comparison to human brains. People use more than just sight recognition to identify new objects, while current ML models do not.
Maybe some current implementations lack the ability to make these connections but it is in now way even a small stretch to conceive a machine that understands "Wheels are for moving" "Seats are for passengers" "Things that have both wheels and seats are probably vehicles".
So when that machine learning algorithm recognizes wheels in a picture and recognizes seats in the same picture, it searches for results that include both wheels and seats.
The human brain does not inject any magic in to this process.
It sort of does, though. Let's say we train an ML implementation so that it can recognize things with wheels and seats as vehicles. Now we show it a hovercraft. What will it do? How about a helicopter? All the human brain needs is a single example of people getting in or on something, and it transporting them from point A to point B in order to infer that the thing is a vehicle of some sort. This is because we are able to infer purpose of an object even if we have never seen it before. ML is just statistics - it implies no meaning or comprehension whatsoever beyond "thing A is statistically most like thing B I have seen before". There's an important difference between recognition and understanding, and current ML techniques are solidly in the former camp.
The often forgotten difference between ML and humans is that we learn from stereoscopic video streams, not from a bunch of static pictures. There's a lot more information in a few seconds of watching cars on the road than in a thousand pictures of different cars. We get to see the 3D picture (we have dedicated circuits for that), hear 3D audio and perceive temporal data. We correlate all that and many more data sources to form categoties.
ML trained on bunch of static pictures is like humans dealing with those abstract geometrical riddles that are used on IQ tests. They're difficult for us, because they're not related to our normal, everyday experience.
> 2. The software can't recognize a feather if it's never seen a feather like that. It's not a sentient being.
Why did this word "sentient" sneak in to your comment? I don't see what "sentience" has to do with what you just described; it's just a more sophisticated form of pattern matching.
"See, it can't do this! It's not self-aware!" is almost never the correct answer, because whatever thing it is you want to do will probably be solved in the future with more of the same techniques. Just about the only thing "sentience" or self-awareness is good for is an entity's private experience, which you wouldn't ever be able to see anyway.
I don't think that's true for shoes. The male equivalent to high heels would be dress shoes, and women wearing male dress shoes would be weird and unusual. The examples shown appear to be casual or athletic shoes, which are indeed neutral.
Sit in a shopping centre, movie theater lobby, or even just out on the street. Watch the shoes of the women as they stroll by[1], and you'll find very few of the ridiculously high heels that are pictured in that tweet. Claiming that tweet's shoe as the typical women's shoe is laughably erroneously stereotyped.
[1] Not just the young fashionistas that specifically dress up, but every woman.
You might be able to explain it, but it still shows that it's wrong. (Though I disagree that these shoes are gender neutral; only ~5% of the shoes in my household look like "gender-neutral" shoes, and they're all mine)
Yes, an absolutely classic example of implicit biases in training sets.
On the one hand, the network should eventually learn to classify high heels as shoes.
On the other, when these classification system actually get used, they're always at some arbitrary point in their training, so you can't just wait for "all the biases to go away."
Erm... high heels are not the only kind of shoes women wear. They're not even the most common kind of shoes women wear. Pointing to this as a 'gender disparity experience' is showing your own bias. Yes, high heels are shoes and it should learn to recognise them, but most women don't actually wear them most of the time.
The comment was pointing out a specific example about how an AI miscategorized something because of a small sample-size in data, something that has been shown to be often the result of unintended biases in the training set, and you say that we're just talking about "gender identities"??
This is the kind of thing AI researchers write papers on (source: AI MSc), not some SJW topic, yet you saw the word "gender" and assumed it didn't belong?
#2 applies to humans as well. For example if I show a human something that looks and has all the properties of a car, the human will think I am showing him a car even if the thing I am showing him is actually called a feather.
Any neural net, artificial or not, can only recognize things as long as it looks similar to what it thinks the thing should look like.
335 comments
[ 3.0 ms ] story [ 270 ms ] threadFun though.
I was asked to draw 'eyeglasses', kept guessing 'glasses'.
It also seems to constantly have a "best guess" to some degree and if that happens to be correct it confirms pretty quickly.
EDIT: Drew a "cake", it guessed "birthday cake". Wrong answer apparently.
I guess it would have been more of an A.I. challenge if the premise was; draw anything and I'll try to guess what it is.
It could figure out most of my drawings but it would get them well in advance of me completing anything substantial (like others, I would be asked to draw a leg and draw just a curved line and it would guess leg before I finished).
Trying to draw what it asked for but with some unusual features (like lines or dot patterns around what it asked for before drawing it) and it gets extremely confused; it doesn't really seem to be good at filtering out any noise: http://imgur.com/a/oE1j2 (gallery of results and what it thought it saw.
Drawing things it didn't ask for just to see what it was guessing resulted in some really strange responses and fits. The answer set it has is extremely limited, so something like a hand giving the horns (\m/) was last guess a duck. A moose was a scorpion, then a duck, then a hand. Godzilla (or a bipedal dinosaur if you prefer) was a vase, then a scorpion, then a boat. My loaf of bread was a washing machine, an anvil, then a postcard. The Deathstar was a bandage, a helicopter, and a lighthouse. And a chainsaw was considered an aircraft.
Between the disruptive patterns and drawing things outside of it's vocabulary, the system seems really confused. Looking at the comparison results, I can see how when drawing some things it got it real fast. (Tennis Rackets were mostly defined by a crosshatch pattern, Harps by a series of parallel vertical lines). This makes sense. For other things, not as much.
It might be a more convincing presentation to give the user a list of items the machine knows (the full list) and tell the user to try to draw some, and then the computer could check it off as it gets them. That seems like a better way of presenting this than "Draw a box. hey! you drew a box! Isn't that cool?"
To me this is another interesting distinction on the NN recognition versus a human recognition - QuickDraw having a limited "vocabulary" to refer to really highlights this, as does my own lack of knowledge of Roobarb. Some of these things can really blindside us, and I suspect that it's going to require a lot of human hand-holding for awhile for the machines to get a strong vocabulary.
For some time I've been pondering how far you could take a machine's tabula rasa learning, for something like language, and how closely it would mimic a child's learning. (Language, color, math, etc).
Also, I realized how incredibly hard I suck at drawing.
This. I drew an Ant but the system couldn't guess it. Later it showed me how normal people draw ants.
Man, i suck at drawing ants
It should give you 8 ~10 words to choose from.
So if you draw a simple shape it start to go trough the list of things he recognize and end the game there.
It works great for this game because he can have very fast answer but only work win the cases when you actually have a feedback that eliminate all the wrong guesses.
Basically if it simply went trought the whole english dictionary fast enough he could get the same result without even looking at the pictures.
So nobody do that, right.
Seriously, don't even think about it.
(More seriously, I think I might have been de-trainging it by trying to draw the correct drawing using a touch-pad... rather too many of my phones seem to look like tornados by the time the dang thing's got stuck in draw mode trying to click-and-drag to draw)
It merely shows that it's able to pick the correct name from a finite list of possible names that could very well be small (e.g. 10 or 20 object names).
A better way to demonstrate that the ANN is truly able to recognize objects with good accuracy would be to ask a person to draw an unspecified object and guess its name.
If the goal is to acquire training data, they could still ask for a name in case of failure.
[1] http://detexify.kirelabs.org/classify.html
The ones I remember: bed, apple, spreadsheet, rollercoaster :D
We are so far from general AI. It's so strange how our brains can refine existance down so easily... how many CPUs is the human brain worth as far as we know?
http://imgur.com/a/xtdYE
For reference, here's my carrier
http://imgur.com/a/DL7js
The drawings at best look like churches (a building with a cross on top), but most just have giant crosses on the side.
http://www.viewroyal.ca/assets/Discover~View~Royal/Photo~Gal...
I guess if I was trying to doodle one, I'd do something like this: http://med-fom-emerg.sites.olt.ubc.ca/files/2012/06/surrey.j...
A carport with ambulances parked in front.
https://plus.google.com/+JohnHattan/posts/c6JWMbrFL9b
I'm also amused that one of the aircraft carriers has wheels :)
It sounds reasonable to call the Crawler-Transporter [1] a spacecraft carrier. From there it's not a big jump to wheeled aircraft carriers
But I'm probably giving people too much credit here.
1: https://en.wikipedia.org/wiki/Crawler-transporter
http://www.how-to-draw-funny-cartoons.com/image-files/xcarto...
After I did add the hole, it looked nothing like a steak to me. The only thing I could think of was to put a pan around it. As soon as I started drawing the pan it said, "Oh I know, steak". Makes zero sense to me.
Picture of my drawing:
http://i.imgur.com/90pcfdH.png
Does that look like a steak to you? In no way.
Here are the rest of my badly moused submissions if you're interested. I literally had no idea how to draw asparagus.
http://i.imgur.com/KT8Vrjw.png
I was pretty shocked it didn't figure out my pencil. In the last few seconds (I was already done) I started coloring in the eraser but it still didn't get it.
---------------------- EDIT:
Just did another one. Shocked it didn't get this!!:
http://i.imgur.com/BpdC1rZ.png
Completely obvious what it is. First I drew the first 3 circles. It didn't get it. I added flashing. Didn't get it. I added a pole. DIdn't get it.
Added a car and arrow. Still didn't get it.
Everyone else just had in the center of their image, the three circles. Some people included "shining" lines.
http://i.imgur.com/91ryOJ0.png
(third from bottom and second from bottom rows.) Obviuosly mine looks a LOT more like a traffic light than these other people's, because I contextualize it!
The microsoft chat experiment from a few months ago shows that people can hack these systems to train them into racist slur-spewers.
Similarly, i wonder how this project aims to get accuracy given that many people are going to draw the porno objects no matter what the prompt was to mess with it.
http://imgur.com/lBtGUKr
Times are changing...
[1] http://www.communia-association.org/2015/01/23/the-little-pr...
[2] http://www.thelittleprince.com/licensing/
> The narrator explains that, as a young boy, he once drew a picture of a boa constrictor digesting an elephant in its stomach; however, every adult who saw the picture would mistakenly interpret it as a drawing of a hat. Whenever the narrator would try to correct this confusion, he was ultimately advised to set aside drawing and take up a more practical or mature hobby. The narrator laments the crass materialism of contemporary society and the lack of creative understanding displayed by adults. As noted by the narrator, he could have had a great career as a painter, but this opportunity was crushed by the misunderstanding of the adults.
It's told in the first chapter of the famous childrens book, "the little prince"
Ignoring the cultural reference everyone here is talking about:
How do you get that perception? We apply the adjective adult to animals to whom we (rightfully or wrongfully) don't attribute intelligence, and even to other lifeforms which we usually categorize as non-sentient (e.g. adult trees).
I think it is perfectly legitimate to assign the adjective adult to a neural network that has either left the training phase, or that is only undergoing marginal changes in further training. This seems to be mostly in line with how the word adult is used in other contexts.
Not that I'm opposed to calling software intelligent, in fact I think it would be weird if we couldn't call something intelligent just because it's silicon-based instead of being based on organic neurons. I just find it odd that you associate "adult" with intelligence at all.
So in this sense, I found it surprising to associate a word that I use for "living"/biological things to a digital thing; specially in the context of A.I.
Maybe the term adult is, for me at least, very loaded with lots of meanings that go far behind the simple notion of "maturity"
Love Le Petite Prince! Read it both in French and English.
The things you draw here is new input for their model.
My bathtub looks too ridiculous for it to be able to guess it with confidence.
I just drew a bunch of dicks.
Perhaps the toes are drawn facing down?
I can show this to someone and say:
1. The software can recognize a feather, as long as it looks similar to what it thinks a feather looks like.
2. The software can't recognize a feather if it's never seen a feather like that. It's not a sentient being.
This is good, because most examples focus on point #1 and -- if enough marketing is involved -- don't go enough into point #2.
People read news articles like "X can recognize cats in a picture with Y certainty!" and are quick to assume that this "AI" can make sense of a picture and understand it, when all it does is apply certain methods for a certain use case.
This does a much better job by letting people write (or draw) their own test cases and figure out the limits intuitively.
For recognizing relatively simple entities, are there advantages humans still have over neural nets (assuming the same scope of knowledge)?
An idea: We can also run several cat photos through image processing algorithms to filter out details. The output would be outlines similar to the drawings in the Google Quickdraw app. We put those through the app to generalize (perhaps the app needs some training with a few categories of objects, not necessarily animals). Voila! Software can now recognize drawings based on photo examples.
Of course, there's severe bias here, in the sense that what we consider abstraction is by definition "human shaped" abstraction
If multiple humans try to "abstract" a cat, the overlap in underlying processes will be pretty big, making it more likely that we can recognise each other's abstractions.
I can read the words here, but I don't understand the meaning.
We abstract to find a common set of features in things that are supposed to be the same but that are not present in things that are not supposed to be the same. Grouping these features then produces higher level abstractions, and so on.
Where would the bias be?
Even if the features differ, the process is the same.
And even the features are often the same. If you reverse a DCNN to see what it uses to classify things as "cats", expect to see whiskers and fur.
Human perception is heavily biased towards features that had evolutionary advantages, and limited by whatever technical flaws our eyes/brains/etc have. That's a selection bias in our perception of information, in our processing of said information, and therefore in the abstractions that result from it.
I presume it's possible that the limitations of our visual system means we may miss powerful features and hence the ability to build some more powerful abstractions. (I didn't even argue this, just pointed out the process is the same even if features differ)
But I don't see how this supports your original claim of bias, which was: "If multiple humans try to "abstract" a cat, the overlap in underlying processes will be pretty big, making it more likely that we can recognize each other's abstractions."
If humans are good at recognizing each others' abstraction, that's a validation that low-pass (for lack of a better term) filtering the features due to human's physical design still creates very good abstractions and classifiers. That is to say, if anything you're confirming that humans are designed in a way that makes the abstractions they can make maximally useful.
... to other humans.
Are you arguing that the classifications themselves are biased?
Think of the Turing test and its criticisms; it's kind of has the same issues.
PS: I've upvoted every comment of yours; asking questions like this should be encouraged :)
My point is that "good" and "bad" are not objective here, but depend on human use-cases.
Now to be clear: I'm not disagreeing with you! These are good abstractions, for humans. It lets us communicate concepts easily, which is great! But it might not be the best abstraction in every circumstance.
For example, I recall reading an article that said that AI is better at spotting breast cancer from photos (which is essentially interpreting abstract blobs as cancer or not). The main reason seems to be that it is not held back by the human biases in perception.
Computers don't look at things from a human perspective; they're still good at abstraction, just different to human abstraction. i.e. there's a human bias in there.
That's OK though; the objective is to make a computer that sees things the way people do; so it's a bias we want.
However the issue isn't that the computer's not a sentient being and therefore can't abstract things it's never seen before; only that the algorithm hasn't been written to sufficiently take account of human bias.
I don't see a fundamental difference between biological and electronic neural nets; so please take the following with a physicalist grain of salt. Imho, precisely because NNs will be fed with nothing else than the reality (physical or virtual) we live in, it should gradually develop the same familiarity as humans have; i.e. nothing more and nothing less than elements of our lives/civs. Visually lots of cats, lots of cars, mountains and coasts; functionally all the tasks we accomplish daily, like driving or cooking or cleaning.
I don't really think you can hard-code "human bias" as it's an emergent property of our biology: too complex (we don't really understand much of it, imho you're bound to miss the mark and induce subjective biases), and somewhat contradictory to how NNs are supposed to evolve (thinking long term here). Basically, I don't think it would be practical nor cost efficient to induce too much perturbations in deep learning, better work on refining the process itself. Think of plants: you can tweak the growing all you want, but the root deciding factors lie in genetics (their potential, and in understanding how to maximize it).
I realize another wording is that we should apply sound evolutionary (Darwin etc.) principles in "growing" AI at large. Because AI and humans share the same environment, we should see converging "intelligence" (skills, familiarity, etc). It's a quite fascinating time from an ontological perspective.
General purpose machine translation is harder, for instance. Brute force algorithms have gotten decent, but aren't in the same ballpark as humans (though professional translation services now often work by correcting a machine translation). However, MT systems trained on a specific domain do much better (medical or legal docs, etc).
What would be the hardest task for machines that's trivial for humans? Maybe deciding if a joke is funny or not?
A literate human scores 100% on this test. No computer system so far scores better than 60%. (And remember that random guessing gets 50%.)
[1] http://cs.rochester.edu/nlp/rocstories/
The book, "We are all Completely Beside Ourselves" is fiction, but refers to findings from real studies.
Second, when we look at a picture of a cat, we're looking at a human's interpretation of what a cat looks like. If we asked a computer to draw a cat, it might look nothing like a cat to us, but another computer could look at it and go "Oh sure, that's a cat." I seem to recall Google did a thing with this a while ago, where they effectively created a feedback loop in a neural net - feeding its own drawing back into itself. As I recall, the result looked like the computer had done way too much LSD.
Google doesn't recognize it as a feline, it recognizes it as Garfield.
I think we do. We see a building we've never seen before and we know it's a building because it has certain features that we use to classify it as a building. The examples aren't scarce.
I also think a good indicator of us doing it is the use of "y" and "ish" and "sort".
As for sthlm's point 2:
>2. The software can't recognize a feather if it's never seen a feather like that. It's not a sentient being.
This is Asimo in 2009:
https://youtu.be/6rqO5eiP7_k?t=5m24s
I feel there is an immense difference between recognizing simple sketches and deriving what an object is based on extended characteristics.
The video you linked furthers that by showing that ASIMO was using three-dimensional observation to calculate certain features and ascertain what that object was.
If you'd give these doodles to people that are not Western males it'll do a lot worse. Someone already pointed out it doesn't recognize woman's shoes.
It is unmistakable how much the difficulty level ramps up when you're paired with those of an unlike-nature to you. Sometimes that level of abstraction is taken way outside of generic context clues.
We use very generic "words" (eg egg, tree, bike, cloud, plate).
When you're using your foot to draw you really have to distill down to the essence of the item. Yes there is a deal of guessing but in some way the image (however unlike the object) has to have some element of the Platonic nature, if you will, of the object being drawn.
Fun!
As for advantages over neural nets, one of the primary ones is that humans can recognise things from unusual angles much more easily. When I tried QuickDraw and doodled things from non-stereotyped angles (like a three-quarter view of a car rather than the usual 2D side view), it had no idea.
The dalmation optical illusion[1] is another example of human ability to pick out patterns and assign them to belong to certain objects. Neural nets have different abilities, and are sometimes better at picking out different sorts of patterns than humans.
[1] http://cdn.theatlantic.com/assets/media/img/posts/2014/05/Pe...
https://rocknrollnerd.github.io/assets/article_images/2015-0...
The software does:
https://rocknrollnerd.github.io/ml/2015/05/27/leopard-sofa.h...
Sure, you can fool a human. But there are things AI is missing that would be embarrassing if a human made the same mistake. It's hard to say, based on anecdotes like this, how big that gap is, but it's there.
http://rocknrollnerd.github.io/ml/2015/05/27/leopard-sofa.ht...
I was prompted to draw a hurricane. I drew something that looked like the typical hurricane doodle used on news reports.
The software didn't recognize it.
When the game was over and I was able to look at all of the doodles that were used to train the software to recognize a hurricane ... the majority of them instead looked like tornadoes!
So maybe we should more precisely say:
1. The software can recognize a feather, as long as it looks similar to what the humans who contributed its training set think a feather looks like.
I'm also ashamed to admit I drew some less than ideal stuff due to forgetting details on things and then panicking because of the timer. Like the spots on a panda's face for some odd reason.
Hopefuly my drawings were treated as outliers.
Idiocracy was prophetic -- except it missed the aspect that "Idiocracy" would first manifest on the Internet.
If I had a dollar for every time someone pattern matched me or a phrase I wrote, then jumped to conclusions about my ideas of internal emotional state then even insisted I am lying when I tried to disabuse them of the notion -- I'd have a whole lot of dollars. (Hint, if you start sniffing around trying to justify that they're right, I haven't left you sufficient evidence and you're probably also doing that.)
Apolitical people? Mostly just as bad.
1. The person can recognize a feather as long as it looks similar to what the other people who contributed to it's learning think a feather looks like.
Like humans brains?
>are quick to assume that this "AI" can make sense of a picture and understand it, when all it does is apply certain methods for a certain use case.
Like human brains?
Remember, this child has never been on the road, never driven a car, never had the mechanics of locomotion taught to them. All they know is that objects that are longer than they are tall with a flat bed on one side and wheels on the bottom are classified as cars.
Once the child (or machine) has more information to associate with the 'vehicle idea' it can call on this information when it sees shapes that are also associated with the 'vehicle idea' in order to extrapolate without having direct previous experience with that object being seen.
So when that machine learning algorithm recognizes wheels in a picture and recognizes seats in the same picture, it searches for results that include both wheels and seats.
The human brain does not inject any magic in to this process.
ML trained on bunch of static pictures is like humans dealing with those abstract geometrical riddles that are used on IQ tests. They're difficult for us, because they're not related to our normal, everyday experience.
Why did this word "sentient" sneak in to your comment? I don't see what "sentience" has to do with what you just described; it's just a more sophisticated form of pattern matching.
"See, it can't do this! It's not self-aware!" is almost never the correct answer, because whatever thing it is you want to do will probably be solved in the future with more of the same techniques. Just about the only thing "sentience" or self-awareness is good for is an entity's private experience, which you wouldn't ever be able to see anyway.
>People read news articles like "X can recognize cats...
may assume sentience when it's not there
https://twitter.com/OdaRygh/status/798872670221856768
[1] Not just the young fashionistas that specifically dress up, but every woman.
On the one hand, the network should eventually learn to classify high heels as shoes.
On the other, when these classification system actually get used, they're always at some arbitrary point in their training, so you can't just wait for "all the biases to go away."
This is the kind of thing AI researchers write papers on (source: AI MSc), not some SJW topic, yet you saw the word "gender" and assumed it didn't belong?
Any neural net, artificial or not, can only recognize things as long as it looks similar to what it thinks the thing should look like.