If I had only seen that video, I would think this was an April Fool's joke. I then read the article and was relieved to hear there is a little more than this. Perhaps I'm just being pessimistic, but I feel like this only raises the bar a little. I would expect checks it does of mouse movements or headers to be spoofable quite quickly after introduction. I think the best measure mentioned is tracking which IPs are bots, but that's still going to have serious shortcomings.
Most of the time it just gives you that one checkbox, but if you use the form multiple times (e.g. testing) it starts to give you the classical text entry box. I have no idea how it works fully and this article only sheds little light on it.
The old recaptcha takes good sites and gives them terrible user experience. I tried to order tickets one time on ticketmaster and actually gave up because I couldn't get past the captcha. I hate current captchas with a passion and I hope they finally die. I understand fighting spam but when it completely ruins a user experience it's not worth it.
edit: Bury me with no explanation why? Please don't tell me you think the UX of using recaptcha is great. I'm a 28 year old dev with near perfect eyesight and It takes me several tries to get these right. They are horrible. I welcome this new change and hope it isn't easily cracked.
"Bury me with no explanation why?" -- because your comment has only the most tangential relation to the linked article.
Downvotes are supposed to penalize uninteresting posts, not just wrong posts. No one likes captchas, and everyone has had shitty experiences with them. Your comment adds nothing to the discussion, doubly so since you're complaining about a type of captcha that has just been replaced!
I'm guessing your third sentence triggered a penalty. Replace "current captchas" with a blank and you'll see why.
I agree that the reCAPTCHA experience is terrible and assume many others agree with you, in part spurring the development of this new approach. I don't believe that every reCAPTCHA has a solution, or at least a consistent one, so I always feel like a percentage of time wasting is built-in. To work around it, I usually regenerate it until I get one that looks easy, but it's still frustrating. Improving the odds of getting it right the first time will help improve the experience a bit. But my biggest gripe is that they can make direct downloads impossible for resources that don't require extra protection.
The tiniest mouse movements I make while tabbing to the checkbox and hitting my spacebar to check it? Or tap it on my touch screen? And why wouldn't this be vulnerable to replaying a real user's input--collected on, say, a "free" pornographic website? Their answer seems to be "security through obscurity".
"Security through obscurity" is a weak concept, however the goal here is not security but fraud detection.
Obscurity is a legitimate component of a fraud detection system, for the same reason that hiding your cards is an important part (but only a part!) of being a good poker player.
I wish they would explain more about how the user interacts with the whole reCAPTCHA leads them to know it's a person and not a robot, but maybe they're worried about people writing bots to get around their protections.
> However, CAPTCHAs aren't going away just yet. In cases when the risk analysis engine can't confidently predict whether a user is a human or an abusive agent, it will prompt a CAPTCHA to elicit more cues, increasing the number of security checkpoints to confirm the user is valid.
Probably using a combination of G+ and GA to check your 'history' to see the activity is like a normal human. Visits a couple news sites each day, checks their gmail, searches for random crap randomly, GA registered a 'conversion' for some company = probably a human
I was thinking they may be looking at how long it takes for a user to click the "I'm not a robot" link. A robot would probably load the page and quickly, without delay, send the HTTP POST but I have to imagine they thought of this already and bots writers would quickly add a sleep() call in there at some point... Yea, I wonder about their internal logic too.
Or more likely they realize that explaining that your robot quotient is based on a statistical analysis of the last 6 hours of your browser traffic would probably freak everybody the fuck out.
They're almost certainly using the adwords cookies that get hit from 90% of the sites out there to figure out if you're a bot or not.
True enough, although it doesn't stop Google Now from coming up with helpful suggestions like "Hey, we noticed you were looking at this movie; would you like to see it on Amazon?"
It probably uses a LOT more info than only the mouse move/mouse click. Remember, google tracking is embedded in probably 99% of websites you visit so from your ip, cookies, tracking, logged in Google profile, etc... they're able to know if you're a human or not.
Nothing as dramatic : if you don't have a strong Google footprint, it is less likely that the system will recognize you automatically as human, and you'll have to answer the picture question.
Think of it as "remember my login on this computer". If you check the box, you sacrifice security and privacy for convenience. If you don't want to be tracked, you need to perform extra steps.
If you don't have advertising cookies set then you're either a robot or a potentially unprofitable human user from Google's point of view. Both would be best avoided.
As others have noted here, that's not the goal here and the captcha will degrade to the current ones in this case, but it highlights an interesting way for internet properties to maximize their revenue per user by only allowing users whose existing advertising footprint suggests they will contribute meaningful value to use the service in the first place.
Detecting bots is the first step to this, but detecting potentially unprofitable humans would be a natural extension.
I just hope it also works for pen-tablets, where the "pointer" can suddenly jump from one location to the next when the pen comes near the surface of the tablet.
Or the much more common case of touch screens. I'm assuming it's fine - the fine pointer movements are just one aspect of it, and tapping/clicking with a pen are likely to produce small movements anyway (whether or not those movements are suppressed by the driver of whatever device you're using is another matter).
> Google also will use other variables that it is keeping secret—revealing them, he says, would help botmasters improve their software and undermine Google’s filters
I'm pretty sure that looking at the javascript calls will tell what "variables" they use, with browser agent, ip, cookies.
Oh, so just program bots to provide a mouse movement toward a form element along a distorted path, and always trigger them through the UI rather than as events. Got it!
You seem to be assuming Google doesn't know of PhantomJS and many other automation frameworks. In fact, I personally wrote some of the code to detect them - not in ReCaptcha, but in a closely related project (which ReCaptcha may be using, even...)
Exactly what I was thinking. Even if Google got smart enough to detect that the distorted path speed was too mechanical, you could record 100's of macros of yourself moving the mouse towards a target. When it's time to submit a captcha, select one of the macros at random and play it back with some slight randomness added. Voilà!
The spammers thought the same when Bayesian spam filtering started working: "I'll just program a Markov chain to insert some none-spammy words". It did not work out quite so well.
I've also been thinking on how to defeat this, but even the mouse movement seems hard. You are trying to beat the big data statistics that Google has on these mouse movements. Programming a distorted mouse path does not take into account micro-movements (compare with saccades) or speed. To simulate mouse movements convincingly, I figure you'd need a model based on actual mouse movement data on these Captcha's. Also, you may succeed once or twice, but the third time the system is detecting your bot, making all your work void, since your code is now a new signature for bot-detection.
Bots want to quickly leave a message and move on to the next one. Real users first read an article, before they comment. If "Time on site before filling in Captcha" is a feature, then there may be no other way around detection, than for your bot to wait 5-10 minutes before filling in the Captcha.
I think this new system really makes it easier for non-bots to quickly fill in a captcha, and makes it harder for bots. Also because reputation (of IP, of cookies and behavior on Google domains) now seems to play a larger role.
The result of this reputation system for Captcha's is that Google admits to tracking their users outside their own domains. This also places the entire Google+ eco-system, analytics code, Chrome, fonts, charts and Google Hosted Libraries in a different light. Google tracks you everywhere, and the actions you take build or break your online reputation. I am not sure as a legit user that I want to trade this privacy concern, just so I can show my "humanity" and that I am not an evil bot. An evil bot would crawl the pages where these captchas are hosted, and join this data with the captcha user data. Then you can get personalized advertisements when you comment on 4chan in political threads.
Sorry if the article mentioned it (i only read the first few paragraphs) but looking at the cat click, that looks like a great way to generate training data for AI.
This is exactly what the previous implementation was as well. The "words" you had to recognize were either scanned from Google Books, or house numbers from StreetView, effectively enhancing their OCR training set.
It seems plausible that as more sites adopt this kind of technology, automated web access (e.g. scraping) the web will become harder -- for whatever purpose, good or ill. This has long been an "arms race" between hiding and detection. I can hope that reasonable uses of automation still remain feasible.
It was good enough for 4chan, a site that used to recieve countless spam from probably dozens of botters before they got recaptcha. And the botters probably kept trying to spam afterwards without much success.
Just use a bot with a clear User Agent, not "Mozilla/5.0 (iPad; CPU OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) CriOS/36.0.1985.57 Mobile/11D257 Safari/9537.53".
And, don't forget to start by reading my /robots.txt.
If you behave yourself and abide by the rules, why should I ban your bot?
If for whatever reason I don't want to allow your bot in, you might still try and contact me to ask, and perhaps I could arrange for your bot to scrape my site.
Automated web access to my site must obey my rules because you're using my bandwidth and resources.
That strongly depends on the data you have. I have written boots to scrape sites with government data so that I could do searches that wasn't possible using their online forms. I did not look at, nor attempt to obey, their robots text, nor would I have given a rats ass about breaking whatever they would put up to stop me.
There is also the argument that when you make something available on the web you make it available to everybody.
This comment seems like a non-sequitur. My comment had nothing to do with a particular site, much less "your" site.
I was making a general comment about automation and detection. If the detection gets better than the automation, it could change the dynamic. There is no fixed rule that says that content providers will or will not allow scraping based on robots.txt or other guidelines. Some could elect to disallow any/all robot behavior, if they have the capability.
I wish these google pages wouldn't automatically assume you speak some language based on your IP address - I get mine in German without any option to switch. Aren't there standards for setting language in web browsers?
> in the last week, more than 60% of WordPress’ traffic and more than 80% of Humble Bundle’s traffic on reCAPTCHA encountered the No CAPTCHA experience—users got to these sites faster.
Does this mean WordPress saw an 60% decrease in traffic from bots?
I attempted tabbing to the checkbox and pressing spacebar (not moving the mouse at all) and it worked just fine. Impressive. But I guess the tell-all is how secure it is against bots. not how easy it is for humans to get through. For all we know it could just be letting everyone through :P
It gives me the new version as well, but it seems google is convinced that I am a bot. Getting a regular captcha after clicking the button and I have to say that this is a lot worse of an experience than regular old captchas. Now I have to wait for a few seconds after clicking a button, then still solve a captcha.
NMC is a little different and goes beyond wasting a users time choosing colors. This is passive user authentication detecting human vs bot and bot vs bot. Today is an announcement of more platforms that can use it. Worlds most user friendly login protection in more flavors: #ClassicASP @DotNet @Drupal @Joomla #JSP #PHP
How long before you will have to answer a series of annoying and difficult questions if you don't allow tracking cookies and google to collect personal information (which I assume the far majority of users allow on a regular basis)? Not sure how I feel about this.
If they create services that you want to use so badly, they can charge you whatever they want for them. If you don't want to pay, I'm sure Bing will be happy to have you.
Google collects all your data and applies machine learning to predict a probability value that you are human. If it is below a certain threshold you have to enter a CAPTCHA.
Sorry, I just thought "all your data" was a bit unspecific. I have no idea what Google collects in order to derive that. I have a lot of guesses, but "all your data" doesn't tell me in fact, how it works. And I thinks that was what the parent poster asked for.
It seems like they're using cursor tracking to validate human-ness. Assuming thats the case: Since the cursor is outside of javascript's control, it would force the attacker one level higher (to the browser/os, instead of the dom). Not impossible, but still a significant barrier.
Not really, you would just need to reverse the js code, look at what data they actually send to google and randomly generate appropriate data like mouse movements.
That's an interesting thought that every time we introduce novel classification schemes to resolve humans from computers we feed a large training set for them.
I never understood how that works, though. If I get a captcha street address wrong then that means they already had the answer, so how am I contributing?
For the street address ones you generally have to answer two questions, one that they know and one that they don't know. You're only 'tested' on one of them, but you don't know which. Once enough people have gotten the known one right and given the same answer to the unknown image they move the unknown image to the known pile.
They already know the correct answer if they are only showing you one image to solve.
When there are two images to solve, they know the correct answer to one. The other is shown to thousands of people, and eventually it is solved with high confidence.
Tangential: From a philosophical perspective, I wonder if notion of asking human beings if they are robots, will soon escape the space we consider virtual? In sci-fi (which more or less informs the masses not involved in such fields and have more sway over public opinion than say HN or LW), the premise focused on is that people seemed more concerned with asking robots/automata if they are human. I'm starting to wonder if such questions will become moot.
Though, I wonder if you can start to defeat such systems by slurping up headers sent on public networks (like coffee shops, public wi-fi in large cities, airports, etc) and with techniques like ssl striping, to obtain local-storage info being sent in the body.
438 comments
[ 4.0 ms ] story [ 409 ms ] threadFrom an implementation standpoint it is utterly painless. The client side is copy/paste from Google's site and the PHP/server side was this:
Most of the time it just gives you that one checkbox, but if you use the form multiple times (e.g. testing) it starts to give you the classical text entry box. I have no idea how it works fully and this article only sheds little light on it.edit: Bury me with no explanation why? Please don't tell me you think the UX of using recaptcha is great. I'm a 28 year old dev with near perfect eyesight and It takes me several tries to get these right. They are horrible. I welcome this new change and hope it isn't easily cracked.
Downvotes are supposed to penalize uninteresting posts, not just wrong posts. No one likes captchas, and everyone has had shitty experiences with them. Your comment adds nothing to the discussion, doubly so since you're complaining about a type of captcha that has just been replaced!
I agree that the reCAPTCHA experience is terrible and assume many others agree with you, in part spurring the development of this new approach. I don't believe that every reCAPTCHA has a solution, or at least a consistent one, so I always feel like a percentage of time wasting is built-in. To work around it, I usually regenerate it until I get one that looks easy, but it's still frustrating. Improving the odds of getting it right the first time will help improve the experience a bit. But my biggest gripe is that they can make direct downloads impossible for resources that don't require extra protection.
Obscurity is a legitimate component of a fraud detection system, for the same reason that hiding your cards is an important part (but only a part!) of being a good poker player.
f(obscurity, time, analysis) = clarity
The details of implementation are left to the reader as an exercise
Probably using a combination of G+ and GA to check your 'history' to see the activity is like a normal human. Visits a couple news sites each day, checks their gmail, searches for random crap randomly, GA registered a 'conversion' for some company = probably a human
They're almost certainly using the adwords cookies that get hit from 90% of the sites out there to figure out if you're a bot or not.
As others have noted here, that's not the goal here and the captcha will degrade to the current ones in this case, but it highlights an interesting way for internet properties to maximize their revenue per user by only allowing users whose existing advertising footprint suggests they will contribute meaningful value to use the service in the first place.
Detecting bots is the first step to this, but detecting potentially unprofitable humans would be a natural extension.
I'm pretty sure that looking at the javascript calls will tell what "variables" they use, with browser agent, ip, cookies.
I have seen video-games bots that does that unbelievably well. Some powerbot.org scripts are sincerely more human than myself.
I've also been thinking on how to defeat this, but even the mouse movement seems hard. You are trying to beat the big data statistics that Google has on these mouse movements. Programming a distorted mouse path does not take into account micro-movements (compare with saccades) or speed. To simulate mouse movements convincingly, I figure you'd need a model based on actual mouse movement data on these Captcha's. Also, you may succeed once or twice, but the third time the system is detecting your bot, making all your work void, since your code is now a new signature for bot-detection.
Bots want to quickly leave a message and move on to the next one. Real users first read an article, before they comment. If "Time on site before filling in Captcha" is a feature, then there may be no other way around detection, than for your bot to wait 5-10 minutes before filling in the Captcha.
I think this new system really makes it easier for non-bots to quickly fill in a captcha, and makes it harder for bots. Also because reputation (of IP, of cookies and behavior on Google domains) now seems to play a larger role.
The result of this reputation system for Captcha's is that Google admits to tracking their users outside their own domains. This also places the entire Google+ eco-system, analytics code, Chrome, fonts, charts and Google Hosted Libraries in a different light. Google tracks you everywhere, and the actions you take build or break your online reputation. I am not sure as a legit user that I want to trade this privacy concern, just so I can show my "humanity" and that I am not an evil bot. An evil bot would crawl the pages where these captchas are hosted, and join this data with the captcha user data. Then you can get personalized advertisements when you comment on 4chan in political threads.
Just use a bot with a clear User Agent, not "Mozilla/5.0 (iPad; CPU OS 7_1_2 like Mac OS X) AppleWebKit/537.51.2 (KHTML, like Gecko) CriOS/36.0.1985.57 Mobile/11D257 Safari/9537.53". And, don't forget to start by reading my /robots.txt. If you behave yourself and abide by the rules, why should I ban your bot?
If for whatever reason I don't want to allow your bot in, you might still try and contact me to ask, and perhaps I could arrange for your bot to scrape my site.
Automated web access to my site must obey my rules because you're using my bandwidth and resources.
There is also the argument that when you make something available on the web you make it available to everybody.
I was making a general comment about automation and detection. If the detection gets better than the automation, it could change the dynamic. There is no fixed rule that says that content providers will or will not allow scraping based on robots.txt or other guidelines. Some could elect to disallow any/all robot behavior, if they have the capability.
There is always a center, just because it moves doesn't mean that people can't be a socially unacceptable distance from it.
There was one day when it was so convinced, it was giving me impossible captchas just to use Google Search.
If you are a recognized as human you could perhaps try it via Tor.
The rationalisation is that users don't change settings like these, so their location is a better indicator than what their browser thinks.
Yes, the Accept-Language HTTP header.
Does this mean WordPress saw an 60% decrease in traffic from bots?
Hopefully it gets better with time.
Looks like the new version needs an active and valid google cookie in order to tell if you're a robot or not.
http://nomorecaptchas.com/
It's very similar. I might go as far as saying that Google copied them.
http://ow.ly/FiuTi
I never understood how that works, though. If I get a captcha street address wrong then that means they already had the answer, so how am I contributing?
When there are two images to solve, they know the correct answer to one. The other is shown to thousands of people, and eventually it is solved with high confidence.
Though, I wonder if you can start to defeat such systems by slurping up headers sent on public networks (like coffee shops, public wi-fi in large cities, airports, etc) and with techniques like ssl striping, to obtain local-storage info being sent in the body.