The box has made browsing using TOR insufferable! It fusses and makes me click storefronts and traffic lights until I run out of patience and close out of whatever webpage I was trying to visit. I assume it has to do with a lack of Google cookies on the browser, essentially punishing me for trying to protect my privacy.
Click all boxes with traffic lights. Ok, well, this one box just barely contains the bottom right corner of the traffic light. Click. Nope, that little corner didn't count. Try again. Ok, well on this one, the right side of the traffic light is only barely over the line, so I won't click it. Nope, that sliver of the light mattered this time. MF!
Thankfully they'll eventually fall back to the "click the images of _object_ until there are no pictures left with a(n) _object_" in it, but those clicking block ones of a specific picture are super frustrating.
Heh, maybe one day they can show a bunch of pictures of sand, where each subsequent pic has a grain removed, with the instructions "click on all the heaps".
I assumed the infuriating ambiguity is intentional, in order to train some algorithm they need to know what the prevailing human correct judgement is in dicey situations
I don't think it's intentional -- it probably just emerges from the training process.
I'm guessing they do something like load up a batch of images and once N people agree on one, record the answer and remove it from the rotation. You end up left with the ambiguous images where people couldn't agree.
It is just a consequence of other humans also having problems with these cases. They do not mind that you have to make multiple attempts, it is just more yummy data for their bots (their machine learning algorithms are trained on this stuff).
I'm pretty convinced they're not really using these for ML, but that their ML algorithms have already run on these and they already know these difficult (read: ambiguous) enough to make you give up. These cases specifically only come up when they seem to think you're probably a bot (based on cookies or IP or whatever). They seem to deliberately put the photo boundaries such that they slice through whatever object they want you to look for. And they intentionally make the delays extremely long. These don't happen when they think you're probably a human and just want to throw an extra hurdle (like if you're Googling a little too frequently from your usual browser/location).
I always figure they're looking for a population consensus. They're doing image recognition at scale and these are clearly ambiguous, hard images to classify. They could easily have a few people at Google say, "I determine this is a storefront" and make that the "correct" answer, but I think they're more interested in a consensus of what most "normal" people would classify as a storefront, especially in potentially-volatile classifications where real humans might argue over the answer. They can skip the argument and just know which side will win it.
What they're actually getting though is the population consensus of what normal people believes Google's image classifier believes. The system incentivizes users to reinforce misconceptions their classifier has.
Google's image classifier would think that's a mountain. If you disagree, google will classify you as a robot. After failing these sort of challenges a few times the user decides to play along and tell google what they think google wants to hear, rather than the truth.
What makes you think Google's image classifer would think that's a mountain?
Especially if this is all used for learning, enough people saying "that is clearly not a mountain" would reinforce that it's, in fact, probably not a mountain. Even if I got classified as a robot, I'm not sure I would think "oh, a system designed to classify images would think this not-a-mountain is a mountain", so I definitely wouldn't double down and keep marking it as a mountain. I'd, well, not. And assume the system is at least as good as classifying the images it chooses to use as I am.
> "What makes you think Google's image classifer would think that's a mountain?"
Because every single time it asks me to classify mountains it rejects my answers if I don't click on trees on the horizon (and often trees on the horizon are the only "mountains" presented) and every single time it accepts the answer that such trees are mountains. I've gotten the mountains challenge dozens of times, the results are very consistent. If there is a group of trees on the horizon, that is asserted to be a mountain.
> "enough people saying "that is clearly not a mountain" would reinforce that it's, in fact, probably not a mountain."
Totally irrelevant because if I am trying to get through a google captcha, it's because that captcha is standing in the way of me doing something. My interest is in passing the captcha, not correcting Google's shitty image classifier. So I have absolutely no incentive to make my life harder by insisting on correct answers, and every incentive to tell Google what they want to hear.
>So I have absolutely no incentive to make my life harder by insisting on correct answers, and every incentive to tell Google what they want to hear.
I guess this is where the misunderstanding is. You don't think Google wants to hear the correct answer?
Trying to guess at what the daily/monthly flavor of "correct" is seems like it'd do more harm than good, resulting in some kind of nondeterministic guessing game of "well, trees on the horizon are probably assumed to be a mountain" that never settles on actually-correct answers (and, I'd wager, is often more inconvenient to the user than just answering correctly would be, because now there's a layer of indirection on what they think a system thinks of an image, rather than just what they think of that image).
If everyone just answered "no, that's trees" instead of a hand-wavy "I think you think it's a mountain", I feel like this captcha would be significantly easier for us humans (because we could actually give real answers), as well as less inconvenient for people who just want to pass on through and get on with whatever they were doing before a site wanted to verify they weren't a bot (because they can just, well, identify images instead of playing a game of "what does the machine think?").
> "You don't think Google wants to hear the correct answer?"
They may want it but they don't reward it. I don't care what sort of answer they want, I only care what sort of answer they accept. I'm not going to donate my time to these bastards by doing anything more than what's necessary to pass their captcha.
> "If everyone just answered "no, that's trees" instead of a hand-wavy "I think you think it's trees","
I actually have a few screenshots where the task was impossible since the data was mislabeled. The latest example was "click all of the buses". It wouldn't let me continue because I wouldn't select the fire truck.
My naive assumption is that you should click the "refresh" button in these cases.
Just click whatever you suspect is needed to pass. Don't go above and beyond trying to give the actual right answer; you're just feeding some proprietary database owned by Google. QA for it is their problem.
> " It wouldn't let me continue because I wouldn't select the fire truck."
Another one is "click the mountains". It typically won't let you through unless you click anything with trees on the horizon, even if the terrain is clearly flat. Google's robot thinks mountains are made out of wood, and any human who disagrees is labeled a robot. It's insanity.
I've recently gotten caught in one of these, where it was "click all of the bicycles" and after a few clicks (it was one of those which fade out to present a new picture) the only "bicycle" left was a bicycle-shaped street decoration. It wouldn't let me proceed unless I clicked on something, so I had to refresh to get a new task.
This might surprise you, but it actually has to do with what traffic coming out of TOR looks like. Well in excess of 90% of traffic coming out of TOR is spam, bots, malicious, or some combination!
Google isn't going out of their way to punish you for trying to protect your privacy. They're trying to stop unwanted traffic. By unfortunate happenstance, you appear to be disguising yourself in the exact same way a shocking amount of bad traffic is.
Nah, it was released back in 2017. I've seen it discussed periodically ever since.
The issue with just doing memory-consuming work client-side is that it only marginally slows down spamming. Spammers tend to use compromised machines they don't own. Unless you can make it prohibitively expensive to calculate something using machines you don't pay for - perhaps not a trivial ask - you wind up needing a different set of tools. This is why Google tends to look at things that will exhibit human variation rather than pure computation.
It's not that your ideas aren't good. I'm sure ARGON2 has a use here! It's that this might not be a problem easily solved by consuming more resources.
Cool, I'll try it out the next time I have a problem with using TOR. You're right that ARGON2 doesn't help if CPUs/RAM are free, it just makes parallelization hard.
Not really, because spam isn't done on the spammer's hardware. Not to mention, an expensive hashing function is precisely something bots can do but humans cannot.
If you're putting constraints on Tor traffic, it's not because of raw throughput. It's because it's extremely poor quality traffic.
> On the other hand, anonymity is also something that provides value to online attackers. Based on data across the CloudFlare network, 94% of requests that we see across the Tor network are per se malicious. That doesn’t mean they are visiting controversial content, but instead that they are automated requests designed to harm our customers. A large percentage of the comment spam, vulnerability scanning, ad click fraud, content scraping, and login scanning comes via the Tor network.
The obvious caveats apply, of course. It's completely possible what Cloudflare saw at the time is no longer true and TOR is no longer mostly spam. It's equally fully possible that the traffic Cloudflare sees is wildly unrepresentative of what TOR traffic actually looks like, and it's mostly people worried about their privacy. This is just the data we have at the moment.
A small percentage of bad actors using automaton can produce a lot of traffic. So although it may be true that a large portion of the requests coming from TOR exit nodes is malicious, it would be unwise to conclude that most users of TOR have bad intentions.
True, but from the perspective of an org like CloudFlare, that doesn't matter. They don't know (or care) about the user breakdown coming from Tor; they just know that the vast majority of traffic coming from it is malicious. And since part of the point of Tor is to make it hard to determine who's who, the good traffic gets binned with the bad.
Isn't cloudflare the org that "Doesnt censor under any circumstances", and then turned around and censored white supremacists? Not that I agree with them (I DONT!), but it was a full 180.
And also, isn't cloudflare also the one to allow booters and stressers to be online behind CF - and they used stolen CC's to boot?
The Tor decisions to screw users over is just the cherry on top. Especially is egregious is when a captcha is demanded on even a simple static page. Seems pretty obvious what's going on here.
Everyone should censor and shun white supremacists. They have no place in modern society. When they shed their noxious views, we can all welcome them back with open arms.
Ok, so you've decided that being white supremacist is bad. I can agree with you on that, but still the question remains: who get's do decide what has a place in modern society? Who decides what "modern society" even is? Today Google might decide to censor white supremacists, tomorrow it can be human rights advocates. I think that allowing any type of censorship, even for such a noble cause as fighing racism is a slippery slope. Especially when done by a private company that is outside of our control (and governments are only marginally better).
You're trying to generalize a useful rule ("shun white supremacists") but it doesn't work in this case. I don't think we need to, either.
We're not robots. We can shun white supremacists and leave everyone else alone. This isn't a slippery slope, it's just good sense (no more white supremacists, hey!). Humankind will get along just fine if we tack on that one extra rule and all follow it.
Whether or not abortion is a murder is not about definition of "murder" it is about definition of "human being".
There's no doubt that abortion involves killing a living creature, the whole pro-choice vs pro-life debate is basically about one simple question: "is fetus a human being?". If you answer that with "yes", then every abortion becomes a murder, plain and simple.
This also explains why there will never be a compromise between two crowds: it is logically impossible to compromise on yes/no questions.
Isn’t the compromise position essentially ‘after X weeks’, where the value of X is highly contested? (And on the binary yes/no question there’s nuances too which get debated eg if continuing the pregnancy would be a significant threat to the mother’s life)
> Good thing the definition of white supremacist is commonly agreed upon and noncontroversial and absolutely isnt subject to definition creep :)
The definition is commonly agreed upon, and what "white supremacist" means is not at all controversial to most people. It certainly isn't so arbitrary as to be meaningless.
Now, the term may be misapplied at times, as may any term, but for it to be misapplied, it has to have an accepted application to begin with. A term without a definition can't be subject to definition creep, and the possible creep of a term like "white supremacist" is that wide to begin with.
Just like the ACLU. Free speech is very important. If someone has something objectionable to say, let them expose themselves. Censorship solves nothing.
Just look at how state-backed censorship laws have turned Germany and Canada into totalitarian hellholes.
Yeah, right.
HN stans for white supremacy because it's mostly white and extremely online and never gets out to see the results of "give everyone a soapbox" on the streets where the proud boys are out there beating the shit out of people.
You people are cowards, and nothing more. There's no bravery here, no principled stand, just a bunch of fucking cowards.
I think you're doing a great job demonstrating why allowing people to expose their horrible ideas does more to dissuade other people than censorship. I'm glad your replies are on display even if I strongly disagree with them.
The obvious explanation is that people were downvoting and flagging your comment because it was unsubstantive and ideological flamewar, not because they are white supremacists.
You continued to post flamewar comments. We ban accounts that do that repeatedly, so could you please stop? We've already had to ask you more than once before.
Cloudflare's documented experience aligns closely with mine; I've been limiting or blocking TOR ever since 2008 because over 90% of the traffic was malicious bots, and the majority of the remainder was malicious humans.
And when you have malicious traffic swimming in an anonymous pool, there's no practical alternative but to block all of it.
I think you are partially wrong. Maybe Google is not doing this intentionally but it also doesn't happen just because traffic is coming out of a tor node. I am using ff with some of the recommended extensions from https://www.privacytools.io/ and I get to fill in traffic signs all the time. And yes I am logged into Google.
I think what OP is talking about is Cloudflare not Google's decision. Google provides the CATCHPA API but Cloudflare decides to flag nearly all Tor traffic and make it go through the CATCHPA.
Cloudflare is a good actor, they offer the PrivacyPass extension that basically generates 30 auth tokens from one CAPTCHA challenge and then uses those until it needs new tokens. Sadly the overwhelming majority of sites doesn't use CAPTCHA through CloudFlare but directly through Google, rendering PrivacyPass moot.
Cloudflare is not a good actor in this, they have shown that they do not care about encryption (allowing non-https backends while showing https to the end user) and embedding trackers in verification pages (the CAPTCHAs on random pages).
Cloudflare is the scum of the internet. They've put a crazy amount of effort towards making wide swathes of the internet unusable for people trying to protect their identity and privacy. I wouldn't trust their implementation of Privacy Pass.
In the case of Cloudflare specifically, they support Privacy Pass[0], an extension that allows solving one captcha to allow you through to multiple sites without de-anonymizing or reducing the security properties that tor provides.
Sigh. We changed this so long ago yet people repeat this over and over again. Do you use the Tor Browser? Please show me a site on Cloudflare which uses CAPTCHA on Tor.
I don't know about TOR, but a couple of years ago we had a site on Cloudflare that had the CAPTCHA come up for visitors from mainland China - where the great firewall blocked the requests to Google. Chinese users were effectively locked out. We contacted Cloudflare about this and got dismissive replies.
I ended up removing a chrome extension that randomises user-agents because of this. It dramatically cut down google captchas.
Another thing that sets it off is virtual machine usage, I can be logged into chrome and gmail on the same residential IP for hours but the moment I try to search google for a problem inside a VM it's a minute of slow loading captchas.
Have moved to bing instead, that sort of wasted productivity is a burden.
Google seems to do the same even if you check the box while in an incognito window; I doubt the issue is TOR itself, but rather the lack of tracking data that Google has on that particular session.
I think they do exactly that. For example disabling browser fingerprinting in firefox and not being logged into Google causes the majority of sites to display the captcha, especially when using a VPN.
This. The reality is, Google (and Cloudflare, and everyone else trying to block scrapers and malicious traffic) use heuristics that boil down to "99% of our traffic behaves like this". If you go out of your way to fall into the 1%, e.g. using Tor, disabling Javascript, randomizing your user-agent, etc., you're going to get CAPTCHAed.
Yeah, blending in seems to work better in many cases. Remember the guy who sent a bomb threat over TOR? The only reason he was caught so quickly was because he's the only person on the organisation's network to have accessed TOR before the incident.
I use Firefox with a few basic extensions (Privacy badger, uBlock, Google Container) yet every time I am presented with having to pick out traffic lights over and over and over again. I usually have about 5 or 6 "challenges" before I give up and use another site.
My timezone has not changed, my IP address and rough location has not changed, my screensize has not changed, my broadband speed has not changed, and my general computer dexterity has not changed, yet I am relentlessly targeted. On chrome I never saw these challenges, but on firefox with the privacy plug-ins I am always always always challenged.
At this stage I think the only signal it is using is "is there a google cookie in this browser? and if so has the google cookie got some 'normal' looking activity logged against it?" I.e. they are checking their server-side logs for a given cookie ID and seeing if that looks normal or not (i.e. seen on google search, seen on youtube, seen ads from a variety of third parties on various different sites, mixed up with time of day and speed of viewing etc etc).
Since I have got Google in a container in Firefox, I am guessing that my google cookie is not present when the captcha loads (due to the containers and privacy badger et al) so there is no identity back in the mothership to compare me against.
> My timezone has not changed, my IP address and rough location has not changed, my screensize has not changed, my broadband speed has not changed, and my general computer dexterity has not changed, yet I am relentlessly targeted. On chrome I never saw these challenges, but on firefox with the privacy plug-ins I am always always always challenged.
That's because Google isn't just profiling "Tor users". They're going after anyone who values privacy in any way or technology.
Simply put, you're being punished for ensuring privacy. And anybody who uses Google's captcha services is an accessory to that.
I think that Google is more than happy to punish people for protecting their privacy. That may or may not be the main goal, but it doesn't appear to be something Google considers a downside.
I think quora over states what Google looks at by a wide margin, just try to access a captcha in incognito, they won't have access to as much info as they do on you and yet you're still presented with the same level of captcha (if not more of them, which is to be expected)
Sometimes just checking the checkbox is enough. Sometimes you need to identify cars and store fronts. I think the better Google knows who you are, the more likely just the checkbox is going to be enough. If you go incognito, you have to train their neural nets, if you give up your privacy, you get in for free.
The clever part from Google's perspective is that you have to trade one of these things to Google in order to get access to sites that do not belong to Google at all. Google convinced site owners to have their users pay a tax to Google.
There are many services out there that can solve Google's recaptcha for fraction's of a penny. When someone puts one up, they can make things more expensive, and perhaps sometimes uneconomical, but in general, the cost is low (~$2.00 for 1,000 recaptchas).
When someone uses a recaptcha, they should think about why they are doing so. It's one thing to use it to save a business model, but it's another to use it to protect information that should be free anyway. The elephant in the room is government data. Many government agencies think that selling their data can be a nice source of side revenue, and a recaptcha is a good way of enforcing it. In reality, they just increase the costs for everyone, and those with means can obtain the information while those without means cannot.
Governments need to release their data, freely, without captchas or fees for single users and bulk users, no exceptions.
Totally agree. Fortunately the Dutch government is trying to make as much data open as they reasonably can, and regularly organise events to encourage developers to use their open APIs.
I've actually been pleasantly surprised at how much data /is/ available, and how much of it is available through common formats like Socrata Open Data API (for use with tools like https://github.com/xmunoz/sodapy)
The counter argument is that they do a great job with trivial stuff like registered dog's names, and less well with sensitive/important issues like policing.
What's the right way to leverage the platform developed for the first into the second?
captcha is google master blow against ad blockers.
a regular user, who they have all the info, give them dollars per ad impression. You, with your doNotTrack (ha! that was a joke) and privacy addons makes them only cents per ad impressions.
you are google's enemy. remember this when you get stuck in captcha hell (and consequently censored from most sites until changing device/ip)
I use chrome with Privacy Badger + uBlock Origin and I have to solve the CAPTCHAs every single fucking time. I even have to solve them multiple times. At this point I just leave a page if they have one of those captchas.
> Google isn't going out of their way to punish you for trying to protect your privacy. They're trying to stop unwanted traffic. By unfortunate happenstance, (...)
This does not agree with my experience. I browse without cookies and severely limited javascript (using umatrix), and I also encounter the myriad of ridiculous inconveniences that the OP was referring to. On the good side, however, the web is much faster and generally less annoying.
> I browse without cookies and severely limited javascript (using umatrix), and I also encounter the myriad of ridiculous inconveniences that the OP was referring to.
Isn't this also something that many bots do (don't run javascript and don't have realistic cookies)? It seems like just another instance of reducing your distance from the "bot" cluster in agent-space.
I have exactly the same experience without using Tor, living in Germany...
I personally don't care too much about the hassle, but I really don't like the idea that I'm basically playing Artificial "Intelligence"/doing clickworking for the not so community oriented efforts of Google.
If Google wants to do that, that’s their prerogative. What pisses me off is when a bank or similar “secure” type of service forces me to train Google’s ML models in order to access my stuff. I didn’t agree to provide unpaid labor to Google.
>This might surprise you, but it actually has to do with what traffic coming out of TOR looks like.
That's a massive load of bullshit. Google has a captcha challenge that only humans can solve. That alone is already sufficient to prevent unwanted traffic. That is how every captcha system works. However google is an exception. If you're logged in to a google account or are using chrome then google can use that information to track your captcha history. Privacy minded people avoid google like the plague and therefore they cannot be tracked.
>Google isn't going out of their way to punish you for trying to protect your privacy.
Except this is exactly what happens. It's not "unfortunate". It works like this by design.
If google cannot track you then the captcha will force you to do something that no other captcha system does: give you even more challenges even if you have solved them correctly. You will spend the next 5 minutes solving captchas correctly and then at the end it will tell you you've failed. This again is unique to google: correct answers lead to failure. The problem immediately goes away if you let google track you, it doesn't matter how bot infested the network is. No other captcha system does it this way.
Google is clearly doing this to get free labour to label their datasets, force people to have a google account and encourage them to use chrome.
It punishes humans more than computers because computers are more efficient multitaskers. A computer can find a productive way to use the second between each tile fade in, but a human has no realistic way to productively use that second. The human sits there staring at the screen waiting, while the captcha-solving computer does other things (perhaps solve other captchas given to it through other connections.)
Slight nitpick but past captcha successes are a characteristic of cyborg accounts, which still act as a bot most of the time.
A lot of the behavior that captcha exhibits is in part a function of feature analysis from ML models - features that may seem ridiculous to layman humans but make sense to a neural net plugged into the data.
If you are using TOR, and not accepting cookies, they are going to have no way of knowing that you are the same user who just solved the CAPTCHA. Every request is going to appear to be from a new user.
If you do everything you can to prevent google from knowing who you are, don't be surprised when they behave like they don't know who you are.
A botnet doesn't need Tor in the first place. And you can limit the use of a single captcha solution. It's not much different from the problem of a legitimate google account being borrowed by a bot.
> That's a massive load of bullshit. Google has a captcha challenge that only humans can solve. That alone is already sufficient to prevent unwanted traffic.
It's not bullshit, it just depends whether your website is being targeted directly or not. We're targeted directly and the robots hitting us are getting the CAPTCHAs solved, presumably with human help.
Have you used Captcha on TOR? It really does feels like they're trying to punish you. They give you about 4 pages of "identify the traffic light", all of which are difficult for humans, then reject and give you another 4 pages. Or that thing where it fades out for about 7 seconds before you click the next image, and then wait another 7 seconds.
As a webmaster I can confirm that I hard block all TOR traffic for this exact reason. 90% of this traffic is malicious robotic junk of some form.
Also, I’m just not interested in the remaining 10% "legit" traffic from people who are aggressively paranoid about their privacy. Almost all of them ended up being dickheads who were using TOR to abuse other members of our community.
To the people who think every website should treat TOR users with respect, please understand that you are intentionally making yourself indistinguishable from the mountain of robotic junk, abuse and human dickheads. It's not my fault that you have chosen to do this, and it's not my job to provide you with tools to prove you're not a dickhead.
To the people voting me down, please understand that I am relaying factual information about my specific experience as webmaster of various large-ish regional websites. If you don't like the facts, voting them down won't change them.
This is a horrible argument. What gave Google the right to be the moral authority of the internet (we, we did)? Even if 99% of exits from tor nodes are malicious, Google should have absolutely no capability to throttle this traffic. Unless you claim most of the traffic in tor are from bots, your argument doesn't make any sense. Captchas serve 2 purposes: slowing down bots, annoying humans. By putting captcha to tor exits, Google not only slows down miniscule amount of bots, but also annoys human traffic (good or bad). It is by no means a "good" thing that Google is capable of this.
CloudFlare at least is using a thing where you only have to solve a ReCaptcha once, and then you can cryptographically prove you did, without compromising anonymity.
It goes like this: "so you want to be anonymous and won't let us track every single thing you do? ok, then you'll help us train our AI so we can improve our self-driving cars and improve how Google Maps extracts information from Street View images"
They seem to keep adding categories though, which makes me suspect that it is all about ML training. Recently it's chimneys and bridges (although that one may be older).
The likely answer is a bit of both. They use the image tests because it's something that is still kind of hard to do for computers and then uses a small percentage of the boxes as unknown tests to improve some ML algorithm. Unfortunately as computer vision has gotten better they've had to make the challenges harder to the point where they're quite low quality and sometimes count very small features qualifying images. My least favorite is labeling 'cars' because it can be hard to tell if it wants to count cars way off in the distance through the adversarial noise they add to the images.
I guess Mozilla hasn't noticed that one yet. They've been removing captcha bypassing add-ons from their site. And because all Firefox versions that aren't buggy require add-ons to be signed by Moz it makes distributing them through other channels rather tedious.
that's good. tor traffic should not leak on the open web. that just diminish tor network and cause headaches to node operators.
if you care about all that, run a node without internet exit, and also strive to make your sites available on tor (hate the "hidden service" nomenclature)
I get the feeling that 90% of the check is if you are signed into a Google account, otherwise you're going to click some images. I've noticed this a lot of incognito mode where I will almost always have to do a captcha.
I don't think so. When I was traveling in Malaysia a few months ago I was always signed into my Google account, but constantly needed to fill in captchas and even got suspended from Google Scholar for a few hours for "suspicious traffic".
TOR doesn't protect your privacy, it just lumps you in with—and makes you indistinguishable from—the worst crap on the internet. If you don't want to be treated like crap, don't try to blend in with the crap.
When trying to detect ad fraud, one problematic scenario is that of replay attacks. It's basically when a scammer records human behavior on a site, and then replays a mix of actual their sessions to fraudulently click an ad.
The Quora answer is interesting but it's not clear to me whether the "I'm not a robot" box cannot be defeated in a similar manner.
It seems like they make that hard by mixing semi-random elements that are different each time, like user interaction data, with the user's browser fingerprint and location. This seems to be why they go to such lengths to obfuscate the algorithm they use to combine these elements. If the pseudorandom and the fixed elements were easy to separate it would make replay attacks much more feasible.
I work on bot detection at hCaptcha.com. (Hiring: reach out if you want to apply machine learning to stop bots and help websites monetize their traffic without ads)
In order to successfully execute a replay attack you would also need to pass the Turing Test, i.e. click the correct images. If you design a bot that starts a combinatoric attacking by trying random guesses we can easily confuse it, so most attackers try to use a solver service.
We can also identify how you interact with semantic content in the images when you click on the image and characterize your mouse interaction as human or non-human. Since confidence increases as more results come in we can also run them after the initial pass and then shadow-ban bots. (And notify the targeted website that we have determined e.g. a particular signup is a bot.)
Ultimately, many techniques beyond simple correct/not-correct are required to defend against the main attack vector: humans hired to solve captchas en-masse and make thousands of fake accounts. Modern ML is pretty effective for these kinds of problems. Browser obfuscation does not add real security, and today's reCAPTCHA (all versions) is easily defeated in practice.
This solution has more or less locked me out of certain accounts, except for when I want to spend a whole evening solving captchas in order to log in. I just don't use those services anymore, which means I've lost paid-for content that I'm practically locked out of.
Is there any place where I can find a comprehensive list of countermeasures to stop Google from recording and analyzing all the stuff that the article lists? According to the article:
It turns out they record and analyse:
- Your computer’s timezone and time
- Your IP address and rough location
- Your screen size and resolution
- What browser you’re using
- What plugins you’re using
- How long the page took to display
- How many key presses, mouse clicks, and tap/scrolls were made
And ... some other stuff we don’t quite understand.
If you want more granularity I’d suggest giving uMatrix a try. You’ll basically break every site at first and have to make adjustments for every site you visit (whitelist certain Ressourcen on a per domain basis) but I think it’s well worth it.
NoScript (which is totally fine) just blocks all JS, uMatrix can block much more.
NoScript lets you whitelist sites. Chrome's built in Javascript blocking is about as usable.
Settings -> Privacy and Security -> Content Settings -> Javascript and then change "Allowed" to off.
You can white list a domain by clicking on the padlock or the thing that says "Not Secure" in the URL bar on the left and clicking "Site settings" and changing Javascript to Allowed.
In my opinion, the only real effect JS blocking has is you start complaining in HN comments about sites not working.
I found umatrix required me to turn so many things on for the average site that I wasn't sure it was blocking anything significant anymore. I suppose I have up after a while
I can see why you’d feel that way but for me it still blocks a lot of stuff I don’t want.
I block some domains/companies via my hosts file and run a pihole but there’s always the odd advertising network etc I, or my pihole didn’t knew about that gets blocked by uMatrix.
> NoScript (which is totally fine) just blocks all JS
Depends on the version of NoScript. If you stick with the older ones before this was broken, you're good. You can't do that with the new Firefox, though.
Personally I prefer uMatrix. It has a nice CSS-ish quality to it, e.g. you can whitelist random subdomains like *.cloudfront.net _if_ the root page is a domain you trust already.
It's still a pain to get rolling at first but it feels more friendly to me than noscript
uMatrix is a serious pain in the ass with reCapcha however, and for any site that embeds video. Plus it treats every site as unique so you have to redo the green boxes on every site.
If you click on the asterisk between the domain name and the on/off button, it switches to global rule mode. Any allows/denies you save will be used across all sites.
I use NoScript, and only allow very specific scripts to run. There are sites that won't work without allowing a metric ton of sketchy scripts to run, but those are both a minority and tend to be sites run by major companies -- so I can ignore them without loss.
Nope, because it's a cat-and-mouse game with robots. It's not in Google's interest to publish it because it would give an advantage to the bot writers.
A bot absolutely can, you just need to use a more sophisticated bot. This article [1] is from August 2017, so the arms race has escalated and techniques improve, but the gist is the same: You just do a better job of simulating the "human" characteristics they monitor. Gen 4 bots (bots that run on an infected user's machine) can circumvent these measures as well.
Honestly, if there were a web standard where I could opt-in to all of this tracking and it meant I would be ‘trusted’ I’d happily have my user agent send them almost anything they want. I trust Google not to fuck me.
It is describing how the checkbox is collecting your browser's characteristics (eg they go to great length describing the webGL fingerprint) and your own characteristics (eg mouse behavior), such that when you click the box, you are determined to be a person or a bot. If they think you are a person, you don't have to do the CAPTCHA.
The whole bit about a double encrypted "VM" is overstating the case. The "VM" is "just" a bytecode interpreter, which at the end of the day can't do anything the browser's javascript engine can't do itself. Yes, it's some heavy obfuscation, and what's more interesting than the interpreter itself is the decision to spend what must have been lots of time/resources to develop it. It's security by obscurity, and in this case it is delivered to the client so obviously it's reversible. Maybe there's a deeper purpose.
EDIT: ah. the purpose is not to obfuscate. it is to fingerprint the CPU characteristics. by running their own interpreter, and changing the opcodes on the fly and such things, they can defeat JIT and learn something about the CPU itself. if they have user info (google cookie) they can know what CPU/CPUs that user typically uses and if "the checkbox" records something different it's a signal.
the answer is basically wrong. robots can and do. captcha has become an AI training tool and it now blocks humans as a consequence. the convenient side-effect being a nudge to those humans to change their behaviour in such a way as to never need captcha and thus support google's business: enable cookies, login to google, use unique identifiers (IP), etc
Why can't I just run headless chrome of firefox and have my bot click it there? "Aha here it is, so I click using system 'fake' mouse click". where's the catch?
Point a webcam at the screen and wire a mouse to the computer controlling the webcam, you'd have to simulate the computer moving the mouse like a human would but I don't see why it wouldn't work.
Because humans still get the picture selection prompts, so you'd have to write code good enough to read the prompt and then select the appropriate images.
You've now forced spammers to purchase webcams. And write code or whatever to make realistic mouse movements. This is expensive. Whatever they're doing likely isn't worth it anymore.
I don't get why you'd need a physical robot arm? Just have a usb device that simulates the mouse movements. Or just software to move the mouse virtually.
You don’t, you wouldn’t even need a webcam, you could just splice into the video signal at output and interpret it directly (much as the HDCP bypasses do).
Lately I've been getting reCAPTCHA prompts all the time even though I'm not browsing in incognito mode and haven't cleared cookies. All I'm doing is running a very basic ad blocker, using Safari (which blocks third-party tracking), and very rarely loading a Google site. The most interaction I have with Google is when I end up having to use my corporate Google account as SSO for some other site.
Given that I'm not doing anything unusual, it really feels to me like reCAPTCHA, for all its complexity, boils down to "what's your history using Google software? Oh you rarely use it? I'm gonna give you a captcha". It didn't used to be this aggressive, but it's really ramped up in the past few weeks.
Nonono, disabled people might be barred from using a service because of excessive recaptcha. That's what he means. I also think he really meant CVAA rather than ADA though.
That only really leaves blind deaf people out, at which point we might be reaching the limits of any technology to provide access to everyone without a tooooon of work.
Yeah but most people won't trigger that. Seems like most of the complaints here about triggering it often are from people who are blocking js/cookies/randomizing user strings. The NoCaptcha check box itself is better than the old system where everyone had to do the Captcha at least.
What's their solution for any other website? It seems like they'd have a very difficult time accessing ANY site.
In a quick search it seems like NoCaptcha is the accessible answer for the issues with regular Captchas. For the most part it seems to work, most of the complaints here seem to stem from people trying to actively block some of the evaluation metrics used by the checkbox (cookies,javascript,user strings,fingerprinting,etc) which makes them look very different from normal traffic which kind of by necessity makes them look a lot more like bots.
>which makes them look very different from normal traffic which kind of by necessity makes them look a lot more like bots.
But if they are doing so because they are disabled, and the difference means they receive a worse experience, may result in an ADA complaint (especially if a government service falling under section 508 is involved).
It happens to me occasionally and I am basically blacklisted from the internet. I have to solve 5 in a row and if I screw up its idea of what a streetsign is, I have to start over. It has made me cut out usage of most sites that use this broken and abusive tech. Welcome to the digital ghetto.
Anecdotally, I've gotten way more reCAPTCHA prompts since disabling third-party cookies and installing Cookie AutoDelete, so I suspect you are correct.
Same here. I have an alternative browser with no ad blocker, no tracking blocker, and sometimes I just copy the website from Safari to that other browser to avoid CAPTCHA.
I recently replaced a bunch of securimage captchas with reCAPTCHA v2. During testing I had to shut it off because it became increasingly more complicated every page reload. First it was just one page of traffic lights, but 20 minutes later I was having to click through 5-6 pages of images. This worries me that user's might get pissed off. I'd really like to know if I've made life harder for my users in an attempt to stop the spam from the horribly broken securimage captcha.
I cancelled and deleted my Spotify account because of that. Now any site that asks me to fill a Google recaptcha is met with a swift click on the Back button.
Really? You gave up one of the best streaming music services because you had to use a captcha every now and then? Huh. Good data point. But you're probably not our target audience if you give up that easy, we sell engineering tools for solving difficult problems.
> During testing I had to shut it off because it became increasingly more complicated every page reload. First it was just one page of traffic lights, but 20 minutes later I was having to click through 5-6 pages of images. This worries me that user's might get pissed off.
Why are you now badmouthing someone else for deleting Spotify over this exact same issue?
It'd be a good strategy if we were aiming for a totalitarian Google-sponsored police state..
Making (online or offline) life more difficult for people who don't want to use company X products could escalate to the point where you either accept the yoke and are admitted to the walled garden of "society" which company X has firmly cemented themselves under -- or you say no and find yourself unable to drive/fly/get a job/go to college/buy groceries in your town. It sounds like a big leap to make right now, but is a real possibility if Amazon/Google/FB don't get broken up soon.
Using a verified human Google history to allow people who would otherwise be flagged as potentially a bot to skip the CAPTCHA is justifiable. Setting up your reCAPTCHA such that the lack of a verified Google history is used as a "probably bot" signal is really quite awful.
This is likely due to the new canvas fingerprinting protections introduced in iOS 12 and Safari for Mojave. Google's NHT analyzers probably don't take well to these measures that attempt to defeat canvas fingerprinting.
I have noticed this too. I've switched to DuckDuckGo for everything and I haven't changed my habits. Started getting more captchas a couple weeks ago and I know I answered several of them correctly (I'd get tested 3 times in a row).
It is also plausible that because google analytics runs on so many sites that they could do something shady like put you in a pester segment if they see you coming from duckduckgo to other sites frequently. It is not hard to imaging using Recaptcha as a nuisance against other search traffic providers.
I doubt that many people would make a connection between their search engine and seeing captchas on other sites. So limited gain for, if anything, many unnecessary complaints.
This is why I love container tabs in Firefox. I like putting all of the recaptcha stuff in one container so it can't snoop on my other stuff (I'm too lazy to look into what it's doing with cookies and whatnot).
Sometime ago we have a provider that give us everyday different IP. Some days we just were not able to do anything without captcha. It seems that some IP addresses were in some sort of spam base
Totally anecdotal: Back when I was using a vpn and would see this recaptcha more often, I found that the recaptcha would often declare me a bot (and give me a second chance) if I clicked the boxes too quickly. Like, the storefronts would load and I'd immediately click+click+click+click then submit. But if I slowed down and staggered my clicks, it would realize I'm just another inefficient human who needs time to move a mouse and make decisions.
Right. I think I have seen similar behavior for mouse movement as well. Not sure, but I think moving your mouse around randomly helps as well. The timing of picking the images, as you say, helps. As does the order you pick them in.
Of course, I think the biggest thing is your browser's fingerprint. If you are using a lot of privacy blocking addons, etc. you are going to be spending a lot of time looking at captchas.
I feel a sense of dread whenever I see this box. Is it going to let me through, or am I going to spend the next few minutes futilely clicking signs and lights, only to give up and leave the site?
Counter anecdote: I was actually never rejected though I’ve been subjected to the picture test multiple times. Most of the time I just get through, though it is hard not to sigh and roll your eyes when presented with one.
Annoying as they are, I don’t really see a better alternative. They’re also pretty easily circumvented with cheap labour like mechanical turk and similar services.
Someone is suing Apple for making 2FA too annoying, I only hope Google can be next for the significantly greater time being wasted on their stupid CAPTCHA. Why should any of us even care if someone accesses a website programmatically?
It's almost definitely not my problem, and it's not even necessarily a problem it's just a way to access pages, made easier by Google in fact because it's a perfectly legitimate way to navigate the internet and Google themselves depend on it. If anyone wants to discourage it just provide an official API to use instead of pages, which are a defacto API for humans.
Apple's 2FA cannot be disabled, beyond that if you ever had two devices connected to your account, good luck getting them to use your verified phone instead... I was trying to get rid of a "your apple account... please sign in" via the app store, and the advice was to sign out then back in again... my only other apple device was in another city...
Phone calls to GF, to get her to login, similar issue, it was now signed out, and after another attempt, one of them showed the 2FA window, which got me in. At no point was I able to use the phone number also associated with my account, and it was a huge, long, pain in the ass. It's actually a huge part of why I won't buy an iPhone.
Why? My time is valuable. If they want such detailed feedback (i.e. free business advice) from me, they can compensate me for it. I would get nothing in return for spending my time explaining why I'm not going to give them any more of my money.
Presumably they'll already make you waste time talking to a human when you close your account. Might as well mention the captcha to make public the distaste for recaptcha as we are now on this forum.
As superficial as it seems, this kind of stuff drives me away from businesses. I almost changed my mind about opening a retirement account with Charles Schwab purely because they block access to their site from the VPN that I used at the time. That said, for services from which I gain a lot of benefit, the annoyance is worth it, so it ultimately doesn't matter.
just preemtively say no and leave the site. this is just another tracking vector for google and it should be discouraged.
i'm generally against this type of gating, where the people doing the right thing get punished disproportionately (even small slices of time add up to wasting thousands of human-years over the population) just to combat the tiny number of bad actors. target the bad actors directly.
it's the same for tsa security theater. let's put all those humans to work training dogs of all sorts and filtering them through people at the airport. the money for those privacy invading scanners can be put toward training and housing the dogs. our collective time is not wasted on silliness and standing in line, and we'd probably save a lot of tax dollars that way.
Yeah, the extreme comments that we see whenever the topic of ReCaptcha comes up are especially amusing because HN uses it on its own register/login page.
Yeah it's ironic. It's more convenient to use the phone to get my balance, even if you count the time spent listening to the recorded message telling me how much better my life would be if I used Internet banking.
> I have also trained myself to wait a few seconds before clicking the box, which seems to help assert my humanity.
I wonder how many different weird rituals are out there for 'beating' CAPTCHA?
For me, I usually make some effort to keep moving my mouse and being "active" after clicking the box, on the idea that an isolated click event looks less human. It's based on a friend's tip and it seems to help, but I have no confidence that it's actually relevant in such a complex system. I sort of suspect Google has created a new generation of meaningless routines fit to rival historical standouts like sports rituals.
Sadly, the parcel tracking functionality of the Royal Mail's site here in the UK uses it, so I have little choice but to go through this rigmarole a few times a week.
On the one hand, fuck Google for wanting another vector with which to track me, on the other hand, why can't it remember that I proved to it that I'm human (allegedly) two days ago?
I was being a bit pithy, but in that case, surely there's some sequence of smart fuzzily-heuristic analysis that can be played out to check that my humanised identification isn't acting like a bot?
yes, the problem is prevalent and (small) forum operators bear the pain acutely. i'm not so dogmatic that i can't sympathize with your need for a working, if imperfect, solution, or as a user, needing to jump through the captcha hoops occasionally.
this is a subtle externality borne from scale combined with zero marginal cost. we need to find a way to make the bad actors bear the cost of that externality rather than the rest of us.
it's a hard problem that resists simple solutions like captchas, certificate signing, delivery fees, taxes, or even just outlawing the practice. as a lowly consumer, one of the few levers i have is curtailing my use of services employing (google) captchas.
I do this after being blackholed by goolgles captcha a dozen times. If your site reli3s on google you are the problem at this point. I am not a data point. I am a free man.
I hardly ever get to just check the box. I think uBlock Origin and Safari's intelligent cookie blocking removes a lot of identifying details, and ReCAPTCHA relies a lot on browsing history and cookies.
I use Firefox with containers (so ReCaptcha can’t see my Google cookies) and also uBlock Origin + a few more extensions. I don’t use NoScript or similar. I get the manual test for ReCaptcha pretty much 100% of the time.
I'm willing to bet if you record how long you spend clicking signs and lights on average, it's going to be more like several seconds than a few minutes. This must be hyperbole. There are the outlier cases where it gets fairly annoying but otherwise, I'm not sure I understand the hostility toward such a benign system that actually does thwart bots very effectively.
Effective or not, I wouldn't exactly call it benign when it's aggressively fingerprinting you based on every bit of data it can access.
I disable most scripts using uBlock, but I have to make an exception for reCaptcha if I want to interact with half the web. I'm pretty sure due to that alone Google is able to track where I go.
> I'm willing to bet if you record how long you spend clicking signs and lights on average, it's going to be more like several seconds than a few minutes.
How much, and at what odds?
The length of the average Google CATPCHA has been steadily going up for me. I haven't pulled out a stopwatch, but I do count how many image sets I go through. I basically never get through on the checkbox unless I've done one on another site shortly before. I sometimes succeed after one set, but if I don't it's consistently 3+. The worst case I've seen was 10 layers of slow-loading images without success, at which point I gave up and tried on another device. (If it had been a site I didn't need, I'd have given up after 5 - which I do fairly often, so I don't have an average count needed to succeed!)
I'm fully aware that average users don't have this much trouble, or people would be furious. But I also see that captcha ramps up to an extremely long process in the face of even modest privacy-protection efforts like not running Javascript or allowing third party trackers by default. (God forbid you're using a VPN for any reason.) It's not assessing your humanity but your familiarity, using the same fingerprinting tools as any site that wants to track you.
Spam is a real problem, and a hard one to solve, but I admit I'm hostile to Google's captcha. Partly because it really is a significant time sink for me. Partly because it lacks any progress indicator or fallback option so it's an indefinite hurdle to accessing sites I'm already committed to using. But largely because, despite what I really believe are good intentions, it's yet another force pushing people to give up privacy and even security if they want websites to work tolerably.
Personally, I've reached the point where this is the first thing that I do. If a site is presenting a CAPTCHA (especially one run by Google) to me, then 90% of the time, that's a site I'm better off avoiding anyway.
That last 10% can be infuriating, though, and I certainly won't feel positively about it.
And if you have a vision issue like me and you try to use the audio version you will often get "Your computer or network may be sending automated queries. To protect our users, we can't process your request right now. For more details visit our help page" and be unable to continue.
Every time I fill one of these out I get the picture test, and I answer them correctly...but am asked 3-5 times to identify which blocks contain a school bus or stop light. It's very annoying.
I think it's speculated that you're recorded as being a useful classifier if you answer correctly on initial test captchas, so you get given Google's datasets for machine learning. It would explain why you get picture tests even after you should definitely pass the check.
In a parallel universe of fluffy niceness we willingly provide our help and in that way we get all those old books converted to ASCII and available for us to read online. Our efforts are for the good of mankind. Similarly with the newer challenges, we help the maps be up to date and again this is for the good of mankind and those needing help getting around.
Clearly this doesn't work in an era where the 'don't be evil' mantra is long forgotten and people only see Google as some advertiser friendly capitalist monopolist beast.
Google need to work on their relationship with their customers, to be a benevolent dictatorship of sorts. They are lousy at customer service and there are other pain points that they are ignorant to. I don't see how this helps.
By offering a free service to website operators that mitigates the ever-growing challenge of abuse on the internet that they and their users have to deal with.
I feel like a "bot" with access to control your mouse and read the output from the graphics system (aka get an image of the screen in real-time) could have little trouble with these things.
Slightly related, but I have a fun conspiracy to share:
I'm convinced that part of the reason Google released headless Chrome is as a honeypot for bot authors to use. The idea is that instead of going through the effort of fingerprinting and identifying new bot software, release something that bot authors will use instead that you have a capability to detect.
Somewhere inside of headless Chrome, there's one or more subtle changes that make it so Google can detect whether you're using headless Chrome or normal Chrome. There's no limit to how subtle the indicator could be - maybe headless Chrome renders certain CSS elements slightly slower than normal Chrome, etc.
It sounds pretty crazy/complicated but I could definitely see it being worth it if it means detecting $X,000,000 worth of ad fraud every year
Most modern credential stuffers use headless browsers with all the bells and whistles, html5, javascript, etc.
Login attempts are usually spread over a massive botnet of residential IPs as well, where they'll only use each IP for one or two login attempts before moving on to the next.
Does it mean that you're breaking the experience for users who deliberately disable js by default? Can I ask you not to do that? Modern web is unusable if you let js on any webpage
It's actually not that complicated. Most headless browser drivers have some global JavaScript functions in the `window` namespace that immediately identify themselves.
I once ran into a piece of code from the scammy advertising world that tried to redirect users to a phishing site. They cleverly tried to hide themselves from the automated quality checks some ad networks do, by checking for these functions and appearing benign if they saw them. One of the checks even created an exception and then inspected the stack trace for certain flags that apparently are only there on some type of headless browser. Clever!
I think recaptcha and captcha in general are very overused today, as they cause way too much inconvenience for the user. Why discriminate against robots so much?
What's so wrong about crawling or using automated tools?
With today's networks and hardware performance most websites shouldn't concern themselves with denial-of-service type of attacks, unless they're past a certain threshold of popularity.
One thing is account creation. It's usually a bad idea that user accounts are created by non-humans. In general, anything that writes to a database may have concerns about robot/automation/abuse.
I agree with you, for "read only" content, let the robots crawl.
There's nothing inherently wrong with crawlers and automated tools, but there's a lot of poorly written scrapers out there that can easily generate a huge volume of traffic along with the associated maintenance.
Why waste time catering to people who are too lazy to write a decent scraper? Most people would prefer to spend their time working on things that will benefit the other 99% of their users.
At work we have a scraper that likes to use a particularly expensive search query using hundreds of different ips dozens of times per second, all so that they can scrape data that is freely available from us as an XML feed at a different url.
Every time I found a way to fingerprint and block them, they'd change their bot to avoid detection. Captcha for rate limiting seemed like the least bad option.
However, eventually I decided that instead of blocking them, I'd return random results from the db. They weren't checking the data too thoroughly because to this day they haven't changed the bot to avoid my most recent pattern detection. They're still merrily scraping useless data.
Rate limiting can be achieved without actually inconveniencing regular users. Put a exponentially growing delay on the server's response for requests coming in too quick succession.
The problem was that with the sheer number of ip addresses they were using, and with the rate that some normal users used that particular endpoint, regular users would have been inconvenienced--either by being forced to wait or by being forced to do a captcha.
I would have set it to only show the captcha if the delay was active, so users would effectively have had a choice of wait or captcha.
Awesome idea giving them crappy data. I do the same thing with loyalty programs. Someone has already registered XXX-555-1234 as a phone number in your area code. I like to think I'm adding a little bit of chaos to their data tracking.
Because 99% of bots on the web are spam bots. Naked email addresses are skimmed and cataloged. Any unprotected contact form will receive dozens of spam messages per day. Forums constantly get new members that only post linkspam.
There's very few friendly bots like archive.org, and those respect meta robots tags anyway.
So basically they do very advanced browser fingerprinting? I wonder if they keep that data around, as that would tell them who uses which third party services (not that they would have trouble working this out by other means) and would make a nice addition to their tracking efforts.
It’s quite depressing to see that almost all sites now require sending data to Google just to log in to them. Not to mention that they help to turn billions of users into clickworkers for annotating AI data. Tell me what you want but I’m pretty sure they show image captchas to users that they are absolutely sure are no bots. I use Chromium on Linux and I’m logged into a Google account, still I have to solve three or four image captures at times to use a login.
647 comments
[ 3.1 ms ] story [ 346 ms ] threadClick all boxes with traffic lights. Ok, well, this one box just barely contains the bottom right corner of the traffic light. Click. Nope, that little corner didn't count. Try again. Ok, well on this one, the right side of the traffic light is only barely over the line, so I won't click it. Nope, that sliver of the light mattered this time. MF!
Thankfully they'll eventually fall back to the "click the images of _object_ until there are no pictures left with a(n) _object_" in it, but those clicking block ones of a specific picture are super frustrating.
Spambots will solve the Sorites paradox!
I'm guessing they do something like load up a batch of images and once N people agree on one, record the answer and remove it from the rotation. You end up left with the ambiguous images where people couldn't agree.
Does this look like a mountain to you? https://0x0.st/zzvr.jpg
Google's image classifier would think that's a mountain. If you disagree, google will classify you as a robot. After failing these sort of challenges a few times the user decides to play along and tell google what they think google wants to hear, rather than the truth.
Especially if this is all used for learning, enough people saying "that is clearly not a mountain" would reinforce that it's, in fact, probably not a mountain. Even if I got classified as a robot, I'm not sure I would think "oh, a system designed to classify images would think this not-a-mountain is a mountain", so I definitely wouldn't double down and keep marking it as a mountain. I'd, well, not. And assume the system is at least as good as classifying the images it chooses to use as I am.
Because every single time it asks me to classify mountains it rejects my answers if I don't click on trees on the horizon (and often trees on the horizon are the only "mountains" presented) and every single time it accepts the answer that such trees are mountains. I've gotten the mountains challenge dozens of times, the results are very consistent. If there is a group of trees on the horizon, that is asserted to be a mountain.
> "enough people saying "that is clearly not a mountain" would reinforce that it's, in fact, probably not a mountain."
Totally irrelevant because if I am trying to get through a google captcha, it's because that captcha is standing in the way of me doing something. My interest is in passing the captcha, not correcting Google's shitty image classifier. So I have absolutely no incentive to make my life harder by insisting on correct answers, and every incentive to tell Google what they want to hear.
I guess this is where the misunderstanding is. You don't think Google wants to hear the correct answer?
Trying to guess at what the daily/monthly flavor of "correct" is seems like it'd do more harm than good, resulting in some kind of nondeterministic guessing game of "well, trees on the horizon are probably assumed to be a mountain" that never settles on actually-correct answers (and, I'd wager, is often more inconvenient to the user than just answering correctly would be, because now there's a layer of indirection on what they think a system thinks of an image, rather than just what they think of that image).
If everyone just answered "no, that's trees" instead of a hand-wavy "I think you think it's a mountain", I feel like this captcha would be significantly easier for us humans (because we could actually give real answers), as well as less inconvenient for people who just want to pass on through and get on with whatever they were doing before a site wanted to verify they weren't a bot (because they can just, well, identify images instead of playing a game of "what does the machine think?").
They may want it but they don't reward it. I don't care what sort of answer they want, I only care what sort of answer they accept. I'm not going to donate my time to these bastards by doing anything more than what's necessary to pass their captcha.
> "If everyone just answered "no, that's trees" instead of a hand-wavy "I think you think it's trees","
That's just not going to happen: https://en.wikipedia.org/wiki/Prisoner%27s_dilemma
My naive assumption is that you should click the "refresh" button in these cases.
Another one is "click the mountains". It typically won't let you through unless you click anything with trees on the horizon, even if the terrain is clearly flat. Google's robot thinks mountains are made out of wood, and any human who disagrees is labeled a robot. It's insanity.
And, does the pole count?
The whole thing is way more stressful than it needs to be for what it is.
Google isn't going out of their way to punish you for trying to protect your privacy. They're trying to stop unwanted traffic. By unfortunate happenstance, you appear to be disguising yourself in the exact same way a shocking amount of bad traffic is.
It should be default for the TOR browser for sure, if just a few people use it, it decreases the anonimity set.
The issue with just doing memory-consuming work client-side is that it only marginally slows down spamming. Spammers tend to use compromised machines they don't own. Unless you can make it prohibitively expensive to calculate something using machines you don't pay for - perhaps not a trivial ask - you wind up needing a different set of tools. This is why Google tends to look at things that will exhibit human variation rather than pure computation.
It's not that your ideas aren't good. I'm sure ARGON2 has a use here! It's that this might not be a problem easily solved by consuming more resources.
If you're putting constraints on Tor traffic, it's not because of raw throughput. It's because it's extremely poor quality traffic.
Do you have any data on this?
> On the other hand, anonymity is also something that provides value to online attackers. Based on data across the CloudFlare network, 94% of requests that we see across the Tor network are per se malicious. That doesn’t mean they are visiting controversial content, but instead that they are automated requests designed to harm our customers. A large percentage of the comment spam, vulnerability scanning, ad click fraud, content scraping, and login scanning comes via the Tor network.
The obvious caveats apply, of course. It's completely possible what Cloudflare saw at the time is no longer true and TOR is no longer mostly spam. It's equally fully possible that the traffic Cloudflare sees is wildly unrepresentative of what TOR traffic actually looks like, and it's mostly people worried about their privacy. This is just the data we have at the moment.
And also, isn't cloudflare also the one to allow booters and stressers to be online behind CF - and they used stolen CC's to boot?
The Tor decisions to screw users over is just the cherry on top. Especially is egregious is when a captcha is demanded on even a simple static page. Seems pretty obvious what's going on here.
We're not robots. We can shun white supremacists and leave everyone else alone. This isn't a slippery slope, it's just good sense (no more white supremacists, hey!). Humankind will get along just fine if we tack on that one extra rule and all follow it.
Murder now, for some, includes abortion.
Censorship now includes, for some, private companies removing bad actors from their private systems.
Come on, that's lazy to dismiss it that way when society literally changes all the time.
Whether or not abortion is a murder is not about definition of "murder" it is about definition of "human being".
There's no doubt that abortion involves killing a living creature, the whole pro-choice vs pro-life debate is basically about one simple question: "is fetus a human being?". If you answer that with "yes", then every abortion becomes a murder, plain and simple.
This also explains why there will never be a compromise between two crowds: it is logically impossible to compromise on yes/no questions.
The definition is commonly agreed upon, and what "white supremacist" means is not at all controversial to most people. It certainly isn't so arbitrary as to be meaningless.
Now, the term may be misapplied at times, as may any term, but for it to be misapplied, it has to have an accepted application to begin with. A term without a definition can't be subject to definition creep, and the possible creep of a term like "white supremacist" is that wide to begin with.
Yeah, right.
HN stans for white supremacy because it's mostly white and extremely online and never gets out to see the results of "give everyone a soapbox" on the streets where the proud boys are out there beating the shit out of people.
You people are cowards, and nothing more. There's no bravery here, no principled stand, just a bunch of fucking cowards.
You continued to post flamewar comments. We ban accounts that do that repeatedly, so could you please stop? We've already had to ask you more than once before.
https://news.ycombinator.com/newsguidelines.html
And when you have malicious traffic swimming in an anonymous pool, there's no practical alternative but to block all of it.
[0] https://blog.cloudflare.com/cloudflare-supports-privacy-pass...
Another thing that sets it off is virtual machine usage, I can be logged into chrome and gmail on the same residential IP for hours but the moment I try to search google for a problem inside a VM it's a minute of slow loading captchas.
Have moved to bing instead, that sort of wasted productivity is a burden.
I use Firefox with a few basic extensions (Privacy badger, uBlock, Google Container) yet every time I am presented with having to pick out traffic lights over and over and over again. I usually have about 5 or 6 "challenges" before I give up and use another site.
My timezone has not changed, my IP address and rough location has not changed, my screensize has not changed, my broadband speed has not changed, and my general computer dexterity has not changed, yet I am relentlessly targeted. On chrome I never saw these challenges, but on firefox with the privacy plug-ins I am always always always challenged.
At this stage I think the only signal it is using is "is there a google cookie in this browser? and if so has the google cookie got some 'normal' looking activity logged against it?" I.e. they are checking their server-side logs for a given cookie ID and seeing if that looks normal or not (i.e. seen on google search, seen on youtube, seen ads from a variety of third parties on various different sites, mixed up with time of day and speed of viewing etc etc).
Since I have got Google in a container in Firefox, I am guessing that my google cookie is not present when the captcha loads (due to the containers and privacy badger et al) so there is no identity back in the mothership to compare me against.
That's because Google isn't just profiling "Tor users". They're going after anyone who values privacy in any way or technology.
Simply put, you're being punished for ensuring privacy. And anybody who uses Google's captcha services is an accessory to that.
The clever part from Google's perspective is that you have to trade one of these things to Google in order to get access to sites that do not belong to Google at all. Google convinced site owners to have their users pay a tax to Google.
When someone uses a recaptcha, they should think about why they are doing so. It's one thing to use it to save a business model, but it's another to use it to protect information that should be free anyway. The elephant in the room is government data. Many government agencies think that selling their data can be a nice source of side revenue, and a recaptcha is a good way of enforcing it. In reality, they just increase the costs for everyone, and those with means can obtain the information while those without means cannot.
Governments need to release their data, freely, without captchas or fees for single users and bulk users, no exceptions.
Totally agree. Fortunately the Dutch government is trying to make as much data open as they reasonably can, and regularly organise events to encourage developers to use their open APIs.
The counter argument is that they do a great job with trivial stuff like registered dog's names, and less well with sensitive/important issues like policing.
What's the right way to leverage the platform developed for the first into the second?
captcha is google master blow against ad blockers.
a regular user, who they have all the info, give them dollars per ad impression. You, with your doNotTrack (ha! that was a joke) and privacy addons makes them only cents per ad impressions.
you are google's enemy. remember this when you get stuck in captcha hell (and consequently censored from most sites until changing device/ip)
I rarely see the "I am not a robot" box, and hasn't seen image recognition tasks for a long-long time.
If they lose enough customers over this, they will probably remove the captcha.
My VPN must have gotten white listed (or cracked down on some of their traffic patterns) because that stopped.
This does not agree with my experience. I browse without cookies and severely limited javascript (using umatrix), and I also encounter the myriad of ridiculous inconveniences that the OP was referring to. On the good side, however, the web is much faster and generally less annoying.
Isn't this also something that many bots do (don't run javascript and don't have realistic cookies)? It seems like just another instance of reducing your distance from the "bot" cluster in agent-space.
I personally don't care too much about the hassle, but I really don't like the idea that I'm basically playing Artificial "Intelligence"/doing clickworking for the not so community oriented efforts of Google.
That's a massive load of bullshit. Google has a captcha challenge that only humans can solve. That alone is already sufficient to prevent unwanted traffic. That is how every captcha system works. However google is an exception. If you're logged in to a google account or are using chrome then google can use that information to track your captcha history. Privacy minded people avoid google like the plague and therefore they cannot be tracked.
>Google isn't going out of their way to punish you for trying to protect your privacy. Except this is exactly what happens. It's not "unfortunate". It works like this by design.
If google cannot track you then the captcha will force you to do something that no other captcha system does: give you even more challenges even if you have solved them correctly. You will spend the next 5 minutes solving captchas correctly and then at the end it will tell you you've failed. This again is unique to google: correct answers lead to failure. The problem immediately goes away if you let google track you, it doesn't matter how bot infested the network is. No other captcha system does it this way.
Google is clearly doing this to get free labour to label their datasets, force people to have a google account and encourage them to use chrome.
A lot of the behavior that captcha exhibits is in part a function of feature analysis from ML models - features that may seem ridiculous to layman humans but make sense to a neural net plugged into the data.
If you do everything you can to prevent google from knowing who you are, don't be surprised when they behave like they don't know who you are.
I took that to mean they were blocking cookies
It's not bullshit, it just depends whether your website is being targeted directly or not. We're targeted directly and the robots hitting us are getting the CAPTCHAs solved, presumably with human help.
Also, I’m just not interested in the remaining 10% "legit" traffic from people who are aggressively paranoid about their privacy. Almost all of them ended up being dickheads who were using TOR to abuse other members of our community.
To the people who think every website should treat TOR users with respect, please understand that you are intentionally making yourself indistinguishable from the mountain of robotic junk, abuse and human dickheads. It's not my fault that you have chosen to do this, and it's not my job to provide you with tools to prove you're not a dickhead.
...or maybe voting me down will change the facts.
Yeah, that's totally going to work.
It's always annoying though.
The image classifications that you do, however, are used to train the computer vision system.
https://addons.mozilla.org/en-US/firefox/addon/buster-captch...
The difficulty is probably cranked all the way up
A very good addon against the shit from Cloudflare and Google.
if you care about all that, run a node without internet exit, and also strive to make your sites available on tor (hate the "hidden service" nomenclature)
TOR doesn't protect your privacy, it just lumps you in with—and makes you indistinguishable from—the worst crap on the internet. If you don't want to be treated like crap, don't try to blend in with the crap.
The Quora answer is interesting but it's not clear to me whether the "I'm not a robot" box cannot be defeated in a similar manner.
In order to successfully execute a replay attack you would also need to pass the Turing Test, i.e. click the correct images. If you design a bot that starts a combinatoric attacking by trying random guesses we can easily confuse it, so most attackers try to use a solver service.
We can also identify how you interact with semantic content in the images when you click on the image and characterize your mouse interaction as human or non-human. Since confidence increases as more results come in we can also run them after the initial pass and then shadow-ban bots. (And notify the targeted website that we have determined e.g. a particular signup is a bot.)
Ultimately, many techniques beyond simple correct/not-correct are required to defend against the main attack vector: humans hired to solve captchas en-masse and make thousands of fake accounts. Modern ML is pretty effective for these kinds of problems. Browser obfuscation does not add real security, and today's reCAPTCHA (all versions) is easily defeated in practice.
It turns out they record and analyse:
- Your computer’s timezone and time
- Your IP address and rough location
- Your screen size and resolution
- What browser you’re using
- What plugins you’re using
- How long the page took to display
- How many key presses, mouse clicks, and tap/scrolls were made
And ... some other stuff we don’t quite understand.
NoScript (which is totally fine) just blocks all JS, uMatrix can block much more.
Settings -> Privacy and Security -> Content Settings -> Javascript and then change "Allowed" to off.
You can white list a domain by clicking on the padlock or the thing that says "Not Secure" in the URL bar on the left and clicking "Site settings" and changing Javascript to Allowed.
In my opinion, the only real effect JS blocking has is you start complaining in HN comments about sites not working.
Depends on the version of NoScript. If you stick with the older ones before this was broken, you're good. You can't do that with the new Firefox, though.
It's still a pain to get rolling at first but it feels more friendly to me than noscript
1. turn off javascript (effective for all websites)
2. block access to google in your router/firewall. (effective only for google)
[1] https://intoli.com/blog/making-chrome-headless-undetectable/
https://www.youtube.com/watch?v=fsF7enQY8uI
It is describing how the checkbox is collecting your browser's characteristics (eg they go to great length describing the webGL fingerprint) and your own characteristics (eg mouse behavior), such that when you click the box, you are determined to be a person or a bot. If they think you are a person, you don't have to do the CAPTCHA.
The whole bit about a double encrypted "VM" is overstating the case. The "VM" is "just" a bytecode interpreter, which at the end of the day can't do anything the browser's javascript engine can't do itself. Yes, it's some heavy obfuscation, and what's more interesting than the interpreter itself is the decision to spend what must have been lots of time/resources to develop it. It's security by obscurity, and in this case it is delivered to the client so obviously it's reversible. Maybe there's a deeper purpose.
EDIT: ah. the purpose is not to obfuscate. it is to fingerprint the CPU characteristics. by running their own interpreter, and changing the opcodes on the fly and such things, they can defeat JIT and learn something about the CPU itself. if they have user info (google cookie) they can know what CPU/CPUs that user typically uses and if "the checkbox" records something different it's a signal.
I can see it being possible - racks of "robot arms" that move mice based on whatever criteria is needed (in this case, reCaptcha).
It works for 3D printing, as well as device testing - so in theory, this could be done too...
It’s always going to be an arms race I guess.
It exists for a long time already.
Given that I'm not doing anything unusual, it really feels to me like reCAPTCHA, for all its complexity, boils down to "what's your history using Google software? Oh you rarely use it? I'm gonna give you a captcha". It didn't used to be this aggressive, but it's really ramped up in the past few weeks.
That only really leaves blind deaf people out, at which point we might be reaching the limits of any technology to provide access to everyone without a tooooon of work.
https://support.google.com/recaptcha/answer/6175971?hl=en
In a quick search it seems like NoCaptcha is the accessible answer for the issues with regular Captchas. For the most part it seems to work, most of the complaints here seem to stem from people trying to actively block some of the evaluation metrics used by the checkbox (cookies,javascript,user strings,fingerprinting,etc) which makes them look very different from normal traffic which kind of by necessity makes them look a lot more like bots.
https://simplyaccessible.com/article/googles-no-captcha/
But if they are doing so because they are disabled, and the difference means they receive a worse experience, may result in an ADA complaint (especially if a government service falling under section 508 is involved).
> During testing I had to shut it off because it became increasingly more complicated every page reload. First it was just one page of traffic lights, but 20 minutes later I was having to click through 5-6 pages of images. This worries me that user's might get pissed off.
Why are you now badmouthing someone else for deleting Spotify over this exact same issue?
Some financial and government benefit web sites query web trackers as an extra factor in the enrollment process.
Making (online or offline) life more difficult for people who don't want to use company X products could escalate to the point where you either accept the yoke and are admitted to the walled garden of "society" which company X has firmly cemented themselves under -- or you say no and find yourself unable to drive/fly/get a job/go to college/buy groceries in your town. It sounds like a big leap to make right now, but is a real possibility if Amazon/Google/FB don't get broken up soon.
So perhaps it's Safari and/or the ad blocker that are to blame? Hard to say, though.
Possible fingerprinting?
But honestly, I wish it would just die.
Of course, I think the biggest thing is your browser's fingerprint. If you are using a lot of privacy blocking addons, etc. you are going to be spending a lot of time looking at captchas.
Annoying as they are, I don’t really see a better alternative. They’re also pretty easily circumvented with cheap labour like mechanical turk and similar services.
It's almost definitely not my problem, and it's not even necessarily a problem it's just a way to access pages, made easier by Google in fact because it's a perfectly legitimate way to navigate the internet and Google themselves depend on it. If anyone wants to discourage it just provide an official API to use instead of pages, which are a defacto API for humans.
https://github.com/GoogleChrome/puppeteer
https://www.trustedreviews.com/news/apple-hit-two-factor-law...
Phone calls to GF, to get her to login, similar issue, it was now signed out, and after another attempt, one of them showed the 2FA window, which got me in. At no point was I able to use the phone number also associated with my account, and it was a huge, long, pain in the ass. It's actually a huge part of why I won't buy an iPhone.
i'm generally against this type of gating, where the people doing the right thing get punished disproportionately (even small slices of time add up to wasting thousands of human-years over the population) just to combat the tiny number of bad actors. target the bad actors directly.
it's the same for tsa security theater. let's put all those humans to work training dogs of all sorts and filtering them through people at the airport. the money for those privacy invading scanners can be put toward training and housing the dogs. our collective time is not wasted on silliness and standing in line, and we'd probably save a lot of tax dollars that way.
I have also trained myself to wait a few seconds before clicking the box, which seems to help assert my humanity.
I wonder how many different weird rituals are out there for 'beating' CAPTCHA?
For me, I usually make some effort to keep moving my mouse and being "active" after clicking the box, on the idea that an isolated click event looks less human. It's based on a friend's tip and it seems to help, but I have no confidence that it's actually relevant in such a complex system. I sort of suspect Google has created a new generation of meaningless routines fit to rival historical standouts like sports rituals.
On the one hand, fuck Google for wanting another vector with which to track me, on the other hand, why can't it remember that I proved to it that I'm human (allegedly) two days ago?
Maybe to prevent bot operators from manually aquiring a human-cookie then let their bot operate for ever ?
I'm using that captcha on my website, and I can tell you, that one works 100% while the other ones don't.
this is a subtle externality borne from scale combined with zero marginal cost. we need to find a way to make the bad actors bear the cost of that externality rather than the rest of us.
it's a hard problem that resists simple solutions like captchas, certificate signing, delivery fees, taxes, or even just outlawing the practice. as a lowly consumer, one of the few levers i have is curtailing my use of services employing (google) captchas.
Identifying the lights and cars and bicycles is you training Waymo's self-driving AI.
I disable most scripts using uBlock, but I have to make an exception for reCaptcha if I want to interact with half the web. I'm pretty sure due to that alone Google is able to track where I go.
How much, and at what odds?
The length of the average Google CATPCHA has been steadily going up for me. I haven't pulled out a stopwatch, but I do count how many image sets I go through. I basically never get through on the checkbox unless I've done one on another site shortly before. I sometimes succeed after one set, but if I don't it's consistently 3+. The worst case I've seen was 10 layers of slow-loading images without success, at which point I gave up and tried on another device. (If it had been a site I didn't need, I'd have given up after 5 - which I do fairly often, so I don't have an average count needed to succeed!)
I'm fully aware that average users don't have this much trouble, or people would be furious. But I also see that captcha ramps up to an extremely long process in the face of even modest privacy-protection efforts like not running Javascript or allowing third party trackers by default. (God forbid you're using a VPN for any reason.) It's not assessing your humanity but your familiarity, using the same fingerprinting tools as any site that wants to track you.
Spam is a real problem, and a hard one to solve, but I admit I'm hostile to Google's captcha. Partly because it really is a significant time sink for me. Partly because it lacks any progress indicator or fallback option so it's an indefinite hurdle to accessing sites I'm already committed to using. But largely because, despite what I really believe are good intentions, it's yet another force pushing people to give up privacy and even security if they want websites to work tolerably.
Personally, I've reached the point where this is the first thing that I do. If a site is presenting a CAPTCHA (especially one run by Google) to me, then 90% of the time, that's a site I'm better off avoiding anyway.
That last 10% can be infuriating, though, and I certainly won't feel positively about it.
In a parallel universe of fluffy niceness we willingly provide our help and in that way we get all those old books converted to ASCII and available for us to read online. Our efforts are for the good of mankind. Similarly with the newer challenges, we help the maps be up to date and again this is for the good of mankind and those needing help getting around.
Clearly this doesn't work in an era where the 'don't be evil' mantra is long forgotten and people only see Google as some advertiser friendly capitalist monopolist beast.
Google need to work on their relationship with their customers, to be a benevolent dictatorship of sorts. They are lousy at customer service and there are other pain points that they are ignorant to. I don't see how this helps.
Google's customers are those who buy advertising. The rest of us are just cannon fodder.
Seems pretty bilateral.
Have come to the conclusion it really means any patch of grass.
For firefox - https://github.com/dessant/buster
That's a bit of an ambiguous sentence - are you saying they would have little trouble, as in not much trouble, or the opposite?
I'm convinced that part of the reason Google released headless Chrome is as a honeypot for bot authors to use. The idea is that instead of going through the effort of fingerprinting and identifying new bot software, release something that bot authors will use instead that you have a capability to detect.
Somewhere inside of headless Chrome, there's one or more subtle changes that make it so Google can detect whether you're using headless Chrome or normal Chrome. There's no limit to how subtle the indicator could be - maybe headless Chrome renders certain CSS elements slightly slower than normal Chrome, etc.
It sounds pretty crazy/complicated but I could definitely see it being worth it if it means detecting $X,000,000 worth of ad fraud every year
I don't think spambots are currently using Chromium or even running JavaScript. Using simple spamfilters in JavaScript still works fine on my setups.
Login attempts are usually spread over a massive botnet of residential IPs as well, where they'll only use each IP for one or two login attempts before moving on to the next.
It's a very fascinating problem space
I once ran into a piece of code from the scammy advertising world that tried to redirect users to a phishing site. They cleverly tried to hide themselves from the automated quality checks some ad networks do, by checking for these functions and appearing benign if they saw them. One of the checks even created an exception and then inspected the stack trace for certain flags that apparently are only there on some type of headless browser. Clever!
https://addons.mozilla.org/en-US/firefox/addon/buster-captch...
What's so wrong about crawling or using automated tools? With today's networks and hardware performance most websites shouldn't concern themselves with denial-of-service type of attacks, unless they're past a certain threshold of popularity.
I agree with you, for "read only" content, let the robots crawl.
(in reality though reddit has a huge bot / astroturfing problem anyway, so perhaps the value of captchas really is overstated)
Why waste time catering to people who are too lazy to write a decent scraper? Most people would prefer to spend their time working on things that will benefit the other 99% of their users.
Every time I found a way to fingerprint and block them, they'd change their bot to avoid detection. Captcha for rate limiting seemed like the least bad option.
However, eventually I decided that instead of blocking them, I'd return random results from the db. They weren't checking the data too thoroughly because to this day they haven't changed the bot to avoid my most recent pattern detection. They're still merrily scraping useless data.
I would have set it to only show the captcha if the delay was active, so users would effectively have had a choice of wait or captcha.
There's very few friendly bots like archive.org, and those respect meta robots tags anyway.
It’s quite depressing to see that almost all sites now require sending data to Google just to log in to them. Not to mention that they help to turn billions of users into clickworkers for annotating AI data. Tell me what you want but I’m pretty sure they show image captchas to users that they are absolutely sure are no bots. I use Chromium on Linux and I’m logged into a Google account, still I have to solve three or four image captures at times to use a login.