Ask HN: A Good Alternative for ReCaptcha?
Are there any good alternatives for reCaptcha? It has come to a point that traditional text/sound captcha challenges are trivial to bypass today, and the more distorted we make them, the more challenging they are for humans.
Google went Sparta with their reCaptcha be, and nobody in their right mind should add a script that fingerprints users, specially from an adtech company.
What solutions do you use to thwart bad boots from submitting your forms and automating things where it should not have been?
206 comments
[ 93.7 ms ] story [ 938 ms ] threadI have yet to see a general-purpose tool to which you can throw any text captcha and it’ll solve it.
Just because there are academic papers that demonstrated it once doesn’t mean there’s still a huge barrier to entry in implementing this solution (which spammers won’t do as long as it’s easier to move onto another target).
There are paid captcha-solving services out there and even those are still powered by humans even though it’s in their commercial interests to automate the process. Them not doing so further suggests that AI is not there yet.
You can create your own Captcha questions / answers. I feel like this is the preferred way of handling spam posts, creating your own custom Captcha implementation.
Of course, given some bots relay the captchas to humans, this arguably works best for a site with a specific niche in mind, since they can ask a significantly more difficult question that only someone interested in the topic would know.
Like say, how a PhD maths related community or blog might ask a university level maths question, a chess forum may ask a chess related one and one about a certain gaming franchise might ask questions about said franchise. Bonus points for keeping out people with no genuine interest in the topic.
I can imagine at the frontend you have some JavaScript, where an input field gets filled in or something. There has to be some server side checking as well, otherwise a simple HTTP POST would submit fine.
I do like the idea, but if you need JavaScript anyway, why not have some invisible inputs. They work for now.
We were getting a lot of automated requests, and right when we put the waterwall on our page, it did a good job of picking out those users and not impacting others.
After a few days though, those users were able to start getting through again, but based on the timing between requests, it looks like they might have had to start operating the page manually.
But if a bot decides to target you specifically, all of these things are trivial to defeat. So once again they are not useful.
It kinda leads me to conclude that each developer has to create "out of the box" solutions instead of some plug and play solution. If a plug and play solution is developed then all the spam bot creators start figuring out ways to crack or simply create a service for human based cracking. If unconventional methods are used on each site then it gets more complicated for the spammers.
https://www.projecthoneypot.org/
Stop Forum Spam is a similar third party service. You send it an ip address and an email address. It will reply on both items if it is spam, together with a confidence level. Quite interesting way to reply :) It is originally intended to fight registration spam, but you can use it for comment spam or contact forms as well.
JavaScript spamfilters can be very usefull. Most spambots do a HTTP GET for a page with a form. They fill in all the fields and submit it with a HTTP POST. They don't run any JavaScript on that page. You can have honeypot and timeout fields on a form that get manipulated by JavaScript, and spambots will not validate. Works really well, and all transparent to the user. The only "risk" is that in the future spammers might start using more sophisticated spambots, like using Electron or Chromium. I implemented spamfilters like this in a WordPress plugin and it works really well for me: https://wordpress.org/plugins/la-sentinelle-antispam/
You’re also making your website unusable for people with Javascript blocked or disabled in their browsers.
I do understand where you are coming from though. And I also think this alternative is better in this regard. reCAPTCHA loads JavaScript from a third-party domain. With JavaScript spamfilters you are loading them from the first-party domain.
https://www.google.com/recaptcha/api2/demo
Perhaps that's just a cloudflare thing.
If your regular website (not a webapp) doesn't work without JS, then you're failing as a dev.
(For those unfamiliar with algoritms and code as solution, it's a reference to this: https://www.reddit.com/r/ProgrammerHumor/comments/5ylndv/so_... )
I think I would start with these, probably looking into what other people are doing.
It can be quite difficult than one might think. For example, now that we are talking about spam, the word "Viagra" shouldn't block my comment, even though my parent post doesn't mention the word or in a situation where nobody else mentioned it.
[1]: https://www.distilnetworks.com/
Just recently, I added the idea of a captcha that might actually be enjoyable for users to my list of "things that should exist":
http://www.gibney.de/things_that_should_exist
The idea is to show the user a random image and ask what is on it. If the image is beautiful, that might even be fun. And there are many sites that offer beautiful public domain images. And have tags for everything in them.
There probably are many other funny and enjoyable captcha ideas one could implement.
Plus with your use case there may be other criteria, for instance, if you have an 'apply now' job application form you can take in other data such as how long it took for someone to fill in the form and where their IP address is.
If you are hiring for a job in London and you are not likely to hire the office manager from Timbuktu who spends less than ten seconds uploading their CV and writing some cover letter then you can make your backend form processing not forward that email on to the HR department.
Putting a timestamp in the form as a form field with it encoded is easy. On the submit side you can unencode it and come to some judgement on the matter.
There is also the hidden checkbox with 'hideit' set to 1, not sure why that works but it does with a form you have written yourself, i.e. not stock Wordpress.
Although we don't like Google doing their deep-stalking of the visitors, fingerprinting them in re-captcha, there is no harm in collecting a little bit about the user. The user agent, screen size and location is useful in a sales/support perspective. If someone has a posh computer that says something about them. If they are using an old copy of a Microsoft browser then that says something about them.
On a general forum there can be standards of English to enforce. If someone is not using capital letters to start sentences, not using punctuation and not spelling so well then that can be flagged before they hit the 'send' button.
I have done a lot of tidying up of email lists created by bots and what surprises me is how easy it is to spot the fakes. It is like the bad guys in movies and games, doing everything possible to make it easy to get 'em. If spammers did real world robbery they would carry a bag labelled 'swag', be wearing 'Groucho glasses' and a stripey jumper.
Even if a CAPTCHA does offer a non visual alternative, it is very common for it to be inaccessible for folks with cognitive disabilities (eg, dyslexia) or motor impairments. Another common issue is assuming that users all speak English fluently. In this example, "beauty" is likely to be sufficiently culture specific to cause localization challenges.
https://www.w3.org/TR/turingtest/ is a good resource for learning about the accessibility implications of many common types of CAPTCHA implementation.
The gist of it was that in a few years, all Captchas will be useless because machine learning is too easy and cheap. The only way to defeat spam will be to use reCaptcha v3 or something like it, because those services will use what they know about you to determine if you're a bot or not, plus their own machine learning of what "normal" behavior is for your website. It sounds like ReCaptcha v3 is basically an app level IDS.
[0] https://www.npr.org/sections/money/2019/04/24/716854013/epis...
No, no, no. There are many ways to combat spam, and there is no silver bullet. "Determining whether you're a bot or not" is just one tool among many, and one that lets human spammers through, or gets too intrusive and starts blocking humans.
IMO the best approach still is to focus on the content they post rather than trying to figure out who/what they are.
This is inherently user-hostile, as it presupposes tracking and identification. I don't want them to know anything about me!
If your goal is to generate, say, 5,000 spammy backlinks you're going to have the choice of building smarter and smarter bots to bypass CAPTCHAs and filters, or just tossing the same dumb bots at a wider pool of target sites. The latter is always cheaper, if you're focused around your basic blog-spam sort of scenario.
I could see it different if you had a specific high-value service that was worth bot writers targeting-- think of registering email accounts en masse, or an ecommerce site getting thousands of test charges an hour on stolen cards. But even then it's still about just a matter of being "faster than the other guy the lion is chasing" -- you just need to be inconvenient enough that the malicious user finds a more accomodating service. That needs little in the way of an AI arms race, it can be something as simple as rate limiting.
I think that I by this time I have the technology to make something like this work, I was wondering if this is a good solution though. What do you think?
My parents don't even trust Amazon. They use Visa prepaid cards that they fill up at CVS whenever they have to.
Arguably they're doing the right thing and we're all doing the wrong thing. Either way, micropayments are one of those "ideal world" scenarios that are unlikely to transpire anytime soon.
The minor cost may be offset by the fact that getting a post through to a site that has almost no spam because of its filter is valuable in itself. If the bot is sophisticated enough that the stuff it posts is hard to distinguish from a human shill or shitposter it may even prefer sites protected with such a system.
Useful for the vintage VIAGRA HERE link dumpers perhaps though, but those can be filtered out with a content filter.
This approach may turn legitimate people away, namely:
1. People from regions where it is uncommon to have means to interface with payment processors.
2. Minors who, for one reason or another, are not able to obtain a debit card/credit card. Similarly, PayPal refuses minors.
Couldn't gnu taler allow this kind of stuff?
¹ https://news.ycombinator.com/item?id=13829545
Since I'm also generally against javascript, ideally the code would open-source and it could be verified that it doesn't do anything malicious...
Say what you want about the site, but 4chans pass method of payment, which just removes all the captcha's when posting and lowers the post timer, was a fantastic idea, and just works.
Some examples:
> It seems more likely they are trying to steal cc details by asking for such a small payment.
This relates to being afraid using a credit card online against a non credible seller.
> Payment processing is nontrivial to set up
This one is about the technical difficulty (And possibly also regulation) for setting up payment processing
> regions where it is uncommon to have means to interface with payment processors.
This one is about inaccessibility to payment means, which also relates to a problem with the payment medium.
> You could use JavaScript cryptocurrency mining instead
I assume that bo1024 suggested this because he implicitly believes that setting up a payment processor will be more difficult, and give bad experience to the user.
The only objection I have seen which is more specific to actually recognizing humans is this one (by Freak_NL)
> What would prevent the bots from using the same system?
First, I think that it should make operating bots on a wide scale much more expensive. Second, if you make money out of bots coming into your website, would you really care that they are bots? I assume that the reason someone would block bots from the first place is that bots harm their money making business. If you had a system to collect this money back, shouldn't this considered as a problem solved?
I am working these days on a decentralized payment system that supports micro-payments. (I promise there are no blockchains inside). Solving the captcha thing can be a very useful use case. I will be happy if anyone wants to join forces on this one (My email is real (at) freedomlayer (dot) org).
I would really love to see a way to make small payments to sites that I visit simply and all managed in one place.
Be sure to use a separate email and give it to readers on your about page via some language like "questions (dash) and (dash) comments (at) (this domain)".
If it's for account signups just send an email confirmation link and possibly include a code in the email that has to be submitted manually as well.
This trick is simple stupid and should not work but somehow the simple spam bots have not improved.
This does not work for sophisticated bots (never met one) or the ones programmed specifically for your site (happens very rarely).
It works surprisingly well.
Depending on where it is the name= would be surname (where the form submission has a name field rather than a first name surname split), website, url, etc
Do you anticipate any problems with form auto-filling tools?
To be sure, you could add aria-hidden="true", which I'd guess most bots don't recognize.
http://alistapart.com/article/now-you-see-me/
If it doesn't, its a bug in the screen reader.
I wonder if bots are smart enough to figure out aria-hidden="true". Possibly empty it out using javascript on submit. I would guess a bot not using CSS would also not be using javascript? Unsure, would need testing.
Disclaimer: I don't know how a screenreader would present this, example only
"Form entry. Input name. Input email. Ignore this field it's for spambots. Input url. Submit" -- In this case does the message more naturally apply to email or url? I'd imagine there'd be a pause after input email (to wait for the input)? I need to set up a screen reader :)
Admittedly, I am not familiar with screen reader standards, but my gut feeling is that they are doing their users a disservice if they are not representing what browser users are seeing as similarly as possible.
It's not a definitive solution, but it's an easy and practically free first line of defense for a young project, and depending on the project, can stand for years.
Overall, it depends on the sophistication of the bots your project attracts.
https://github.com/markets/invisible_captcha
[1] https://github.com/markets/invisible_captcha/issues/52
[1] https://www.businessnewsdaily.com/10900-ada-website-requirem...
They even have a compliance tester.
To be illegal the website must be run by a business which employs 15 or more full-time employees. Or the business is some form of public accommodation like a hotel. From what I have read.
Of course it would be better to make sure the website is accessible, but I'm mostly commenting on the statement that it is illegal.
https://www.businessnewsdaily.com/10900-ada-website-requirem...
just google relevant keywords https://www.google.com/search?q=accessibility+scan+website
and if you want the full experience: just enable the screen reader and try to use your website with it.
/edit: and i almost forgot: chromes build-in Audit tool in the Developer Panel includes some Accessibility tests as well
Screen readers often cost significant amounts of money, and are not trivial to "just turn on".
* WAVE (Chrome or FF plugin, https://wave.webaim.org/extension/)
* AXE (https://www.deque.com/axe/)
* AChecker (https://achecker.us/checker/index.php)
* Funkify (Chrome plugin, tries to emulate various disabilities)
* Lighthouse in Chrome Dev Tools also checks some accessibility rules
The full list of things that you need to take care of: https://www.w3.org/TR/WCAG21/ (it's huge I know, it takes 5-7 days to test everything from this list)
My team at Microsoft recently open sourced a tool called Accessibility Insights (https://accessibilityinsights.io). The web version is a chromium extension that includes both automated scans and also a guided assessment option that leads you through how to test for and fix the stuff that has to be found manually. This is the tool Microsoft pushes its own teams to use as part of their release processes.
First, can you navigate your entire site without using a mouse (including any widgets, forms, embeded stuff)? You should also have a "Skip to main content" button that is the first element you hit when you tab into your page.
Next, download the NVDA screen reader, which is free, turn off your monitor (or close your eyes), and navigate your site using it. I recommend using FireFox for this.
Finally, use a color contrast analyzer plugin for your browser to ensure you have enough contrast between all of you elements.
From there, you can review the WCAG 2.0 spec to get into the fine details. If you have the budget, hire a consultant/contractor. What I described above doesn't make your site pleasent to use for a disable person, just usable.
An element that is not displayed should not be 'displayed' by screen-readers either.
It's also always place this field after the Submit button with the idea that a user with a screen reader would never make it that far. Bots still see it and add it to the post request since I don't think they care about the order of the form fields.
That said, we encounter many sophisticated bots and also a decent number of what I'm pretty sure are real people in low-wage countries pasting data into forms. That last one is tough.
The basic prerogative of a sophisticated bot is to ensure you believe that.
Having written a variety of sophisticated bots - some from pre-existing libraries, others from scratch; for specific websites and for general purpose - I'm reasonably confident most people who think they've never seen a sophisticated bot are mistaken.
I agree with you that anything more sophisticated or bespoke than a mass-spam bot is rare. But rare things happen often to most websites with nontrivial traffic. The types of bots with the most funding and skill behind them are the ones which don't try to spam anything on a website at all.
So if you have a contact form, this simple method may reduce the spam content to a low enough level that the effort to implement some type of third-party service is not necessary. There is not enough incentive for someone to target the form directly.
In addition, the captcha service may actually deter real submissions, whereas this is completely invisible to non-bots.
If you have a form that has a greater incentive for bots to abuse, then you need something more sophisticated.
Ideally it would have a varying id / name and a varying ARIA attribute for blind users, saying something like "human users, please ignore this".
It would not stop a really sophisticated bot that runs an actual browser and uses machine vision to detect page elements. But unless your site is very high-profile, running such a sophisticated bot to defeat its protections likely won't be profitable.
W3C is requesting feedback for the document, if you'd like to make suggestions, please open an issue: https://github.com/w3c/apa/issues
Captchas work well for telling humans from bots for the purpose of denying automated/scripted access. But here a simple IP-based blacklist works well, because of how many bots now live on Amazon's properties and some such.
You don't need a captcha to filter out bot spam. That's a massive overkill.
stopforumspam.com works well. You can combine it with a simple keyword based filter, have it tag hits with a cookie, temporarily blacklist the IP and then filter them out based on that as well. Auto-submit it to stopforumspam too. Obviously, also have whitelisting in place, e.g. to let through existing customers, previously cleared posters, etc.
For bonus points, first-time posts that look OK may be put into a "shadow ban"-ish mode, whereby they are visible to the posters and mods, but not anyone else. Until they are cleared. This works equally well.
The bottom line is there's no spam that doesn't try to promote something and they aren't likely to target just you, so there's always a keyword/URL you can latch onto, and it also makes sense to participate in a distributed monitoring framework to piggy-back on each other's first hits.
Not all media manipulation is commercial. Not by a long shot.
You're fighting the last war, if not older.
[1] https://www.geetest.com/
It require minimum user interaction, and you will eliminate most of spammers bot, since it will lose its cost-effectiveness. You can implement something like coin-hive proof-of-work, without having to mine monero anyway
I have had zero spam the last 8 years.
The code is ancient and runs on an even older version of lispworks with Auth details hard coded all over the place, so I the time it would take for me to share it would be longer than to rewrite it in some hip language.
Had I been lazy and not as privacy conscious I would have let Gmail do the spam filtering for me.
I'm reasonably sure the answer to your questions is yes.
There is nothing technically novel about it. Heck, python even includes an SMTP server in the stdlib. You could.probably write a PoC in a couple of hours.
Don't be fooled. Text-based CAPTCHAs are still very effective, unless you're a really large target on the scale of facebook or google. If you design your own text-based CAPTCHA, it's highly unlikely that someone is gonna pull out their ML skills to read your CAPTCHAS just to spam. Too much effort. I wrote my own PHP CAPTCHA more than a decade ago and have it used ever since with virtually no modifications, and not a single piece of spam has made it through on my websites (if you don't count my friends trolling me with silly messages once in a while).
I helped vendor-select and lead implementation on a fraud solution that was an integration with SiftScience (yc-funded, https://sift.com/), which relies on fingerprinting. This was years ago but I still think about the project and how it plays with user privacy etc. I will say that -- fingerprinting as a component in fraud management is/can be highly effective.
The problem is, once you get into payments fraud through bots, I think the conversation becomes way more nuanced. If you're looking for a solution to bots spamming or throwing bad data into your app, maybe that's a little extreme. But if the choice between privacy and becoming a front for credit card fraud and chargebacks, you're in a choice between who the victims of your service are going to be, and how much ill is done.