Ask HN: How to stop bots/spammers in their tracks on an entry form?

5 points by jrod2121 ↗ HN
We're trying to come up with a solution to a major spam and bot issue we've been having. Basically, we have an entry form for people to fill out asking for basic information (name, email, address, etc. --sometimes we ask for just email).

Our forms are posted on thousands of websites and domains, and so we've started to see incoming spam and bots executing the forms, submitting fake information.

So far we've employed these things to stop them:

1) IP is blocked after 3 submissions from the same IP.

2) We check the http referrer to ensure its coming from the expected page. If is not, we kill the entry.

3) If an email has already been entered once, the system won't record it again.

4) We've added captcha, but most of our customers don't want to use it since it reduces form submissions.

Our form is submitted via AJAX request. Currently the bots are submitting the request to the AJAX file and the HTTP_REFERRER is coming from the expected page, and no information is coming across that would suggest they are using a proxy.

I could be incorrect, but it appears as though they have a macro set up that is rotating through IPs using a new one each time, and automatically filling in our form fields, and submitting them, exactly as a human would do.

As of late we've seen these bots submitting our forms thousands of times, all using the same type of address (like @yahoo or @hotmail) but the submissions are all coming from different IP addresses, so we can't detect and block them. After entering some of these IP's into an IP checker, it looks like a majority of them are coming from proxy servers.

One idea we've come up with to try and stop the spam is that when a user types into one of the fields, javascript invokes a cookie. When they go to submit the form, if the cookie isn't found, we drop the entry. Would this work?

Any suggestions/ideas on how to best combat this would be greatly appreciated!

9 comments

[ 5.0 ms ] story [ 35.9 ms ] thread
Cookie the user on first visit, and if the cookie is too fresh (e.g. less than 1 minute, then require captcha)
I've had good success with adding a text field to forms that must remain blank. Hide it from humans with CSS and then let the form fail silently if there is content on that field upon submission.
I second this.

I've also noticed that bots always fill fields based upon their name attribute, so call your fields something other than "email" and "username" and reject submissions where fld9624-v2 is not a valid email address.

Isn't this basically the honeypot captcha or invisble captcha method? I don't think it would work as well since we our forms are submitted via ajax request, but I could be mistaken?
It's my understanding that the spammers parse the form's HTML to determine which fields must be submitted and the URL to which they must be POSTed.

Remember, you don't need a perfect solution to this. When you and another person are being chased by a tiger, you don't need to outrun the tiger, just outrun the other guy.

What about something like activating the form with another xhr call when the user does something like presses tab or clicks on a field? It can set a code from the server and then check for it when the form is submitted.

Not foolproof but if you are getting caught in very wide net this might be enough.

EDIT: Missed the part about the macro, this may not help at all.

In my experience, bots don't execute JavaScript (yet).

I add a hidden "spam" field to the form. When the form is submitted, I use Javascript to remove it. If the field is still present when sending to the server, flag the entry as spam. This has blocked all spam bots for us.

In addition to javascript cookies help as well. Many don't handle cookies.
definitely this - I've done this a ton on Drupal/Wordpress also the "confirm I am not a spammer" checkbox does seem to cut down on submissions considerably.

IMO, a losing battle no matter what. A certain amount of spam is always going to get through. The real challenge is to do this without compromising your UX.