Spamhaus Nightmare: Domain Shut Down, No Notice, Over A Million Pages Down
Our company provides tools to help people put together pages for their businesses. Our free tool has been used to create over million page tabs on Facebook. Unfortunately but predictably, sometimes bad people use our app. Like spammers.
Overnight, our domain was blacklisted by Spamhaus because one of our pages contained spam. (Anybody want a free iPad?)
We run our infrastructure on Heroku, and use Bluehost for domain names. Well, as soon as Bluehost recieved notice from Spamhaus, they shut off the DNS for our domain. All million plus pages, gone in the blink of a DNS propagation.
Thankfully we were able to switch over to [appname].heroku.com for now and most of the pages are back, but we have paying customers who are in the dark because they rely on our custom domain name.
Our product, that over a million people rely on, suddenly ceased to exist. No advance notice. Nothing we could have done to stop it. Because of ONE bad apple.
This kind of thing will happen in SOPA world, if we let ourselves get there. But instead of being able to call my registrar and yell at them, I would have had to call the government, and oh-by-the-way they might fine or imprison me for having hosted spam.
Let me end with a practical, really-important-to-me-right-now question: is there any possible way to not get randomly nuked by Spamhaus?
98 comments
[ 2.8 ms ] story [ 157 ms ] threadAlso: you've emailed your customers the new address yes? Even if it is only temporary? Maybe buy a new domain and point them towards that:"Please use [FINGSOPA].com while we get everything back to normal."
I guess the first step is to set up better monitoring services to prevent your system from being abused by even one bad apple. Try to catch the abuse as quick as possible so you won't raise red flags.
Additionally you should possibly work on segmenting out your customers. If your paying customers are important to you, use a different system for them. If this has the possibility of happening again you don't want to hurt those customers from a similar thing happening again.
Not pretty, but it would help ensure against it.
I'd love to, but when you have the volume we do there are going to be false negatives. Bad apples will slip through. And if somebody slips through, we're vulnerable for getting blacklisted.
> If your paying customers are important to you, use a different system for them.
This is a very good plan. I'm definitely looking into that...
This doesn't mean you have to auto-ban people - but you could easily setup listings that you can quickly glance at to see what your monitoring found. If you are picking up too many false positives, then you can refine your monitoring. Yes - you can't prevent all of them, but once your system starts getting abused you have to assume that others will do it as well. Additionally, you could very well be losing money in service costs due to these people (I don't know your business model so it's just an assumption) - you want to protect that as well.
The internet is full of user-generated-content sites and the core objection to SOPA is that we cannot police every posting. We do not currently have a legal obligation to do anything other than respond to complaints.
Further, as a veteran of the hosting industry, I'm really disappointed in BlueHost for taking action against a paying customer's domain name. Be sure to read the SLA and TOS when you sign up for services.
As for the disappointment at BlueHost - they should have probably let the customer know before taking action - but other than that I think they did the right thing. As a veteran of the hosting industry, if one of your customers wordpress blog was hacked and hosting a phishing site, would you not disable the site and let the customer know right away that they need to clear things up? That's just a random example, but any sane company will protect their servers via a TOS - if they didn't I would be quite concerned about the service they are offering. Just my $0.02 on this..
There is nothing intrinsically bad about DNS that it needs to be turned off; the OP has already said they were using different hosting.
BlueHost was in no way vulnerable, and in no way needed to protect itself, as the only traffic was DNS requests. GoDaddy tries to pull the same thing with disabling DNS[1].
I feel like we need a Chris Crocker video about DNS systems this month.
1: http://en.wikipedia.org/wiki/Go_Daddy#Suspension_of_Seclists...
If its mission critical for your business, then you can't afford to think like a victim. Take charge of your infrastructure where you have to. Relying on third parties is lean, but not always effective - a small amount of fat in the right areas can give you a lot of flexibility (and insurance) that you might not get when you rely on a third party.
By the way, do you know off the top of your hand any such registrars with more complimentary policies?
I work for a registrar, so here are my biased recommendations - me (hover.com) or EasyDNS. But don't take my word for it - do your own homework. Its worth knowing what your risks are and no amount of free advice on HN can replace that.
Boycott Bluehost!
SOPA's censorship problem is not the explicit endorsement of censorship but the precedent of mechanisms and principles that will make censorship far easier to implement and "justify".
If SOPA passes, and this happened, we would be dead. Users won't wait around for months while the courts slog through a case, they'll just move to the next app.
I also feel that maybe real-world examples of servers and domains being seized by the government, justified or not, might be more appropriate. Especially when the domains and hardware seized were due to potential copyright infringements and not overly-stringent spam rules.
http://www.wired.com/threatlevel/2011/12/wyden-domain-seizur...
http://torrentfreak.com/feds-return-seized-domain-111208/
http://www.wired.com/threatlevel/2011/07/domain-seizures-def...
http://www.electronista.com/articles/11/11/25/doj.ice.seize....
http://news.softpedia.com/news/Spamhaus-We-Blocked-Google-Do...
There's absolutely no reason to be giving second chances to online services with so much competition about, on what is, essentially, a commodity.
As any email administrator will tell you, "no". The best you can do is take measures to prevent abuse coming from your domain name/IP, but bad things still do happen. You are still at the mercy of spamhaus (and other rbl providers).
It could be your registrar is just running an automated process based upon that.
The only people who don't like Spamhaus, in my view, are those ISPs who were happy to make money from selling connectivity to spammers while pretending in public that they hated spam. Them, and people who don't understand what Spamhaus do, like the author of this article, and who think Spamhaus are to blame for their troubles.
As one who has worked in the trenches as a mail admin (small potatoes, granted: a few small clients and a couple of small hosting companies), my observation has been than customers bitch way more about the MX servers which reject mail from our servers than the amount of spam in their own in-boxes. They don't give a shit that the recipient is rejecting legitimate mail -- they blame us for their problems. All because some asshat with a copy of TheBat! signed up and managed to send out a couple hundred "Russian bride" spams before we were alerted and nuked the account. I could probably fund a semester of college for some random kid with the time I've been paid to waste on de-listing and convincing idiot admins that one of their customers really wants to get mail from one of mine.
Sure, 99.9% of email hitting the typical in-bound relay is spam, but CPU, RAM, and disk I/O are cheap. Do per-inbox statistical filtering and let the user decide what spam is. Better yet, let client-side filters do the work. Do you think any person would stand to allow a US Postal carrier decide what was junk mail and then not deliver it? People just need to buck up and put in a little of their own effort.
I haven't used an RBL (even if its just one in a battery of weighted tests, such as with Spam Assassin) due to my loathing for the vigilante nature of the RBL scene as a whole. If you operate an RBL -- fuck you. If you are an admin that rejects mail based solely on being listed in RBLs, then fuck you, too. I know I sound like an asshole myself here, but the existence of RBLs has caused me and various mail end-users way more pain than any spammer has.
Bitter? Nah.
As a mail admin, I want to throw SMTP out the window. It wasn't spammers that killed the protocol, but rather the growth of use of RBLs.
Rant aside, I do have a question to contribute to the discussion: Has one of the larger RBLs ever listed one of the huge mail providers (Gmail, MSN/Hotmail, Yahoo?) for any length of time? I know I've gotten spams and scams from all three.
Funny that. The RBL providers found out about the spam and tracked it down long before you did.
> I could probably fund a semester of college for some random kid with the time I've been paid to waste on de-listing and convincing idiot admins that one of their customers really wants to get mail from one of mine.
Or you could hire a competent mail administrator and not have to deal with these issues at all. One that knows how to keep his outbound mail clean, one that makes sure he knows about spam in his outbound connection before the recipient does, and one that makes sure that his upstream is a reputable and trustworthy provider and not some fly-by-night cheap hoster who happily sells space to spammers.
> Sure, 99.9% of email hitting the typical in-bound relay is spam, but CPU, RAM, and disk I/O are cheap. Do per-inbox statistical filtering and let the user decide what spam is. Better yet, let client-side filters do the work
CPU, RAM and disk are not that cheap. But why should ISPs have to pay for a vast panoply of hardware to handle the garbage being poured out by disreputable providers elsewhere? If an ISP allows spam on their connection, they get cut off, period. Their connectivity to the internet, the commodity they sell, is dependant upon the goodwill and trust of their neighbours. If they abuse it, they lose that commodity. As for per-inbox statistical filtering, why bother, when running an RBL like Spamhaus removes the crap and, more importantly, forces the hosters to do their job or lose their connectivity.
> If you operate an RBL -- fuck you. If you are an admin that rejects mail based solely on being listed in RBLs, then fuck you, too.
I did run them when I was a mail administrator - my users demanded that I do something about spam, so I did. RBLs work, and I wholeheartedly support Spamhaus in their efforts, and wish that more SMTP providers would use them. As for your attitude, that merely demonstrates that you are a two-bit amateur.
> As a mail admin, I want to throw SMTP out the window. It wasn't spammers that killed the protocol, but rather the growth of use of RBLs.
Don't call yourself a mail admin while at the same time spouting asinine garbage that demonstrates that you know nothing whatsoever about being a mail administrator. RBLs never did harm to email - they were a solution invented to try to save SMTP from the spam deluge. Not only did they stem the tide of spam, they also provided lots of evidence which could be used to name and shame the worst spam hosts and provide evidence to legal authorities to take action against the criminals running the spam operations. No amount of squandering money on client-side filtering or statistical analysis is going to do that. And that is why no competent mail administrator rejects the use of RBLs, since they know that they are the only solution that makes any actual difference.
Get the list (like the level1) of "evil" corporations/governments ip ranges and show a picture of a pink elephant to them instead of your real content.
Because of major internet infrastructure run at whim by 3rd party blacklists, you mean.
is there any possible way to not get randomly nuked by Spamhaus?
Spamhaus and every service like them.
It's true of a lot of the guys running blacklists. And more generally, of a lot of people in the position of police. You tend to become a mirror of whatever bad guys you're fighting. Your tactics have to match theirs, and pretty soon your principles start to as well. I suspect this tendency is so universal that you have to make a conscious effort to avoid it.
Your registrar pulling your domain for this is ridiculous. I would switch ASAP to someone who cares more about their clients.
This, however, is just a symptom of a bigger issue: DNS is fundamentally broken. We need a scalable, open-source, free alternative solution for SSL and DNS that does not rely on any central authority. Namecoin seems cool and it'd be sweet if people started using that.
The other idea is to have a new "anti censorship" root zone, and mirror all COM/NET/ORG etc TLDs. We could pass around this info and in the event of mass censorship, people could migrate onto the new root servers.
We're putting way too much power in the hands of ICANN / Verisign / any random registrar or host with our current system.
I pay for DNS and hosting - it's 'just another bill'
Firstly, Paul's grand "A Plan for Spam" method of using Bayesian filters to stop all spam ("I think it's possible to stop spam, and that content-based filters are the way to do it."). Uh, so, how'd that work out? Spammers quickly figured out how to make a mockery of Bayes based solutions. And who is still out there filtering spam using IP addresses & domain names? Spamhaus.
Then, what really got his goat was back in 2005 (yes, long grudge holding, one wonders what he feels about the mail-carrier who lost a letter of his back in '78 ;-) when his vanity site, shared-IP-hosted at Viaweb which had become Yahoo! Stores was blocklisted at Spamhaus. Back then, Yahoo, and Yahoo Stores were a spammer-hosting cesspool and Paul's page was wallowing in the center of it. Rather than get to the bottom of it, Paul just got on a high-horse and ranted about the evils of Spamhaus. A good take on the rant can be read here: http://www.circleid.com/posts/we_hate_spam_except_of_course_...
So, multiple biases. How often people forget to mention those when they post attacks. Now one must ask, who is the "bad guy" and is "corrupt" here?
But the Spamhaus people should be happy with the irony in Paul's hypocrisy. How so? Well, his paulgraham.com's email is filtered by Spamhaus, as is his ycombinator.com's email. As are the emails of most of the social/blog sites he's on (posterous.com, etc.) One wonder how many of these still use "A Plan for Spam"? Okay, that was rhetorical.
Lastly, the pop-psychology in his posting attests that Paul's degrees are in philosophy, not psychology.
I do agree that the rant is quite unnecessary.
For what it's worth, the SpamBayes plugin for MS Outlook has reliably trapped at least 99.9% of spam for me for several years now, with essentially no false positives at all (where false positives are defined as legitimate mail that bypasses the Unsure folder by receiving a spam score greater than 90%.)
In practice, this is enough to keep my email account essentially spam-free despite the arrival of over 1,000 spam messages per day.
On the other hand, blacklists accomplish nothing beyond interfering with my own legitimate outgoing email, just because somebody else with a Comcast account happens to be infected by a spam-spewing trojan. Gee, thanks, guys.
As a result, I see blacklists the way some people see unions -- as defensive tools that may have been needed at one time, but that are now just unnecessary, parasitic middlemen.
That's a very attacking statement to call people "bad guys". Not just "they're not doing a good job", but actually "bad people". Ouch.
Worse, I think it's totally wrong.
While I do agree with the problem of "power corrupts", I believe that Spamhaus have been highly successful at avoiding that.
I've dealt with and spoken with people at Spamhaus regularly, and they're smart people fighting the good fight. They know what they're doing, and take their responsibilities seriously. They don't blindly attack people or use threats or accuse them of misdeads. They gather evidence about bad behaviour and act on it.
The proof is in the pudding. 100's of RBLs have come and gone over the years, either run as temporary projects that the owner gave up on one day, or gone the "power corrupts" option and just ended up listing so many IPs they've generated too many false positives (e.g. SPEWS).
Spamhaus RBLs are still being run today, and being done so very successfully. Virtually every small/medium email server I know uses them in one form or another (blocking or scoring). They generally have very high block rates, and very low false positive rates.
From a time when Al Iverson was still keeping stats on the various RBLs out there, you can see the Spamhaus zen RBL on it's own generally caught 75% of spam with 0% false positives.
http://web.archive.org/web/20080703181952/http://stats.dnsbl...
Certainly some listings are controversial (eg google docs), but it's always been for a good reason, and forced the provider of the service to come to terms with the fact their spam policies were lax or their service was being seriously abused by spammers. They were thus forced to take action, something they should have been doing anyway.
Without Spamhaus, the internet would be a way worse place, with way more spam/junk emails and websites.
I think that's an incredibly harsh accusation for people that are doing an awful lot of work collecting evidence and fighting real spammers on the internet (http://www.spamhaus.org/rokso/index.lasso), and again, I totally disagree with you.
> though you have to admit that running a blacklist might tend to attract a certain type of person
I think you could tar so many people in so many industries with broad brush stroke sterotypes like that, it seems an unhelpful generalisation to make.
From the article you link:
> As of this writing, any filter relying on the SBL is now marking email with the url "paulgraham.com" as spam.
The SBL is an IP based RBL, nothing to do with domains, so the above statement is patently false. And if anyone was doing IP lookups of URI's in emails and using the SBL for that (which I've never even heard of), that's clearly a misuse of the SBL anyway, because that's not what the SBL is supposed to be used for.
As the policy clearly says:
--- http://www.spamhaus.org/sbl/policy.html
The Spamhaus Block List ("SBL") Advisory is a database of IP addresses which do not meet Spamhaus's policy for acceptance of inbound email and therefore from which Spamhaus does not recommend the acceptance of electronic mail. ---
So it should only be used to block machines sending email, nothing about the content thereof.
There's RHSBLs (like SURBL and URIBL) that are related to dealing with URI's in emails, that's nothing to do with IP RBLs like SBL.
> Why? Because the guys at the SBL want to pressure Yahoo, where paulgraham.com is hosted, to delete the site of a company they believe is spamming
What's that got to do with the SBL again? The SBL is purely about what IP addresses "from which Spamhaus does not recommend the acceptance of electronic mail", nothing about websites. So that whole accusation feels wrong. Mixing up email sending servers and websites, domains and IPs, and absolutely no evidence for it at all.
You seem naive about the nature of evil if you think that it somehow precludes doing constructive work. Bad people don't wake up every morning thinking "what evil shall I do today?" What distinguishes them is that they cross lines other people won't. But the situations that test them may come up fairly infrequently.
Putting this into context, this wouldn't have been the first step. This type of measure was typically implemented after it becomes increasingly clear that Yahoo would not, or could not, adopt measures to reduce the amount of spam coming from their mail servers.
One of Yahoo's general weaknesses is that it takes over 24 hours from sending a complaint until appropriate action is taken (that's why comments on their main sites - e.g. News - contains oodles of spam, and other types of abusive comments). On the typical life-cycle of email spam this is far too long - if a site is ever closed at that point, and so these abusive sites tend to still be up when the email recipient is clicking on those links. Closing a site after the damage has been done is just a never-ending game of whack-a-mole.
Blackholing bigger and bigger chunks of Yahoo Stores is then an escalating step until either Yahoo addresses the spam situation appropriately, or their customers see that Yahoo cannot sustainably provide the service customers are paying for and they either leave or seek legal remedies. At that point innocent customers are paying the price for living in a bad neighbourhood. The question is, why didn't Yahoo do a better job in controlling the level of abuse through Yahoo stores? That Spamhaus felt it necessary to escalate through to blocking chunks of ip addresses indicates Yahoo Stores fell significantly short of what was needed to reduce the spam coming from their servers. The indicative belief from the anti-spam community at that point is: it's mostly clear that the revenue generated from hosting spammers is more important to Yahoo Stores than being able to provide their innocent customers with the level of service they paid for.
From my perspective, SpamHaus were one of the cleaner, more diplomatic black lists around the time of the Yahoo Stores problem. It's been a few years since I last poked around in the anti-spam community. Last I've seen of Spamhaus they didn't defend a legal challenge in California raised by a confirmed spammer, because California doesn't have jurisdiction over UK-located organisations, and so the spammer got a default ruling in his favour ( http://www.theregister.co.uk/2007/03/23/e360insight_lawsuit/ , http://www.spamhaus.org/organization/statement.lasso?ref=3 ).
Spamhaus are not the bad guys in any way, it seems like this was a false positive, Spamhaus didn't know who op was (that he was an ISP hosting other's content), and his host apparently suck a bag of dicks and he needs to switch asap.
Put the blame where it belongs, man.
Spamhas should be used as part of a body of evidence, like in spamassassin scores.
Spamhaus are not the villains here. First of all, you make the absurd complaint that Spamhaus "blacklisted" your domain. That is a lie. Spamhaus runs an SMTP blacklist of ip addresses that some other SMTP providers use, not all. There is no way for Spamhaus to blacklist anyone's domain.
So what actually happened? Spamhaus detected a spammer website hosted on your company's ip addresses, and they did the responsible thing. They reported the spam website to the ISP hosting it.
As for your claim that Bluehost shut off the DNS, why aren't you ringing up Bluehost to demand that they restore it? You might find that a better use of your time than making these absurd allegations and trying to win sympathy by making comparisons to SOPA where none exist.
And it's worth bearing in mind that all DNS providers, along with all hosting providers, are private companies who can cut you off for any reason, from non-payment of bills to the owner of the company disliking your politics. Talking of "due process" in the matter of private contracts between private individuals is misleading and wrong.
Also, Spamhaus makes recommendations. Third parties use their lists to filter spam. It sounds unusual for a Spamhaus listing to result in a domain name shutdown, unless the DNS provider did that based on a listing. So this is not really Spamhaus' mistake (if indeed their evidence listing shows a history of hosting spamvertised websites - then there is no mistake on the listing. You could be listed either because your site/host/network has a solid history of not dealing with spam/abuse reports quickly, or because a big spam operator has landed on using your services. Are you sure it was just one site (and just advertising a free ipad)?)
Yes, I understand you run a facebook static html tab content site. But that isn't a million miles away from bog standard cheap/free hosting solutions that form the bulk of spamvertised websites. Might be worth investing some time looking at the parallels and how good cheap webhosts approach dealing with spamvertised websites and spammers.
So I'd suggest finding the evidence file, dealing with the problem(s) listed, then contacting Spamhaus with details of what you've done, and what's in place to reduce future abusive activity (if it's more than one site offering a free ipad). Then do something about your web hosting solution - that seems like a very weak link - either build up a better relationship with them, or move.
<body> <form name="redirect_form" action="https://statichtmlapp.heroku.com/tab/1/show method="post"> <input type='hidden' name='signed_request' value='fJSfey7ELpgNY4r3gZFT5DyXp0MoW4TF2DsNQWwcoTY.eyJhbGdvcml0aG0iOiJITUFDLVNIQTI1NiIsImlzc3VlZF9hdCI6MTMyNTM1ODI1NywicGFnZSI6eyJpZCI6IjI4MjczNjc1ODQxMjg4MCIsImxpa2VkIjpmYWxzZSwiYWRtaW4iOmZhbHNlfSwidXNlciI6eyJjb3VudHJ5IjoidXMiLCJsb2NhbGUiOiJlbl9VUyIsImFnZSI6eyJtaW4iOjIxfX19' ></input> </form>
https://www.facebook.com/FreeiPAdd2
action="https://statichtmlapp.heroku.com/tab/1/show method="post"
value='fJSfey7ELpgNY4r3gZFT5DyXp0MoW4TF2DsNQWwcoTY.eyJhbGdvcml0aG0iOiJITUFDLVNIQTI1NiIsImlzc3VlZF9hdCI6MTMyNTM1ODI1NywicGFnZSI6eyJpZCI6IjI4MjczNjc1ODQxMjg4MCIsImxpa2VkIjpmYWxzZSwiYWRtaW4iOmZhbHNlfSwidXNlciI6eyJjb3VudHJ5IjoidXMiLCJsb2NhbGUiOiJlbl9VUyIsImFnZSI6eyJtaW4iOjIxfX19'