Help HN: Google has blocked our entire domain for harmful programs
start.page is the primary domain for hosting Buffer's link page product. Eg: https://buffer.start.page . About 24 hours ago, a spammer created a start page which linked to a .rar malware file hosted on Google drive. We did not host the file. Just carried a link to it.
That page was detected during our routine content moderation this evening but it had also been reported to Google. We have removed the content at this time and submitted the start.page domain to Google's review process.
In the meantime however, instead of blocking the individual subdomain that had linked to the malware, Google has blocked our entire domain start.page which means that all valid customers are also affected by this. Any customer start page visited on desktop/android now shows the scary red screen warning.
Reaching out on HN right now to see if there's anyone at all on Google who can help expedite the review process so that our customers aren't further affected by this.
Also, if anyone from Google sees this I can further help by sharing information to the linked google drive file. It's password protected so I'm guessing that that helps it bypass detection.
Thanks. Fingers crossed for this since I've never done/had to do this before.
81 comments
[ 2.5 ms ] story [ 161 ms ] threadSee the instructions here: https://support.google.com/webmasters/answer/6347750?hl=en
In our case, Google's search console shows clearly what subdomain was guilty of the issue and that subdomain has been cleared now. Just really want to expedite the review process since Google's safe browsing has decided to block the entire domain instead of the offending subdomain :-/
I've submitted a review but they do say that reviews related to malware take a few days to process. This is a little hard to be honest given that it's not even our site that is hosting the malware. It was a page linking to google drive which is where the malware actually is hosted.
Hoping we get a response soon. Appreciate the supportive chime in.
This does, unfortunately, seem to be the right call. There's no way to differentiate between the subdomain user being malicious, the domain owner being malicious, or the domain owner getting hacked. The only granularity of data available is that something under the start.page domain was distributing malware, so it makes sense to quarantine the whole domain.
I hope this gets resolved quickly! I think the response is likely the correct one though.
If there’s a risk that each time that happens the entire domain could be blocked, that’s a lot of risk to try and mitigate. Especially seeing that many of the bigger providers also struggle to mitigate this kind of content despite having technical teams that are larger by an order of magnitude (or more).
If that rationale were valid, it would apply to everyone, including google.
If the rationale does not apply to google, then it does not apply to anyone else.
Please gooogle (if they let you) the concept "double standard".
Try googling the "Golden Rule" and you'll find a version that says "they who have the gold make the rules".
We can argue about whether this is fair to small players, but it's hardly self-dealing for Google to include themselves in their list of high-profile content hosters, and there are very rational reasons for maintaining such a list.
It's not fair to smaller and newer players, but it's perfectly rational and isn't a uniquely special feature of Google.
Or, you know, take into account the number of subdomains serving malware relative to the total number of subdomain.
1 out of 1000’s seems unfair reason. If it was 10’s or 100’s of subdomains out of 1000, then it makes more sense to punish the whole domain. But when it’s just a few, blocking the individual subdomains would be the better way.
If safe browsing only blocked the subdomain when you have a certain threshold of "safe" subdomains, then attackers would just have a sufficient number of "safe" subdomains.
Also how do you set the threshold? It's dependent on the market that the subdomain hosting provider targets, it's dependent on how good their moderation is, it's dependent on how quickly they get indexed, all sorts.
Any solution needs to work for the case of malicious users, and needs to work at a scale of billions of pages, i.e. you can't use any human review or non-machine-identifiable information.
If a website has a.example.com, b.example.com, foo.example.com, baz.example.com and they serve malware on baz, I’m saying put that subdomain on the bad list. If they serve malware from many subdomains, block the whole domain.
The issue is that Google blocked a whole domain for just one bad subdomain. That seems too strict, and is very sad for all of the users of that domain.
At least when done at the domain level there's a cost involved for getting a new domain, which disincentivises the creation of many malware hosting domains.
Seems unnecessary to add that as not everyone would've been aware of the PSL.
The parent post bringing up PSL was a helpful addition already.
On that note though, I'm perplexed as to how people would manage this kind of thing if using paths instead of subdomains. So instead of <user>.start.page if we used start.page/user. In the latter case, I'm not sure how one would prevent their entire domain from being taken down if malicious users kept linking to malware hosted on file hosting/sharing sites. Is there something similar to the PSL for this?
At its core the issue in my head is user generated content linking out to malicious software being a point of trigger for entire sites being blocked. Does this mean that an entire publication site could be blocked if someone used a comment widget to link out to malware and the site got reported? That seems like an effective DoS mechanism at some point.
I guess what I'm struggling with is why the domain gets blocked instead of the actual url that contains malware (or even the single path that links out to it). Fwiw the google drive link hosting the malware is still active.
For people curious about what the public suffix list is: https://publicsuffix.org/
For example, it will become somewhat inconvenient to share data across your subdomains. Instead of just setting a session cookie with domain=.start.page, you will need to implement a proper single sign-on mechanism. Email might be affected, too, especially when it comes to DKIM and DMARC.
You also need to make sure that your domain is listed in the private section of the PSL. There was a thread a while ago when someone got mistakenly listed in the ICANN TLD section and couldn't get a wildcard certificate for their domain. Let's Encrypt and most other CAs have a policy of rejecting domains like *.co.uk, and may rely on the PSL to tell which is which.
There is also a bit of security risk since browsers use this list to set cookie restrictions. If it were in DNS, which the vast majority of people use unencrypted, an adversary could manipulate responses to either (a) drop the TXT record altogether so the domain is not restricted or (b) craft a response in which the domain disables the behavior.
Many systems - ex: rate limits, malware domain lists - would be very easily and cheaply gambled if domain owners could "disown" subdomains at-will, just with a change in DNS. There's a fairly long review process to get onto the public suffix list for exactly this reason.
There's also the historical aspect, that DNS is a much older technology than the need for the public suffix list. Mozilla at the time couldn't expect that all registries would adopt a new standard quickly or at all. Since there was a need for this information for browser security improvements, the list was born, and gradually become the de-facto standard source of such information.
seems like a great system
Also, this is bound to happen in the future where people are going to link to malware that’s hosted elsewhere.
I’m trying to wrap my head around how to mitigate the risk of the entire site being blocked vs a single subdomain.
Will update here though when it is resolved in case there are any lessons to be shared with folks
In this case it would be tricky since the link is to google drive and we can’t block those. Also we’ve seen people work around url blocks by either using short links or by using html pages hosted for free to redirect using JS instead of an HTTP redirect. Always another mole to whack :,)
Maybe allow your customer to use their own domain? Sites that host user generated contents such as wiki, blog or personal page often offer this option.
We had that. The site got hacked and was hosting a trojan distribution point. Very discreetly. Once removed, we requested re-evaluation via the Google Webmaster's console and the flag was removed.
I understand maybe wanting "total control" over your own devices, but to just completely reject security warnings seems like a net negative overall.
"It would be too slow (and privacy-invasive) to contact a trusted server every time the browser wants to establish a connection with a web server. Instead, Firefox downloads a list of bad URLs every 30 minutes from the server (browser.safebrowsing.provider.google.updateURL) and does a lookup against its local database before displaying a page to the user."
It has some utility, and some costs: traffic noise, some privacy leak, system complexity, centralization.
On balance it's not for me so I disabled it.
By the same standard of guilty until proven innocent, should they block the Google Drive domain, and warn all users that Google Drive is unsafe/malicious, during the same review period?
"Here is a test suite that can show if your AV/whatever detection tool works"
Google: "Kill it with fire, I get to choose what people see on the internet"
Blocking THOSE two domains would likely resolve half the malware issues on the web, create a temporary flurry of confusion, and then accidentally solve the other half as people are forced to understand what saving files to a cloud actually entails.
How does github pages or neocities, for example, handle this kind of thing? Surely I can't bring down every github page by linking to a malicious Google drive file from my own page?
Sure, I understand how github.com is protected here, but I'm also unaware of github pages being totally blacklisted by Google even though I'm certain there have been malicious pages hosted there.
You can only limit the blast radius so much if the product itself is "hosting user content."
Where we, and in this case, I, missed a very important step by the looks of it is in adding ourselves to the public suffix list. I have done research on subomdain content management but for whatever reason, today is the first day I've come across the PSL. Something I definitely take responsibility for and makes me wonder what other stuff I might be missing that is obvious to other folks who've done this at scale.
-- Game over.
https://news.ycombinator.com/item?id=25802366
- Outlook365 blocking any emails containing our domain
- ISPs blocking our domain via DNS filtering
In each case the blocklisting process was far from transparent and mitigation was difficult and stressful.
If you're in the same boat, reach out to me (email in profile). I believe we can make this topic a little less scary by connecting and sharing learnings.
I really appreciate the community here sharing thoughts, similar experiences, and ideas on what to do. First time I've heard of the public suffix list for this.
A quick question to anyone who happens upon this: How does one prevent this issue affecting an entire site in general? Is there a grace period that Google gives a verified (via search console) site with a security issue? If not, then I'm curious how to protect a site which is targeted by malicious groups via comment widgets or if they host content using paths instead of subdomains. Eg: medium.com uses paths to go to user generated content. How would they defend from having their entire domain blocked if someone created a publication that linked out to malware?
Cheers all!