Tell HN: My entire company's Gsuite access has been banned
Google support is refusing to let us know what happened and not even giving access to retrieve emails to other drive data.
We are facing completely hopeless situation where no one is providing us any answers or providing support for PAID service.
Can anyone help or provide any clues on how we can talk to actual support team?
147 comments
[ 5.6 ms ] story [ 204 ms ] threadYesterday I hit the "support" button in Fi, did the dog and pony show of reading the "suggested resolutions", for the privilege of being directed to "community support".
When someone first said that to me I was 100% sure they were joking. Then I realized they weren’t joking at all.
Google drive is hard as the only way is to setup continuous Google takeout's or use a third party service for backups which the last time I looked several years ago.
If my comment helped one iota to make backups more likely next time, I did good.
Like does this give you a sense of superiority?
I could have said: Been there bought the T-shirt and I feel your pain. Which would be true. The person would have not felt better in any way because the data is gone, and would not have learned something either.
Does the person feel worse because of my comment? Don't think so. Either he already new and agrees with 'Yes, d* it should have done that, you're right' - what I felt someone said this to me - or he learned something. Perhaps if getting kicked hard enough he will not skip backups next time.
Does he feel bad? Sure.
Does anyone think about the people who he dragged down with him because he didn't have backups which would have been something he is paid for but didn't - do risking the jobs of everyone around him.
When consulting the FIRST thing I ask:
DO YOU HAVE BACKUPS FOR YOUR AWS/GOOGLE/... DATA?
And if not do it now - although the usually answer is: This is the cloud I don't need backups because they backup the data/redundancy/S3/... What about someone deleting it? By accident or itention? What about you getting sued by A/G/MS/...? What if they kick you of like XXX (Parler, ...)?
Most do backups to a another provider the same day.
Not having backups if you're the one responsible is not the same as getting cancer or being hit by a car. It's you've screwed up very badly endangering many other people.
For the rest, you're completely right. Everything possible should've been backed up. It just has nagged me that a real backup isn't possible, and it has bitten me in the past (corrupted drawing, which corrupted related documents as well). Since then, I have only used the Google sheets/documents/etc. for throwaways and drafts.
If you have critical data that can cost your company that you can't backup, your're living a risky life. Hopefully they've checked with their colleagues that those agree with living that risky life.
Being as google less as possible nowadays is just common sense.
I just cannot recommend anyone seriously use GSuite or GCP until Google starts treating customers better.
The situation is getting worse, not better. :(
https://support.google.com/mail/answer/10173182
It's on (and can't be disabled) if you have the Advanced Protection Program enabled.
Started around three months ago.
Hilariously, they warn you when doing the Takeout export that because you have APP enabled, your export may take weeks or even months to be generated.
https://s.sneak.berlin/@sneak/105599881937545540
I don't like that they don't have native IMAP.
https://hackaday.com/2020/10/21/google-meddling-with-urls-in...
FWIW, Microsoft have had a similar feature long before Google:
https://docs.microsoft.com/en-us/microsoft-365/security/offi...
I don't use Google Accounts without it. TOTP codes are vulnerable to phishing, and SIM swapping is very much a thing.
Wow.
Of course, odds are that the company's One Critical Sheet that reimplements SAP in Excel is critically reliant on the 4DMATRIXTRANSFORM.OLDCHURCHSLAVONIC() function that falls into the remaining 1%.
VBA is more than 1%, and Sheets doesn't support it at all.
Wonder when people will realize that they should not rely on Google (or Facebook) (or Apple) for doing business.
You get things like issues with a user not having visiblity of items in a folder, and support ask you to check all folders shared with the user to see how many are affected. There are tens of thousands of these, so this is obviously impractical.
I had an issue with their "eventually consistent" admin interface not showing whether a user was part of a group, and their response was to wait 24 hours to see if they have been sucessfully added to it, or to email the entire group and ask the user to check whether they got the test email.
We had a Google Sheets file that, for whatever reason, was not able to be opened. Months later, the issue is unresolved. If we didn't have daily backups that use the API to make a copy, exporting Google Docs, Sheets and Slides files as MS Office documents, using a thrd-party service, we would have just effectively lost this data.
It is clear that Google does not treat this as a business class product, and it hurts to business users.
It is getting worse and worse IMHO, while the MS competition is getting better and better. It is only a matter of time until we switch I think, and I would strongly caution anyone against adopting it, especially if you plan to use it as the primary file sharing method for more than 10 users. For us, the thing making it hard to move is that our data on Google Drive is a mess. We haven't been using Team Drives, because we have been using Google Drive since before they were a thing, so have complicated nested permissions, meaning we can't simply move things to Team Drives. If you have used it for as long as we have, when there was no way to prevent users from doing this, and didn't have policies about how it was used to prevent this (whiuch we didn't, we started using it before I joined the company) I don't see how you can have avoided this.
I'm not sure how GCP compares, but I have been pretty disappointed by how slow they have been to support recent versions of PostgreSQL and MySQL in their hosted DB service. We use AWS mostly, and, while there are things about GCP that look nice, they seem to fail to do the basics that would not be at all hard for them to do, and it looks like they don't take it seriously as a product either really. It is sad really, given that there really needs to be more competition for AWS and Azure IMHO.
I don't think I'm the only business owner who is wary of using GCP for anything, as it vastly increases the surface area of unintentional or malicious actions that could make our Google Workspace and all our data in it go poof with no recourse.
It may be that I am out of touch with the other HN users on this one, but "$megacorp does something that on purpose/accidentally blocks my service" doesn't surprise me at all. That the megacorp is legally in the right on their own turf is even less unsurprising and not the point. There is a price to not running your own infrastructure and this price becomes tangible when something like this happens for whatever reason.
Do you have sales people? Enable any "mail merge" or "customer contact" apps lately?
Google rarely shuts down Gsuite domains without a good reason. If they really aren't telling you the reason - it is abuse related.
(Former Googler, opinions are my own, etc etc)
It sounds like they've already contacted support. They said "Google support is refusing to let us know what happened" and "no one is providing us any answers or providing support".
> Do you have sales people? Enable any "mail merge" or "customer contact" apps lately?
Cool so Google does not like what I do to my mailbox? I know, some of those apps are crap, but banning an entire org because of that?
Good to know that even paying customers are treated by Google as the product
It's not your mailbox with Google.
I won't get in to specifics, but the first few times the user will get spanked. If you keep reenabling accounts that send spam, eventually the domain as a whole gets spanked.
The purpose of an account lockout is either:
1. Stop a clear abuser from trashing the network, e.g. by sending spam that will get Google IPs blacklisted (yes this happens, there are no special breaks for Google).
2. Challenge an account in such a way that false positives have a chance to 'clear their name' by doing things that serial abusers have difficulty with, like passing phone verification or in the good old days, solving CAPTCHAs.
Virtually all responses to service violations in the consumer world are (2). They're automated, people pass the challenge when they log in and never think about it again so you just don't hear about it.
Corporate services pose special problems. Companies expect to be able to automatically create large numbers of accounts using APIs, which is something not allowed for consumer accounts. The latter are carefully monitored, throttled, and ultimately the services may just refuse to let you create an account. Tough cookies. That doesn't work for GSuite domains where the administration is devolved, so what happens when spammers notice this and create such domains then add lots of users at once? Yes they're meant to pay, but some abusers can get past CC fraud checks and other mechanisms that would otherwise make this approach too expensive.
Unfortunately the rapid, automated nature of abuse means that the response also has to be rapid and automated. It's not as easy as saying, hmm, foobar.com looks suspicious, let's get a trained expert to mount an investigation. By the time the specialist has woken up and got into the office the attack is over already and you sent 10 million spams or hosted 50,000 phishing sites. That's the flip side of very scalable systems with convenient signup, in which the admins don't suffer any consequences of abuse they proxy or allow (i.e. when you buy GSuite part of what you're buying is Google's spam reputation). That's also why these stories so often resolve with an ending like "A googler got in touch and we're back in now". This creates the impression that the only way to get support is to flag it up on Hacker News or Twitter. That's sometimes true for consumer services where getting people to look at your case is harder but for companies with support contracts, it's often just due to the time lag involved between someone doing a manual investigation and resolving it.
This
> there are no special breaks for Google).
and this are not the same thing. There are absolutely "special breaks" for Google, particularly (though not solely) since they are now the server of choice for a huge percentage of the email being sent and received, whether @gmail.com or on Gsuite accounts of various types.
So even if there were some kind of massive spam event, Google isn't going to block emails from its own IP blocks or domains from reaching Gmail or Gsuite customers. And frankly, I would be shocked if any of the various DNS blocklists that people use for determining who's a dirty spammer would actually put a Google IP block on there without massive and obvious changes in Googles mode of operation.
Meanwhile, us mere mortals, if we dare to try to set up our own mail server, can be listed in a blacklist because of a) something the previous owner of the IP address did, b) something someone else in the same class B block did, or c) something that was never expected before, but now because of escalation by spammers, it's required without any documentation. And then the blacklist just doesn't respond to requests for reconsideration or elaboration because obviously, we're dirty spammers, and why should we be listened to?
When I worked there it absolutely did do this. Yes it seemed stupid to me at first too, but all it takes to end up like that is to decide not to give yourself unfair special exceptions to your own policies, which is how things should be!
frankly, I would be shocked if any of the various DNS blocklists that people use for determining who's a dirty spammer would actually put a Google IP block on there
Be shocked. It did happen when a particularly clever spammer got through the controls. There are no bits of logic in most email servers that say "if IP is owned by Google, ignore". Spam filters are all relative. If senders send a lot more good mail than bad, they're good. If they send a lot more bad mail than good, they're bad. Google IPs are rarely blocked because they send a lot of good mail and invest a huge amount of money and time in keeping it that way.
If it is indeed abuse related (whether there was any abuse or Google's algorithm just decided there was some abuse) then telling the abuser what got them banned is counterproductive from Google's PoV. Same reason reddit does vote fudging.
Do schools or parents not teach philosophy or ethics to these FAANG employees?
You haven't been to school this century, have you.
Also at minimum I'd think I'd allow them access to past emails as a default.
https://en.wikipedia.org/wiki/The_Trial
It's amazing that Google doesn't get sued in a case like this, regardless of what's in the ToS.
If one of my employees did something bad, I would want warnings and ultimately a "hey, since your employees cannot behave, we're shutting your account as of <date> and have limited <functions> until then so you can migrate your company to another service.
This sort of binary response is why I will never commit to Google, and am increasingly evaluate the likelihood of ban when selecting cloud services.
Particularly since these bans are immediate, summary, and often based on algos, not humans.
...and this isn't just unfounded fear. We've had our Google My Business listings suspended when we had issues with our 2FA/recovery details. That was 4 months ago, and support also said they could do nothing, and My Business support never responded.
Edited: I have his Google email but I’m not sure if I can post it here. Email me instead at lawgimenez@hey.com
https://lwgmnz.me/2018/08/05/g-suite-horror-story/
The hype by dhh was some next level stuff, but is the service actually good and usable?
At some point, we need to recognize this for what it is - unregulated monopolies having no accountability.
I'm wondering whether there are legal actions that people can take. Many years ago, I was having a problem with Anthem "losing" my insurance claims. Eventually, I got sufficiently frustrated and filed a complaint with the California Attorney General. Shortly after that, I got a long letter of apology, a promise to open an internal investigation, and most importantly, finally got my insurance claims processed.
Insurance companies have unique regulatory requirements specifically because they used to do what Google does now: ghosting paid-up customers. In their case, many insurance companies welcomed regulation to maintain a level playing field. It would otherwise become too easy for some to collect premiums but not pay out obligations, and thereby be able to undercut more honest companies. But for this to work, state Attorneys General need to take enforcement seriously, and resist capture. California's Attorney General is elected, which helps there.
Given all these terms and stipulations, it's easy for the insurance company to say the policy doesn't cover the specific event that led to your loss, even if it should. Which is when you have to take them to court. So in the interest of keeping the insurance business an actual legitimate business people trust, you have oversight.
Insurance payouts delayed are insurance payouts denied.
There are Nash equilibria at many different points on the curve, but only a narrow range where the value proposition of insurance is appealing to potential customers. An Insurance Commission's job is to force it into the range where negligence and arson are not paid, but genuine misfortune is paid immediately.
If an insurance company is regularly stiffing people then it won't last long regardless of whether an insurance commission exists or not.
If you lose your family and house in a fire, with marginal savings, then you may not be able to afford a lawyer and your contract is useless in the truest sense of the word. Unlikely situation? Yes, but then, that's exactly what insurance is all about.
Please make sure to include the following information: - Domain - Your Domain Admin's Email - Any Google contacts that you're supposed to have
I'll do my best to have one of the Workspace agents reach out. I work closely with them, so should be pretty easy. :+1:
Generally, I've seen issue like this where a bad Apps Extension is added or various other external, but generraly preventable things happen.
Maybe someday address scraping bots will learn base64. Maybe it will be at about the same time you learn to read.
I replied to your claim that base64 should be preferred over "foo at bar dot com" since it would be "unlikely to be automatically checked by bots". I find that claim unlikely since a scraper that is adjusted to handle the former will easily be adapted to the latter and likely already is today.
I'm happy to hear a counterargument to that position, but please don't include an insult.
This is the claim that requires backing up.
Since offering emails in base64 form is not usually done, I consider it pretty unlikely that bots have been specially modified to go around checking everything which looks like base64 to check whether them codes have an email inside.
If you don't want people to be insulting, maybe you should think through your arguments more carefully before treating someone else like the idiot.
GSUITE banning us would put us out of business, and I worry about this more than I should have to.
This can literally kill the company outright.
We are a GSUITE company, and these stories give me nightmares. If one of my employees violated the ToS, even repeatedly, I expect to be told "Get off our system. We're giving you x days to move elsewhere, and have restricted <functions> until such time".
An immediate and total shutdown with literally NO response from Support is so drastic that it would trigger a lawsuit if it were any other company.
The only reason it doesn't in these cases is because it's David vs Goliath, and the victim company is already too dead and broke to sue.
Hopefully some lawyers will file a class action against Google - I have a Google My Business account that was suspended without cause (no abuse) with zero response from support.
Which is exactly what Google does. At $dayjob we get warnings from Google all the time about users misbehaving.
Remember you are hearing one side of the story here. Innocent people and the people worthy of banning for legitimate reasons will both proclaim innocence until the end of days.
...because I've also had things banned from Google (not the entire gsuite account thank god), and support has no ability to help. So I'm not at all going to assume that they deserved this ban.
Seriously you see the reflexive responses from in this thread from people who work/worked at google. They refuse to believe that google could have messed up and the OP did something wrong.
Google has a fatal cultural problem. Just like Intel. With the exception that Google has nothing that someone else can't do better.
takeout.google.com should work even when the accounts is banned.
https://www.businessnhmagazine.com/article/soldati-wins-unan...
You might have similar options?