I’d say that really depends on what you need. If you want a full platform with a lot of integrated services there’s really only GCP, AWS, Azure, Tencent, AliBaba and maybe something else I forgot. So I wouldn’t say it’s a ton of better things that are available. Sure if you only need a bunch of VMs you have a lot of options.
I had AliCloud experience similar to OP but they gave me 24 hour deadline.
"We have temporarily closed your Alibaba Cloud account due to suspicious activity. Please provide the following information within [24 hours] by email to compliance_support@aliyun.com in order to reopen your account:
...
If you fail to provide this information your account will be permanently closed, and we may take other appropriate and lawful measurers. Best regards, Alibaba Cloud Customer Service Center
"
I provided the documents in ~30 hours because that's when I saw the email. There was no further communication from Alibaba. I assumed everything is ok, but in 2 weeks my account was terminated.
* When you have invoicing setup, the above shouldn't happen. You need to keep a payment method in good standing, but you have something like 10 days to pay your bill. -- They do a little bit more vetting (KYC) on the invoice path, and that effectively gets you out of dodge.
* Without paying for premium support, there's effectively no support.
I think if someone didn't pay their bill on time, you might shut off their service too, wouldn't you?
"Oh hey, it looks like $customer suddenly started a bunch of coinminers on their account at 10x their usual usage rate. Perfectly fine. Let them rack up a months billing in a weekend; why not?"
A hypothetical but not unheard of scenario in which immediate shutdown might be warranted.
It's a rough world and different providers have optimised for different threat models. AWS wants to keep customers hooked; GCP wants to prevent abuse, Digital Ocean wants to show it's as capable as anyone else.
If you can afford it, you build resilient multicloud infrastructure. If you can't yet do that; at the very least ensure that you have off-site backups of critical data. Cloud providers are not magic; they can fail in bizarre ways that are difficult to remedy. If you value your company you will ensure that your eggs are replicated to more than one basket and you will test your failover operations regularly. Having every deploy include failing over from one provider to another may or may not fit your comfort level; but it can be done.
> A hypothetical but not unheard of scenario in which immediate shutdown might be warranted.
Not without warning, no. It is possible that the customer intended to start a CPU-intensive process and fully intended to pay for it.
Send a warning first with a specific description of the "suspicious activity" and give the customer a chance to do something about it. Don't just pull the plug with no warning.
There's a degree of complexity that comes with multi-cloud that's ill-suited for most early stage companies. Especially in the age of "serverless" that has folks thinking they don't need people to worry about infrastructure.
My point is that the calculus has more to it than just money. The prudent response, of course, is to do as you described. Have a plan for your provider to go away.
Offsite backups and the necessary config management to bring up similar infra in another region/provider is likely sufficient for most.
For an early startup, though, I would think it's not necessary to be "fully" multi-cloud.
Rather, it would likely be enough to have a cloud-agnostic infrastructure with replication to a warm (or even mostly-cold to save on cost) standby at the alternate provider with a manual failover mechanism.
Most folks overestimate their need for availability and lack a willingness to accept risk. There are distinct benefits that come with avoiding "HA" setups. Namely simplicity and speed.
> Most folks overestimate their need for availability and lack a willingness to accept risk.
I disagree. More specifically, I think, instead, many [1] folks just don't make that assessment/estimate in the first place.
They just follow what they perceive to be industry best practices. In many ways, this is more about social proof than a cargo cult, even though the results can resemble the latter, such as elsewhere in this thread with a comment complaining they had a "resilient" setup in a single cloud that was shut down by the provider.
> There are distinct benefits that come with avoiding "HA" setups. Namely simplicity and speed.
Indeed, and, perhaps more importantly, being possible at all, given time ("speed") and money ("if you can afford it").
The same could be said of "scalability" setups, which can overlap in functionality (though I would argue that in cases of overlap the dual functionality makes the cost more likely to be worth it).
None of this is to say, though that "HA" is synonymous with "business continuity". It's much like the conceptual difference between RAID and backups, and even that's not always well understood.
[1] I won't go so far as to say "most" because that would be a made up statistic on my part
Agreed for the most part. Availability for very many is a binary operation. They either do none of it or all of it.
A clever man once said, "you own your availability".
An exercise in BC planning can really pay off. If infra is code, and it and the data are backed up reasonably well, then a good MTTR can obviate the need for a lot of HA complexity.
> Availability for very many is a binary operation. They either do none of it or all of it.
I assume I'm missing some meaning here, particularly since the premise of much of the discussion in the thread is that there can be high availability at one layer, but it can rendered irrelevant by a SPoF at another (especially when the "layer" is the provider of all of ones infrastructure).
Do you consider that a version of "none"? Or are you pointing out that, despite the middle ground under discussion, the "binary" approach is more common, if not more sensible?
The binary approach is that it either isn't considered or people opt in for all of it without consideration for what is actually needed. The Google SRE book goes into this at length. For each service, they define SLOs and make a considered decision about how to meet them.
Oh, so what you're saying is that they're no considering the notion that there may be a medium-availability (for lack of a better term) solution, which could be perfectly adequate/appropriate?
> There's a degree of complexity that comes with multi-cloud that's ill-suited for most early stage companies. Especially in the age of "serverless" that has folks thinking they don't need people to worry about infrastructure.
Perhaps we'll start seeing a new crop of post-mortems from the "fail fast" type of startups failing due to cloud over-dependency issues. They're (presumably rare) edge cases, but easily fatal to an early enough startup.
> There's a degree of complexity that comes with multi-cloud that's ill-suited for most early stage companies. Especially in the age of "serverless" that has folks thinking they don't need people to worry about infrastructure.
I just heard a dozen founders sit up and think "Market Opportunity" in glowing letters.
CockroachDB has a strong offering.
But multi-cloud need not be complicated in implementation.
A few ansible scripts and some fancy footwork with static filesystem synchronization and you too can be moving services from place to place with a clear chain of data custody.
Everything I have runs in kubernetes. The only difficulty I have to deal with is figuring out how to deploy a kubernetes cluster in each provider.
From there, I write a single piece of orchestration that will drop my app stack in any cloud provider. I'm using a custom piece of software and event-driving automation to handle the creation and migration of services.
Migrating data across providers is hard as kubernetes doesn't have snapshots yet.
There are already a lot of startups in this space doing exactly the kind of thing that I just described. Most aim to provide a CD platform for k8s.
Yes, there is or they wouldn't turn it off. Companies aren't in the habit of trying not to take your money for services without a pretty damn good reason.
And if it was that critical it should have support and a SLA contract, and you know, backups.
Right. Because big companies never ever do anything unjustified. Particularly when they put automatic processes in place with no humans in the loop, because we all know that computers never make mistakes.
Whether AWS or Google Cloud, if you’re running a real business with real downtime costs, you need to pay for enterprise support. You’ll get much faster response times so that you can actually meet your SLA targets, and you’ll get a wealth of information and resources from live human beings that can help even into the design phases.
Feel free to budget that into your choices on where to host, but getting into any arrangement where you rely on whatever free tier of support is nonsensical once you’re making any kind of money.
Paying for Enterprise Support simply so that they don’t fuck you over sounds a lot like hidden costs to me. The service should be enterprise-grade even if you don’t.
If you run an enterprise, pay for enterprise support. If you aren’t (r&d accounts, startups that aren’t monetized yet, etc.) then don’t pay for it. Flexibility is the model here, and it’s utilized by the big players, so being naive to the business model will burn you on many public clouds.
Totally agree. There’s so many missing redundancies on this project. CTO / Dir. Eng should have their own card. They should have some contact with a Google account rep, and at least the basic support package.
We’re not a big Google customer, couple thousand a month, but when we migrated we instantly reached out to account reps and have regular quarterly check ins.
This has nothing to do with enterprise support, which is terrible and everything to do with the fact that Google has automated as much decision making as possible and is terrible at interacting with customers.
I'm not sure if it's the culture of secrecy or sheer cluelessness, but it's pretty bad.
I'm not trying to slam them either, I still really like their products and use them.
Wow as a CTO this is a nightmare scenario. Is this common? I guess this means google.com does not use Google Cloud because I’m sure they have uptime targets. They cannot handle incidents like this and expect people to take them seriously as a cloud provider.
I think the reference is to this book, "Site Reliability Engineering" [1]. As you will see on the main page [2], it is part of Google's effort to describe "How Google Runs Production Systems", which is basically what the parent comment was asking about.
I wonder how prevalent this behavior is. Mozilla behaves the same towards browser extensions, which put business depends on. They removed our extension multiple times, each time before asking for something different, be it uncompressed source code, instructions for how to build it, a second privacy policy separate from our sites policy and more. Each time we would have happily responded to a request promptly, but instead you find out when you’ve been shut down already.
Grace periods that respect your business should be a standard that all service providers hold themselves to
It sounds to me like Mozilla identified your extension as potentially malicious and prioritizing user safety, shut you down first.
As far as I know, Mozilla has no business relationship with extension developers, so I would actually be very concerned if their first action wasn't to cut you off.
I can confirm Mozilla handles this very poorly. I had the exact same experience with them. It was so bad that I actually just left the extension off their store and now focus on Chrome.
There is nothing dodgy about the extension. Mozilla was just being ridiculous.
Nah, not that keen personally (I don't even use Chrome). I was just pointing out that it would have been useful to have the URL to reduce confusion. :)
Browser extensions that say they help with comparison shopping are a very common type of "Potentially Unwanted Application" (PUA - aka malware with a legal team). The infamous Superfish is an example of this type of thing, and there are many others.
I don't know anything about your business or the extension, I'm just pointing out that you're in a space that makes you suspicious by association.
Fair enough. But this has nothing to do with Mozilla's actions. It was as GP said. It includes things like their incompetence in dealing with a build process that creates transpiled/minified code. Even when I gave them all the source and the build instructions (npm run build) they still couldn't comprehend what was going on. Yes, I know it's strange since Mozilla makes a browser with a JavaScript engine.
Edit: I should add that after 2 weeks of back and forth emails the dude was finally able to build it then blamed me for not mentioning he needed to run "npm run build", even though I did mention it AND it's in package.json AND it's mentioned in the (very short and concise) readme.txt.
So after this exasperating experience he just took down the extension without warning and said it's because it contains Google Analytics.
I would have happily removed Google Analytics from the extension. The dude had my source for 2 weeks and could have told me about that at any time, but decided to tell me after 2 weeks of mucking around, after he had already removed the extension.
It was me that decided it was not worth the hassle to have the extension on their store. I just left it off.
So piggybacking on this, I have a similar story to tell. We had a nice young startup, infra entirely built out on Google Cloud. Nicely, resiliently built, good solid stuff. Because of a keyword monitor picked up by their auto-moderation bot our entire project was shut down immediately, wasn't able to bring it up for several hours, thank god we hadn't gone live yet as we were then told by support that because of the grey area of our tech, they couldn't guarantee this wouldn't keep happening. And in fact told us straight out that it would and we should move.
So maybe think about which hosting provider to go with, don't get me wrong I like their tech. But their moderation does need a more human element, to be frank all their products do. Simply ceding control to algorithmic judgement just won't work in the short term if ever at all.
The point is that anyone could fall into that category when laws change.
Imagine you're running a cosplay community, and all of a sudden all your content is being deleted because the SESTA/FOSTA bill gets passed in a country where your "cloud" happens to reside in: https://hardware.slashdot.org/story/18/03/25/0614209/sex-wor...
Projects in the context of GCP can encompass all the necessary infrastructure to build a highly available service using standard practices. There's no indication anywhere from GCP themselves that a project could be a domain of failure. If asked, I doubt they would consider it as such.
A prudent person might consider a cloud provider to be a domain of failure and choose a multi-cloud option, which would probably be the correct way to address this resiliency issue. However, that's not really an appropriate approach for an early stage startup, where availability is generally not that much of a concern.
In other words: It wasn't resiliently built stuff.
Is an exploding car safe because it is built by an early stage startup?
Just because you decide that implementing resiliency isn't a good business decision for some early stage startup, doesn't magically make the product resilient, it just isn't and that may be OK.
There are many options to choose from for implementing resiliency, it could be having multiple providers concurrently, it could be having a plan for restoring service with a different provider in case one provider fails, it could be by setting up a contract with a sufficiently solvent provider that they pay for your damages if they fail to implement the resiliency that you need, whatever. But if you fail to consider an obvious failure mode of a central component of your system in your planning, then you are obviously not building a resilient system.
Edit: One more thing:
> There's no indication anywhere from GCP themselves that a project could be a domain of failure. If asked, I doubt they would consider it as such.
Then you are asking wrong, which still is your failure if you are responsible for designing a resilient system.
If you ask them "Is a complete project expected to fail at once?", of course they will say "no".
That's why you ask them "Will you pay me 10 million bucks if my complete project goes offline with less than one month advance warning?", and you can be sure you will get the response to the problem that you are actually trying to solve.
> A prudent person might consider a cloud provider to be a domain of failure and choose a multi-cloud option, which would probably be the correct way to address this resiliency issue. However, that's not really an appropriate approach for an early stage startup, where availability is generally not that much of a concern.
If you replace "multi-cloud" with "multi-datacenter" (in the pre-cloud days), this premise is fairly unassailable. In those same days, applying it to "multi-ISP", it becomes more arguable.
Today, though, the incremental cost (money and cognitive) of the multi-cloud solution, even for an early startup, doesn't seem like it would be high enough to make the notion downright inappropriate to consider.
I'd even argue that if a cloud provider makes the lock-in so attractive or multi-cloud so difficult that that's a sign not to depend on those exclusive services.
The economics don’t work out if you are trying to do this with just vanilla VMs across AWS, GCP and Azure and managing yourself. You either do it the old fashioned way renting rack space and putting your own kit in, or you make full use of the managed services at which point - by design - you are locked in.
I’m starting to favour buying physical rack space again and running everything 2005 style with a light weight ansible layer. As long as your workload is predictable, the lock in, unpredictability, navigation through the maze of billing, weird rules and what-the-fuckism you have to deal with on a daily basis is merely trading one vendor specific hell for another. Your knowledge isn’t transferable between cloud vendors either so I’d rather have a hell I'm totally in control of and of which the knowledge has some retention value and will move around vendors no problems. You can also span vendors then thus avoiding the whole all eggs in one basket problem.
More importantly, there is a wealth of competent labor in the relatively stable area of maintaining physical servers (both on the hardware and software side). The modern cloud services move fast and break things, leading to a general shortage of resources and competent people. As a business, even if slightly more expensive initially, it makes more sense to start lower and work up to the cloud services as the need presents itself.
Hybrid is what you are looking for. Have a rack or two for your core and rent everything else from multiple cloud vendors, integrated with whatever orchestration you are running on your own racks (K8s? DC/OS? Ansible?).
Still works out cheaper for workloads than AWS does even factoring staff in at this point.
AWS always turns into cost and administrative chaos as well unless it is tightly controlled which in itself is costly and difficult the moment you have more than one actor. GCP probably the same but I have no experience with that. Very much more difficult to do this when you have physical constraints.
Two man startup, perhaps but I think the transition should go:
VPS (linode etc) for MVP, colo half rack, active/active racks two sites then scale out however your workload requires.
Have you actually done this, or are you repeating stuff off the website? Because everyone I've talked with about kubernetes federation says it's really not ready for production use.
The approach we have taken is to create independent clusters with a common LoadBalancer.
Basically, the LB decides which kubernetes cluster will serve your request and once you're in a k8s cluster, you stay there.
You don't have the control-plane that the federation provides and a bit of overhead managing clusters independently, but we have automated the majority of the process. On the other hand, debugging is way easier and we don't suffer from weird latencies between clusters (weird because sometimes a request will go to a different cluster without any apparent reason <-- I'm sure there's one, but none that you could see/expect, hence debugging).
My people's time is more important than your complex system.
You can but that’s another costly layer of complexity and distribution to worry about.
One of the failure modes I see a lot is failing to factor in latency in distributed systems. Mainly because most systems don’t benefit at all from distribution and do benefit from simplification.
The assumption on here is that a product is going to service GitHub or stackoverflow class loads at least, but literally most aren’t. Even high profile sites and web applications I have worked on tend to run on much smaller workloads than people expect. Latency optimisation by flattening distribution and consolidating has higher benefits than adopting fleet management in the mid term of a product.
Kubernetes is one of those things you pick when you need it not before you need it. And then only if you can afford to burn time and money on it with a guaranteed ROI.
Sure. The idea is that you get the benefits of public cloud and cost savings of BYO hardware for extra capacity at lower cost. Of course, you're now absorbing hardware maintenance costs as well. I haven't seen a cost breakdown really making a strong case one way or the other, but my company is doing it anyway.
That's exactly what we are thinking too. We've looked HARD into AWS/GCP/Azure, but for all the reasons you mentioned we don't want to go that route. Owning the entire stack is so much cheaper, both money and time wise.
I can tell a similar story with Amazon MWS, where even if we had access to "human support", it felt like talking to some bad ML, not understanding what we were saying. Ultimately that start up was disbanded, never violating any rule they had, but flagged because of a false positive, and we couldn't even prove we didn't violate anything because we didn't even go live yet. It felt Kafkaesque, punishing one of a myriad possible intents due to malfunctioning ML, with no recourse.
Maybe support just needed to satisfy their quota of kicked out companies for the month, who knows?
Is it only me or does it seem if you are not a "famous" person that has a lot of public visibility and is able to create pressure through a tweet or blog post you are lost, no number to call, no mail to write. Over the years I saw a lot of similar stories, youtube or in general "google accounts" blocked for no clear reason and no way to contact somebody to solve the issue... kinda scary...
I've gotta whole heartily disagree. I've never encountered this on GCE. I run a DevOps consulting company and for standard EC2/machines I much prefer GCP. It's not even close. AWS for the most part lacks any or little user experience testing on UI's and developer interfaces. AWS region specific resources are a nightmare, billing on GCP with sustained use and custom machine types is vastly superior. Disks are much easier to grok, no provisioned IOPS, EBS optimized, enhanced networking hoopla.
By chance are you located out of the United States? These are not downtime issues, but anti-fraud prevention and finance issues.
I've noticed that over the last few years it's become increasingly difficult to do things with US based services (especially banking) if you are outside of the US. And this goes double if you are a US citizen with no ties to the States other than citizenship. Americans as a general rule have never been terribly adept at anything international; banking, languages, or even base geography. We have offices in Cambodia and Laos and I have been told by more than one US based-service/company that Laos is not a real country. I suppose they think the .la domain stands for Los Angeles :) We are looking to set up an office in Hong Kong or Singapore and use that to deal with Western countries. But we're a small not-for-profit operation and HK and Singapore are EXPENSIVE.
stopped yes, deleted the project if the photo id of the credit card account holder cannot be reached in 3 days might be an over-reaction though.
I hope there is a possibility to put a backup contact person / credit card so organisations can deal with people going on vacation or being sick or whatever.
IMHO this should be nicely documented as any other technical material you get to learn about the cloud product when you create an account (e.g. important steps to ensure your account remains open even in case of important security breaches, yadda yadda it's possible we'll need a way to prove that you are you yadda yadda, this can happen when yadda yadda, be prepared, do yadda yadda)
I agree that it seems like an over-reaction. But on an account with intense usage, a single credit card on file, no backup, and a fraud warning it does seem very suspicious.
AFAIK, Google Cloud credit card payments are processed through Google Pay, which supports multiple credit cards, debit cards, bank accounts, etc.
Ideally, in this case the company shouldn't be using the CFOs credit card, but entered into a payments agreement with Google, receiving POs, invoices and so on, including a credit line.
Never set up a crucial service like you'd set up a consumer service.
yes that's a very good description of the best practices that sadly many companies are not really following.
In many situations the "right thing" must be explained, otherwise when people fail to get it they can argue that wasn't really the right thing after all (sure that's ultimately because they just want to deflect the blame from themselves; so don't let them! clearly explain the assumptions under which anti-fraud measures are operating so people cannot claim they didn't know)
No truly production and especially revenue critical dependency should go on the card. Have your lawyer/licensing person sign agreement with them with actual sla and customer support. If it’s not worth your time you shouldn’t complain when you loose it.
That's a great point. These cloud hosting companies don't make this a natural evolution though, because there's no human to talk to, you start tiny and increase your usage over time. But every company depending on something and paying serious money should have a specific agreement. I wonder if this could still happen though, even if you have a separate contract.
There's a mismatch between how much you spend and how much business value is there. The spend for management systems of physical infrastructure like wind turbines is tiny relative to revenue compared to the typical pure software company, especially freemium or ad-driven stuff where revenue-to-compute ratio is very low. Calibrating for this wouldn't really be in Google's DNA.
> These cloud hosting companies don't make this a natural evolution though, because there's no human to talk to
This is not true at all. Once you start spending real money on GCP or AWS, they will reach out to you. You will probably sign a support contract and have an account manager at that point. Or you might go with enterprise support where you have dedicated technical assets within the company that can help with case escalation, architecture review, billing optimization, etc.
It makes sense that would happen. So they just didn't have the contact info for the people here? Maybe they just were spending a little, but their whole business still depended on it.
Amen to that. Once you reach 1,000 USD monthly you can switch to regular invoiced account (subjected to verification) and you have dedicated account manager.
> What if the card holder is on leave and is unreachable for three days? We would have lost everything — years of work — millions of dollars in lost revenue.
Indeed, presumably they were then also at the mercy of the credit card company cancelling or declining the card at the critical billing renewal moment.
Usually there are a bunch of comments about power plants being connected to the internet. I doubt the connections from the control rooms or cloud back to the machines are read only unless they have protocol level filters to remove write commands from the wan to plant networks.
Just the way it is unless it is nuclear plant probably.
If maximum uptime is critical to the business, your infrastructure should be cross-provider.
I've been running three providers as peers (DO, Linode, Vultr) as a one-man shop for years, and I sleep better at night knowing that no one intern can fatfinger code that takes me offline.
Surely! Our business is mainly an API that B-to-B customers consume. The strategy is to create identical "pods" in different cities across different providers on identical distros. Between Vultr, Linode, and DO, that's 20+ cities you could place a pod in the States alone. Each pod has a proxy up front, a database slave, and a pair of app server and cache machines.
Ignoring tweaks for international customers, each proxy is in an A record round-robin with health checks via Route 53. US-based requests get forwarded to one of the pods, and the proxy either handles the request with local servers, or points to servers in another pod if it has servers that are down. If any pod has a power outage or goes down for any reason, Route 53 automatically pulls the entire pod out of the rotation. If an entire provider goes dark, all of the pods get pulled out of the rotation, but all the others keep running.
This is very cool. Where can I learn more about stuff like this, and what are the prerequisites for learning something like this? I have a BSc in CS and understand OSes and programming languages pretty well.
At that Point wouldn't hosting it yourself running on something like vmware vsphere be a simpler option? At least you would have a nice hardware abstraction and a consistent api to build your tooling on.
I hear you. For me, the abstraction is the Linux distro. Build scripts abstract out creating a clean, secure box before installing any custom software, so regardless of the provider, every machine is exactly the same.
you really can't compare switching between generic vps providers from switching between the big cloud providers that provide lots of more useful services.
It is a very different model, but my take is that you shouldn't be building anything on one provider that you couldn't easily move over to another. Provider lock-in is scary.
Very true! I have a script that automatically commits any DNS changes to source control, so if Route 53 bites it I could quickly move somewhere else, but you're right, on the off chance my registrar decides to vanish, there'll be some panicking.
As someone who is currently struggling with Google Cloud's mediocre support, this is not surprising. We pay lots of money for support and have multiple points of contact but all tickets are routed through front-line support who have no context and completely isolate you from what's going on. For highly technical users the worst support is to get fed through the standard playbook ("have you tried turning it off and on again?") when you're dealing with an outage. Especially since the best case is your support person playing go-between with the many, siloed teams trying to troubleshoot an issue while they apparently try to pass the buck.
Not to mention the lack of visibility in changes - it seems like everything is constantly running at multiple versions that can change suddenly with no notice, and if that breaks your use case they don't really seem to know or care. It feels like there's miles of difference between the SRE book and how their cloud teams operate in practice.
I’ve seen other companies walk away from Google Cloud for similar reasons. Automate everything to scale doesn’t work for the Fortune 500. They should absolutely own this market.
This is why AWS and Azure continue to gain market share in cloud, while Google remains relativity stagnant, despite (in many cases) superior technology.
Their sales staff is arrogant and has no idea how to sell into F500 type companies.
Source: 10+ meetings, with different clients, I attended where the Google sales pitch was basically "we are smarter than you, and you will succumb". The Borg approach. Someone needs to revamp the G sales and support approach if they want to grow in the cloud space.
Yes, this is our experience as well, and the root cause of their many problems with GCP. Tech is nice but matters little if the account team just ignores us.
Yes that's exactly what I'm talking about they are super arrogant and unwilling to discuss things at a practical level.
And I've seen it cause them to lose at least 10 potentially good sales.
They have advantages but they're so arrogant that it puts people off.
It's more than 10 times or more people told me they prefer Google's solution to Microsoft or Amazon's but they're going with a competitor because they can't stand Google's arrogant attitude. It's close to laughable because of throwing money away just because they won't back off.
Even for small businesses their sales is pretty bad. I once got a package in the mail from them with a URL containing a tracking code printed on it to contact them that was so obviously Google being Google and treating people as part of a funnel. There was no phone number to be found and nothing personalized.
The other funny thing is the package had a neoprene sleeve for a Chromebook. Eventually a sales person reached out via email assuming I owned a Chromebook and acted like I owed them a phone call because they gave me a neoprene sleeve I couldn’t use.
The entire package ended up going in the trash, which was an unfortunate waste of unrecyclable materials.
If you filled in a form at the link provided from one of the bits of paper in the box they would have sent you a Chromebook for the sleeve. I'e got one here gathering dust. My boss threw away the same package but I was curious and looked through it carefully.
That's interesting. I was of the understanding that everything at Google office tries to de-stress you/undistract you. I thought that would result in people being calmer/ more empathetic.
Reminds me of a thread I saw on the Google Inbox mobile app a while back. Brilliant app, but no 'unread message counter'. There was a huge number of people on the thread begging for that feature and going so far as to say that it was the one thing that prevented them from using the app. Their thinking was apparently that you should have filters for everything and it all should've fallen neatly into little boxes, but for people that have been using email 10 times longer than those developers have been out of college, that's not very practical. One G dev chimed in and said 'But that's now how I use email' and closed off the discussion.
It blows my mind that GCloud, with arguably superior tech and performance compared to AWS/Azure, can't handle support. I have my own horror stories from 2 years ago, but still they haven't fixed it.
Google just doesn't seem to be able to focus on products that requires service and customer support. Maybe they just don't care about it while they have an infinite revenue stream from search and advertising. Whatever it is, they should be humiliated.
I love the tech, and the ux details like in browser SSH (AWS hasn't improved UX EVER) but they can't get support right? Amazing.
Yeah, Cloud9 is billed as an IDE, but it's really more useful as a terminal inside your cloud environment that happens to have a text editor. Workspaces has been great for a cloud-based development environment, and the new Linux Workspaces will be more useful than the Web-based "cloud IDEs".
They are very different things: Workspaces runs a full desktop environment (Windows or Linux) on an EC2 instance, and enables you to remotely access it through client software. The client software uses Teradici PCoIP, rather than VNC or RDP, and Teradici is amazing: it is so fast that the desktop feels like it is running on your local computer.
This means that you can run whatever development tools that you want on the EC2 instance, rather than the very limited code editor that Cloud9 provides. You can easily run a full copy of Visual Studio on a Workspace, and get the full resources of an EC2 instance with SSD drivess.
That's the meme, but my experience with the business support for G Suite does match it at all: I can easily call the phone support, get a competent human quickly, and they are very helpful.
In my limited experience, Google has worse support than facebook (when it comes to advertising agencies). They simply don't care because you are a tiny multimillion euros company and they are THE GOOGLE.
I didn't write that article, but last week I came to the same conclusion and began my migration from GCP to AWS. I admire Google's tech but Cloud Platform lacks fit and finish. It's not fully productized. It's not even fully documented. (Is it indolence or arrogance to publish a link to the source code as the only explanation of
an important API?) I'm sorry, Google, you ignored me when I was crushing on you. Now I have Amazon.
I think they still are mainly focused on their ad business as the core of the company and cloud is something they 'do on the side'. For Microsoft, Azure is core business, it's the future of the company. If they fuck it up, they're dead. Google apparently doesn't see their cloud offering as their core business and therefore doesn't get the attention it needs.
I'd just like to take this opportunity to praise Vultr. I've been using them for years and their support has always been good, and contrary to every other growing company, has been getting better over time.
I had an issue with my servers 2 days ago and I got a reply to my ticket within 1 minute. Follow-up replies were also very fast.
The person I was talking to was a system administrator who understood what I was talking about and could actually solve problems on the spot. He is actually the same person who answered my support requests last year. I don't know if that's a happy accident or if they try to keep the same support staff answering for the same clients. He was answering my requests consistently for 2 days this time.
I am not a big budget customer. AWS and GCP wouldn't think anything of me.
Thank you Vultr for supporting your product properly. And thanks Eric. You are very helpful!
Google Cloud provides more than just VMs and Containers. It has a bunch of services backed in, from a variety of databases such as Firebase (that have powerful built in subscription and eventing systems) to fully baked in Auth, (Google will even handle doing two factor for you!) to assisting with certain types of machine learning.
Vultr looks like they provide more traditional services with a few extra niceties on top.
Within Google's infrastructure, I can deploy a new HTTPS REST endpoint with a .js file and 1 console command.
Could I set up an ecosystem on a Vultr VM to do the same? Sure, it isn't magic. But GCP's entire value prop is that they've already done a lot of that work for you, and as someone running a startup, I was able to go from "I have this idea" to "I have this REST endpoint" in couple of days, without worrying about managing infrastructure.
That said, articles like this always worry me. I've never seen an article that says "Wow Google's support team really helped out!"
Thanks for the info. You may be right about the downvote reason (though it's a pretty ridiculous reason), but I don't think that matters since they are in the same industry providing a similar service and there's no reason why GCP can't provide the same service as Vultr, especially since they charge a lot more for their instances than Vultr does.
Well, I don't really care about the precious internet points disappearing. I'd much rather hear from someone what their reasoning is since I might actually learn something.
But it is telling that there have been at least 5 downvotes but no one is willing to comment as to why.
Edit: since I see you have a downvote (surprise, surprise) I'll clarify that it wasn't me.
Using such proprietary features sounds like a great way to subject yourself to vendor lock in and leave you vulnerable to your cloud provider's every whim. I understand that using ready made features is alluring, but at what point are you too dependent on somebody else? All these cloud services reminds me a bit of left-pad, how many external dependencies can you afford? Maybe I'm too suspicious and cynical, but then I read articles like these from time to time...
You're not alone. I worry the same about many things, but everyone just thinks I'm a negative nancy for discounting THIS AWESOME SERVICE with these awesome, 100% non evil people behind it!
The difference, IMO, is that you're generally leveraging the cloud providers platform in addition to using their hosting.
There are ways to make the hosting relatively agnostic, but choosing a pub/sub solution (for example), that operates at 'web scale' will have a distinct impact on your solutions and force you into their ecosystem to maximize value. Why bother with BigCorps UltraResistant services if you're only going to use some small percentage of the capabilities?
I've made systems that abstract away the difference entirely, but I think the 'goldilocks zone' is abstracted core domain logic that will run on anything, and then going whole-hog on each individual environment. Accept that "cloud" is vendor lockin, and mitigate that threat at the level of deployment (multi-cloud, multi-stack), rather than just the application.
Please don't break the site guidelines by going on about downvotes. That's a strict reduction in signal/noise ratio, which mars your otherwise fine comment. We're trying for the opposite here.
Downvotes can irritate, but there are at least two balms that don't involve adding noise to the threads. One is to remember that people sometimes simply misclick. The other is that unfairly downvoted comments mostly end up getting corrective upvotes from fair-minded community members (as happened with yours here).
It costs a pretty penny, but I’m very happy with AWS enterprise support. When we had a ticket that we didn’t escalate get a crappy answer, our TAM escalated on his own initiative to get us a better answer.
AWS sends you emails 9 months in advance of needing to restart individual EC2 instances (with calm, helpful reminders all the way through). IME, they're also really good about pro-active customer outreach and meaningful product newsletters... Even for tiny installations (ie less than $10K yearly).
Anecdotally: I've been an MS gold partner in a bunch of different contexts for years. The experiences I had as 'small fish' techie with AWS were on par or better. YMMV, of course, but I'd be more comfortable putting my Enterprise in the hands of AWS support than MS's (despite MS being really good in that space).
This is fatal. I have a small pilot project on Google Cloud. Considering putting up a much larger system. Not now.
The costs of Google may be comparable or lower than other services, but they don't seem to get that risk is a cost. Risk can be your biggest cost. And they've amplified that risk unnecessarily and shifted it to the customer. Fatal, as I said.
Making a decision purely based upon some posts on HN and the original artical isn’t a good idea either as there is little data on how often this happens and how often (and pulling the plug could happen with another IAAS). You need to weigh up your options for risk management based upon how critical your project is, the amount of time/money you have to solve the issues.
You might never see this happen to your GCP account in it’s lifetime.
I lost Gmail account (locked due phone verification)[lost on holiday sim card] all my photo from vacation important email is gone (done backup day before) and that was like 3 years ego still waiting for explanation from Google 20 email send no response
I can't speak to the specific incident. We've been running almost 400 servers (instances and k8s cluster nodes) for over a year on GCP and we've been quite happy with the performance and reliability, as well as the support response when we have needed it. I did want to address this comment...
> What if the card holder is on leave and is unreachable for three days? We would have lost everything — years of work — millions of dollars in lost revenue.
You should never be in this position. If this were to happen to us we would be able to create a new project with a different payment instrument, and provision it from the ground up with terraform, puppet and helm scripts. The only thing we would have to fix up manually are some DNS records and we could probably have everything back up in a few hours. Eventually when we have moved all of our services to k8s I would expect to be able to do this even on a different cloud provider if that were necessary.
Restarting the service from scratch is one thing, but what about all your data? Some of these services have 100's T's of data hanging of them and if Google would delete that because of some perceived violation of their terms then that is not something you can recover from in a couple of hours, if at all.
This is one of the reasons I always implore people to have a backup of their data with another provider or at least under a different account. That protects against all kinds of accidents but also against malice.
Backup is a thing. If your company is making millions of dollars off your business you should have a redundant backup of everything including (especially) your data.
Why does it matter whether you're making millions of dollars? If you have any information which you would like to not lose for any reason, back it up in as many formats and locations as is feasible.
> I felt the person I replied to implied that 100s of terabytes of data are too expensive to backup.
Well, you felt wrong. Of course you should back up those 100s of terabytes, in fact that it is that much information is an excellent reason on top of all the other ones to back it up, re-creating it is going to be next to impossible.
It's just that the companies I look at - not all, but definitely some - seem to be under the impression that the cloud (or their cloud provider) can be trusted. Which is wrong for many reasons, not just this article.
No problem, it's just that with 'This is one of the reasons I always implore people to have a backup of their data with another provider or at least under a different account.' that passage I thought I had the backup angle more than covered.
What bugs me about it is that there are some companies that give serious pushback because their cloud providers keep on hammering in to them how reliable their cloud is and that any back-up will surely be less reliable than their cloud solution and oh by the way we also have a backup feature that you can use.
They don't realize that even then they still have all their eggs in the one basket: their cloud account.
It's strange, but I completely missed the last part about backup from your comment. I have no idea how I missed it. Had I seen it would make my comment redundant and I would have never replied at all.
I only saw that part of the comment much much later.
Well, it definitely wasn't added in a later edit, or at least, not that I'm aware of, though I do have a tendency to write my comments out in bits submitted piece-by-piece. Even so, I wouldn't worry about it, I tend to miss whole blocks of text with alarming regularity while reading through stacks of pdfs and when comparing notes with colleagues we always wonder if we've been reading the same documents (they have the same problem...). Reading in parallel is our way of trying to ensure we don't miss anything and unfortunately it is not a luxury.
Often the effects are more subtle, reading what you think something said rather than what it actually said, or missing a negation or some sub-clause that materially alters the meaning of a sentence.
Even in proofreading we find stuff that is so dead obvious it is embarrassing. On the whole visual input for data is rather unreliable, even when reading stuff you wrote yourself, which I find the most surprising bit of all.
Studying this is interesting, and to some extent important to us due to the nature of our business, missing critical info supplied by a party we are looking at could cause real problems so we have tried to build a process to minimize the incidence of such faults, even so I'm 100% sure that with every job we will always miss something, and I live in perpetual fear of that something being something important.
I forget where I first saw this quoted, but it's relevant here: "There is no 'cloud', only someone else's computer". That's part of why I store very little data online, compared to most people (or the data I actually have/want). Anything I'm not okay with someone else having on their computer is backed up and stored on hard physical media. No cloud provider can be trusted - the moment the government wants in, they'll get in; and the moment it's considered more profitable for the provider to quietly snoop in your stored data, rest assured that they will.
Yes, I'm aware of that. But you'd be surprised how many businesses are under the impression that using 'the cloud' obviates the needs for backups. Especially if their data is in the 100's of terabytes.
Non-technical owners making faulty assumptions is not the fault of "Cloud" providers. It's probably common (I faced it myself personally, in a non-cloud situation), but there is nothing the providers can do about unprepared users.
While true. I was specifically referring to this part:
> What if the card holder is on leave and is unreachable for three days? We would have lost everything — years of work — millions of dollars in lost revenue.
The comment suggests they are using personal GCP account instead of enterprise account.
Millions of dollars worth of work + imply no backup + non-enterprise account (but expecting enterprise support) + not having multiple forms of payment available.
Combining all these together, it seems like all sorts of things are going wrong here.
I have never used GCP (or any of the big three cloud providers), so I don't know how they are in general, but in this specific case there seems to be faulty planning on the user end.
Agreed, that wasn't smart. But, to their defense, this is how these things start out, small enough to be useful, and by the time they get business critical nobody realizes the silly credit card is all that stands between them and unemployment.
This vastly simplifies the situation, especially when the cloud is involved. Having a backup, much less a replica of such data requires an enormous infrastructure cost, whether it's your own or someone else's infrastructure. The time to bring that data back to a live and stable state again also is quite costly. (note the stable part)
It's a simple truth that even if you are at the millions of dollars point, there is a data size at which you are basically all-in with whatever solution you've chosen, and having a secondary site even for a billion dollar company can be exceptionally difficult and cost prohibitive to move that sort of data around, again especially when you're heavily dependent on a specific service provider.
Yes, the blame in part lies with making the decision to rely on such a provider. At the same time, there are compelling arguments for using an existing infrastructure instead of working on the upkeep of your own for data and compute time at that scale. Redundancy is built into such infrastructures, and perhaps it should take a little more evidence for the provider to decide to kill access to everything without hard and reviewed evidence.
It might be too expensive for some people. But really there is no other solution other than full backup of everything. Relying on a single point of failure, even on an infrastructure with a stellar record, is just a dead man walking.
And then of course there is the important bit that from a regulatory perspective 'just a backup' may be enough to be able to make some statements about the past but it won't get you out of the situation where due to your systems being down you weren't ingesting real-time date during the gap. And for many purposes that makes your carefully made back-up not quite worthless but close to it.
So then you're going to have to look into realtime replication to a completely different infrastructure and if you ever lose either one then you're immediately on very thin ice.
It's like dealing with RAID5 on arrays with lots of very large hard drives.
About ~6 years ago, I was involved in a project where data would increase by 100gb per day and the database would also significantly change every day. I vaguely remember having some kind of cron bash script with mysqldump and rsync that would have a near identical offsite backup of data (also had daily, monthly snapshots). We also had a near identical staging setup of our original production application which we would use to restore our application from the near-realtime backup we had running. We had to test this setup every other month - it was an annoying thing to do at first. But we were exceedingly good at it over time. Thankfully we never had to use our backup, but we slept at night peacefully.
Backup is a bit of an art in itself, everyone has a different type of backup requirement for their application, some solutions might not be even financially feasible. You might never end up using your backup ever at all, but all it needs is one very bad day. And if your data is important enough, you will need to do everything possible to avoid that possible bad day.
That's a good scheme. Note how things like GCP make it harder rather than easier to set something like that up, you'd almost have to stream your data in real time to two locations rather than to bring it in to GCP first and then to stream it back out to your backup location.
> Backup is a bit of an art in itself
Fully agreed on that, and what is also an art is to spot those nasty little single points of failure that can kill an otherwise viable business. Just thinking about contingency planning makes you look at a business with different eyes.
Yes, we have backups. The problem with `data` is not the fact I have backups, is that at a certain scale I will have so much data that "moving providers" could take on the order weeks.
If OP happened to me, sure yes I could have my entire infra on AWS/Azure/whatever else Terraform supports in an hour, maybe more to replace some of the tiny cloud specific features we use. But if it takes me a day to me to just move the data into Azure, thats an entire lost business day of productivity.
People usually end up using backup or move infrastructure on short notice in catastrophic situations, which is presumably rare. Days worth of work to bring back your business in catastrophic downtime - doesn't seem like a bad thing at all to me. If anything, it sounds like a very well organized development flow with very optimistic time-frame.
Note that I did not imply that restoring the service to a different project or provider would always be easy or fast (certainly in the case of very large data volumes it would be neither of those things). I was addressing the prospect of losing "years of work" as was stated in the OP. That sort of implies that most or all of what they did over that time is recorded only in the current state of the GCP project that was disabled, and that is a really terrifying position to be in.
If it takes weeks then you should choose a second provider where you can show up with your backup hard drives or whatever you use and plug them in. Moving data physically is an option.
We seem to hear a lot of bad google customer support stories. I guess it really shouldn't be surprising. Amazon grew as a company that put customers first. Google is kind of known for not doing that. They shut down services all the time. They don't really put an emphasis on customer support.
I’ve used AWS support many times and it’s actually really awesome. You can ask them basically anything and they have experts on everything. Really impressive. Yes you pay every month for it but it’s really good.
That's pretty bad of Google. I just looked in detail at a company using Google Cloud exclusively for their infra and the application is somewhat similar to what these guys are doing. I'll pass the article on to them. Thanks for posting this.
You should be have been able to deploy your git repository on another system pretty quickly, as well as have your own backups of your database.
The most time consuming thing should be setting up the environment variables.
Let me see, what else would be tricky: if you are using google analytics that data might be gone, but your other metrics package should have had many snapshots of that data too
Not OP, but according to his business case, being down for a few days could bankrupt the company. Re-deploying from git doesn't solve the use-case of your public cloud provider pulling the plug on your machines.
From a Google shareholder’s perspective, this approach is unacceptable:
The only way to use their product safely is to engineer your entire business so that cloud providers are completely interchangeable.
Forcing the entire industry to pay the cost of transparently switching upfront completely commoditizes cloud providers, which means they’ll no longer be able to charge a sustainable markup for their offerings.
This is fiscally negligent. Upper management should be fired.
However, it’s great for the rest of the industry — Google nukes a few random startups from orbit, some VCs take a bath, early and mid-range adopters bleed money engineering open source workarounds, and everone else’s cloud costs drop to the marginal costs of electricity and silicon.
If you use cloud services, a crucial scenario in your disaster recovery planning is "what if a cloud provider suddenly cuts us off?". It's a single point of failure akin to "what if a DC gets demolished by a hurricane?" or "what if a sysadmin gets hit by a bus?". If you don't have a plan for those scenarios, you're playing with fire.
371 comments
[ 2.7 ms ] story [ 327 ms ] thread"We have temporarily closed your Alibaba Cloud account due to suspicious activity. Please provide the following information within [24 hours] by email to compliance_support@aliyun.com in order to reopen your account: ... If you fail to provide this information your account will be permanently closed, and we may take other appropriate and lawful measurers. Best regards, Alibaba Cloud Customer Service Center "
I provided the documents in ~30 hours because that's when I saw the email. There was no further communication from Alibaba. I assumed everything is ok, but in 2 weeks my account was terminated.
I personally prefer using their API and tools over AWS. They seem more sensible. Azure is horrible.
GCP offers inter-region networking and more sensible firewalls and routers.
Pricing is pretty comparable to AWS, slightly better for instances, slightly worse for storage.
Storage performance for persistent SSD seems a little better than EBS GP2.
Cheap local disks on any kind of instance.
I, for one, wouldn't ever want to operate a firewall with enough rules where I'd have to be concerned about such a thing. Eek.
For just VMs with the fastest CPU, storage, networking and the easiest billing compared to other major clouds, GCP wins.
And had they converted their project to monthly invoicing: https://cloud.google.com/billing/docs/how-to/invoiced-billin...
* When you have invoicing setup, the above shouldn't happen. You need to keep a payment method in good standing, but you have something like 10 days to pay your bill. -- They do a little bit more vetting (KYC) on the invoice path, and that effectively gets you out of dodge.
* Without paying for premium support, there's effectively no support.
I think if someone didn't pay their bill on time, you might shut off their service too, wouldn't you?
What does that have to do with anything? The account was not shut down for non-payment, it was shut down because of unspecified "suspicious activity."
But even in case of non-payment I would not shut down the account without any warning. Not if I wanted to keep my customers.
Then they unplug the Ethernet cable and wait a week or two.
But as you said, this isn’t about non-payment.
A hypothetical but not unheard of scenario in which immediate shutdown might be warranted.
It's a rough world and different providers have optimised for different threat models. AWS wants to keep customers hooked; GCP wants to prevent abuse, Digital Ocean wants to show it's as capable as anyone else.
If you can afford it, you build resilient multicloud infrastructure. If you can't yet do that; at the very least ensure that you have off-site backups of critical data. Cloud providers are not magic; they can fail in bizarre ways that are difficult to remedy. If you value your company you will ensure that your eggs are replicated to more than one basket and you will test your failover operations regularly. Having every deploy include failing over from one provider to another may or may not fit your comfort level; but it can be done.
Not without warning, no. It is possible that the customer intended to start a CPU-intensive process and fully intended to pay for it.
Send a warning first with a specific description of the "suspicious activity" and give the customer a chance to do something about it. Don't just pull the plug with no warning.
There's a degree of complexity that comes with multi-cloud that's ill-suited for most early stage companies. Especially in the age of "serverless" that has folks thinking they don't need people to worry about infrastructure.
My point is that the calculus has more to it than just money. The prudent response, of course, is to do as you described. Have a plan for your provider to go away.
Offsite backups and the necessary config management to bring up similar infra in another region/provider is likely sufficient for most.
Rather, it would likely be enough to have a cloud-agnostic infrastructure with replication to a warm (or even mostly-cold to save on cost) standby at the alternate provider with a manual failover mechanism.
I disagree. More specifically, I think, instead, many [1] folks just don't make that assessment/estimate in the first place.
They just follow what they perceive to be industry best practices. In many ways, this is more about social proof than a cargo cult, even though the results can resemble the latter, such as elsewhere in this thread with a comment complaining they had a "resilient" setup in a single cloud that was shut down by the provider.
> There are distinct benefits that come with avoiding "HA" setups. Namely simplicity and speed.
Indeed, and, perhaps more importantly, being possible at all, given time ("speed") and money ("if you can afford it").
The same could be said of "scalability" setups, which can overlap in functionality (though I would argue that in cases of overlap the dual functionality makes the cost more likely to be worth it).
None of this is to say, though that "HA" is synonymous with "business continuity". It's much like the conceptual difference between RAID and backups, and even that's not always well understood.
[1] I won't go so far as to say "most" because that would be a made up statistic on my part
A clever man once said, "you own your availability".
An exercise in BC planning can really pay off. If infra is code, and it and the data are backed up reasonably well, then a good MTTR can obviate the need for a lot of HA complexity.
I assume I'm missing some meaning here, particularly since the premise of much of the discussion in the thread is that there can be high availability at one layer, but it can rendered irrelevant by a SPoF at another (especially when the "layer" is the provider of all of ones infrastructure).
Do you consider that a version of "none"? Or are you pointing out that, despite the middle ground under discussion, the "binary" approach is more common, if not more sensible?
Perhaps we'll start seeing a new crop of post-mortems from the "fail fast" type of startups failing due to cloud over-dependency issues. They're (presumably rare) edge cases, but easily fatal to an early enough startup.
I just heard a dozen founders sit up and think "Market Opportunity" in glowing letters.
CockroachDB has a strong offering.
But multi-cloud need not be complicated in implementation.
A few ansible scripts and some fancy footwork with static filesystem synchronization and you too can be moving services from place to place with a clear chain of data custody.
Everything I have runs in kubernetes. The only difficulty I have to deal with is figuring out how to deploy a kubernetes cluster in each provider.
From there, I write a single piece of orchestration that will drop my app stack in any cloud provider. I'm using a custom piece of software and event-driving automation to handle the creation and migration of services.
Migrating data across providers is hard as kubernetes doesn't have snapshots yet.
There are already a lot of startups in this space doing exactly the kind of thing that I just described. Most aim to provide a CD platform for k8s.
Yes, there's nothing wrong with that. You have their credit card and can even authorize certain amounts ahead of time to make sure it can be charged.
And if it was that critical it should have support and a SLA contract, and you know, backups.
Feel free to budget that into your choices on where to host, but getting into any arrangement where you rely on whatever free tier of support is nonsensical once you’re making any kind of money.
I do, but that wasn't my point.
We’re not a big Google customer, couple thousand a month, but when we migrated we instantly reached out to account reps and have regular quarterly check ins.
Even still, I feel like it's still reasonably likely that the robot would shut down a project for "suspicious activity".
I'm not sure if it's the culture of secrecy or sheer cluelessness, but it's pretty bad.
I'm not trying to slam them either, I still really like their products and use them.
Enterprise Google support: one day after your stuff goes down, you get a phone call saying the algorithms decided you're fraudulent.
[1] https://landing.google.com/sre/book.html [2] https://landing.google.com/sre/
Grace periods that respect your business should be a standard that all service providers hold themselves to
As far as I know, Mozilla has no business relationship with extension developers, so I would actually be very concerned if their first action wasn't to cut you off.
There is nothing dodgy about the extension. Mozilla was just being ridiculous.
Thank you for judging my business without even knowing it.
The extension is currently a proof of concept that I plan to revisit later.
I don't know anything about your business or the extension, I'm just pointing out that you're in a space that makes you suspicious by association.
Edit: I should add that after 2 weeks of back and forth emails the dude was finally able to build it then blamed me for not mentioning he needed to run "npm run build", even though I did mention it AND it's in package.json AND it's mentioned in the (very short and concise) readme.txt.
So after this exasperating experience he just took down the extension without warning and said it's because it contains Google Analytics.
I would have happily removed Google Analytics from the extension. The dude had my source for 2 weeks and could have told me about that at any time, but decided to tell me after 2 weeks of mucking around, after he had already removed the extension.
It was me that decided it was not worth the hassle to have the extension on their store. I just left it off.
So maybe think about which hosting provider to go with, don't get me wrong I like their tech. But their moderation does need a more human element, to be frank all their products do. Simply ceding control to algorithmic judgement just won't work in the short term if ever at all.
Can you elaborate on that? What do they monitor with the moderation bot?
"told us straight out that it would and we should move"
Sounds shady. I bet this would make more sense if OP explained what his company actually does.
Not everything is outright "likely to get banned" (eg pron things). ;)
Imagine you're running a cosplay community, and all of a sudden all your content is being deleted because the SESTA/FOSTA bill gets passed in a country where your "cloud" happens to reside in: https://hardware.slashdot.org/story/18/03/25/0614209/sex-wor...
I assumed they could tell that via CPU usage with they already monitor for quotas.
Erm ... no, evidently not?
A prudent person might consider a cloud provider to be a domain of failure and choose a multi-cloud option, which would probably be the correct way to address this resiliency issue. However, that's not really an appropriate approach for an early stage startup, where availability is generally not that much of a concern.
Is an exploding car safe because it is built by an early stage startup?
Just because you decide that implementing resiliency isn't a good business decision for some early stage startup, doesn't magically make the product resilient, it just isn't and that may be OK.
There are many options to choose from for implementing resiliency, it could be having multiple providers concurrently, it could be having a plan for restoring service with a different provider in case one provider fails, it could be by setting up a contract with a sufficiently solvent provider that they pay for your damages if they fail to implement the resiliency that you need, whatever. But if you fail to consider an obvious failure mode of a central component of your system in your planning, then you are obviously not building a resilient system.
Edit: One more thing:
> There's no indication anywhere from GCP themselves that a project could be a domain of failure. If asked, I doubt they would consider it as such.
Then you are asking wrong, which still is your failure if you are responsible for designing a resilient system.
If you ask them "Is a complete project expected to fail at once?", of course they will say "no".
That's why you ask them "Will you pay me 10 million bucks if my complete project goes offline with less than one month advance warning?", and you can be sure you will get the response to the problem that you are actually trying to solve.
If you replace "multi-cloud" with "multi-datacenter" (in the pre-cloud days), this premise is fairly unassailable. In those same days, applying it to "multi-ISP", it becomes more arguable.
Today, though, the incremental cost (money and cognitive) of the multi-cloud solution, even for an early startup, doesn't seem like it would be high enough to make the notion downright inappropriate to consider.
I'd even argue that if a cloud provider makes the lock-in so attractive or multi-cloud so difficult that that's a sign not to depend on those exclusive services.
The economics don’t work out if you are trying to do this with just vanilla VMs across AWS, GCP and Azure and managing yourself. You either do it the old fashioned way renting rack space and putting your own kit in, or you make full use of the managed services at which point - by design - you are locked in.
Good times.
Cryptocurrency?
Still works out cheaper for workloads than AWS does even factoring staff in at this point.
AWS always turns into cost and administrative chaos as well unless it is tightly controlled which in itself is costly and difficult the moment you have more than one actor. GCP probably the same but I have no experience with that. Very much more difficult to do this when you have physical constraints.
Two man startup, perhaps but I think the transition should go:
VPS (linode etc) for MVP, colo half rack, active/active racks two sites then scale out however your workload requires.
Basically, the LB decides which kubernetes cluster will serve your request and once you're in a k8s cluster, you stay there.
You don't have the control-plane that the federation provides and a bit of overhead managing clusters independently, but we have automated the majority of the process. On the other hand, debugging is way easier and we don't suffer from weird latencies between clusters (weird because sometimes a request will go to a different cluster without any apparent reason <-- I'm sure there's one, but none that you could see/expect, hence debugging).
My people's time is more important than your complex system.
One of the failure modes I see a lot is failing to factor in latency in distributed systems. Mainly because most systems don’t benefit at all from distribution and do benefit from simplification.
The assumption on here is that a product is going to service GitHub or stackoverflow class loads at least, but literally most aren’t. Even high profile sites and web applications I have worked on tend to run on much smaller workloads than people expect. Latency optimisation by flattening distribution and consolidating has higher benefits than adopting fleet management in the mid term of a product.
Kubernetes is one of those things you pick when you need it not before you need it. And then only if you can afford to burn time and money on it with a guaranteed ROI.
If you run into an issue, send me a note and I will get someone to reply to your issue.
1. https://cloud.oracle.com/compute/bare-metal/features
Maybe support just needed to satisfy their quota of kicked out companies for the month, who knows?
The nature of the tech in question seems important in this story.
By chance are you located out of the United States? These are not downtime issues, but anti-fraud prevention and finance issues.
I hope there is a possibility to put a backup contact person / credit card so organisations can deal with people going on vacation or being sick or whatever.
IMHO this should be nicely documented as any other technical material you get to learn about the cloud product when you create an account (e.g. important steps to ensure your account remains open even in case of important security breaches, yadda yadda it's possible we'll need a way to prove that you are you yadda yadda, this can happen when yadda yadda, be prepared, do yadda yadda)
AFAIK, Google Cloud credit card payments are processed through Google Pay, which supports multiple credit cards, debit cards, bank accounts, etc.
Ideally, in this case the company shouldn't be using the CFOs credit card, but entered into a payments agreement with Google, receiving POs, invoices and so on, including a credit line.
Never set up a crucial service like you'd set up a consumer service.
In many situations the "right thing" must be explained, otherwise when people fail to get it they can argue that wasn't really the right thing after all (sure that's ultimately because they just want to deflect the blame from themselves; so don't let them! clearly explain the assumptions under which anti-fraud measures are operating so people cannot claim they didn't know)
This is not true at all. Once you start spending real money on GCP or AWS, they will reach out to you. You will probably sign a support contract and have an account manager at that point. Or you might go with enterprise support where you have dedicated technical assets within the company that can help with case escalation, architecture review, billing optimization, etc.
> What if the card holder is on leave and is unreachable for three days? We would have lost everything — years of work — millions of dollars in lost revenue.
Just the way it is unless it is nuclear plant probably.
I've been running three providers as peers (DO, Linode, Vultr) as a one-man shop for years, and I sleep better at night knowing that no one intern can fatfinger code that takes me offline.
Ignoring tweaks for international customers, each proxy is in an A record round-robin with health checks via Route 53. US-based requests get forwarded to one of the pods, and the proxy either handles the request with local servers, or points to servers in another pod if it has servers that are down. If any pod has a power outage or goes down for any reason, Route 53 automatically pulls the entire pod out of the rotation. If an entire provider goes dark, all of the pods get pulled out of the rotation, but all the others keep running.
Not to mention the lack of visibility in changes - it seems like everything is constantly running at multiple versions that can change suddenly with no notice, and if that breaks your use case they don't really seem to know or care. It feels like there's miles of difference between the SRE book and how their cloud teams operate in practice.
Their sales staff is arrogant and has no idea how to sell into F500 type companies.
Source: 10+ meetings, with different clients, I attended where the Google sales pitch was basically "we are smarter than you, and you will succumb". The Borg approach. Someone needs to revamp the G sales and support approach if they want to grow in the cloud space.
And I've seen it cause them to lose at least 10 potentially good sales.
They have advantages but they're so arrogant that it puts people off.
It's more than 10 times or more people told me they prefer Google's solution to Microsoft or Amazon's but they're going with a competitor because they can't stand Google's arrogant attitude. It's close to laughable because of throwing money away just because they won't back off.
The other funny thing is the package had a neoprene sleeve for a Chromebook. Eventually a sales person reached out via email assuming I owned a Chromebook and acted like I owed them a phone call because they gave me a neoprene sleeve I couldn’t use.
The entire package ended up going in the trash, which was an unfortunate waste of unrecyclable materials.
"People who will look through every bit of advertising crap company x sends", vs those who don't.
Something, somewhere is probably making stats on that. ;)
Well, that seems to be the approach at Google. Starting with hiring
Not surprising they end up with a hivemind that can't see past their mistakes.
https://support.google.com/inbox/forum/AAAAg4HSpy4qi4kQLXV6w...
It blows my mind that GCloud, with arguably superior tech and performance compared to AWS/Azure, can't handle support. I have my own horror stories from 2 years ago, but still they haven't fixed it.
Google just doesn't seem to be able to focus on products that requires service and customer support. Maybe they just don't care about it while they have an infinite revenue stream from search and advertising. Whatever it is, they should be humiliated.
I love the tech, and the ux details like in browser SSH (AWS hasn't improved UX EVER) but they can't get support right? Amazing.
This means that you can run whatever development tools that you want on the EC2 instance, rather than the very limited code editor that Cloud9 provides. You can easily run a full copy of Visual Studio on a Workspace, and get the full resources of an EC2 instance with SSD drivess.
That's literally any product that people pay for (instead of viewing ads).
Customer support isn't and never has been in their DNA. It's often rage-inducing how hard it is to contact a human at Google.
They seem to think they can engineer products that don't need humans behind them.
I'm actually going to take this back to my company as a principle: "Treat locked-in/subscribing customers as well as our salespeople treat prospects."
I had an issue with my servers 2 days ago and I got a reply to my ticket within 1 minute. Follow-up replies were also very fast.
The person I was talking to was a system administrator who understood what I was talking about and could actually solve problems on the spot. He is actually the same person who answered my support requests last year. I don't know if that's a happy accident or if they try to keep the same support staff answering for the same clients. He was answering my requests consistently for 2 days this time.
I am not a big budget customer. AWS and GCP wouldn't think anything of me.
Thank you Vultr for supporting your product properly. And thanks Eric. You are very helpful!
Vultr looks like they provide more traditional services with a few extra niceties on top.
Within Google's infrastructure, I can deploy a new HTTPS REST endpoint with a .js file and 1 console command.
Could I set up an ecosystem on a Vultr VM to do the same? Sure, it isn't magic. But GCP's entire value prop is that they've already done a lot of that work for you, and as someone running a startup, I was able to go from "I have this idea" to "I have this REST endpoint" in couple of days, without worrying about managing infrastructure.
That said, articles like this always worry me. I've never seen an article that says "Wow Google's support team really helped out!"
The best thing to do is to simply ignore them as asking about downvotes just invites more of them.
But it is telling that there have been at least 5 downvotes but no one is willing to comment as to why.
Edit: since I see you have a downvote (surprise, surprise) I'll clarify that it wasn't me.
Seems like a recipe for breeding unfairness. "Don't talk about the system", :sigh:
There are ways to make the hosting relatively agnostic, but choosing a pub/sub solution (for example), that operates at 'web scale' will have a distinct impact on your solutions and force you into their ecosystem to maximize value. Why bother with BigCorps UltraResistant services if you're only going to use some small percentage of the capabilities?
I've made systems that abstract away the difference entirely, but I think the 'goldilocks zone' is abstracted core domain logic that will run on anything, and then going whole-hog on each individual environment. Accept that "cloud" is vendor lockin, and mitigate that threat at the level of deployment (multi-cloud, multi-stack), rather than just the application.
Downvotes can irritate, but there are at least two balms that don't involve adding noise to the threads. One is to remember that people sometimes simply misclick. The other is that unfairly downvoted comments mostly end up getting corrective upvotes from fair-minded community members (as happened with yours here).
https://news.ycombinator.com/newsguidelines.html
I'm happy for you to leave your comment. It might help someone else in future.
Anecdotally: I've been an MS gold partner in a bunch of different contexts for years. The experiences I had as 'small fish' techie with AWS were on par or better. YMMV, of course, but I'd be more comfortable putting my Enterprise in the hands of AWS support than MS's (despite MS being really good in that space).
Prices seem high though: https://cloud.google.com/support/?options=premium-support#op...
The costs of Google may be comparable or lower than other services, but they don't seem to get that risk is a cost. Risk can be your biggest cost. And they've amplified that risk unnecessarily and shifted it to the customer. Fatal, as I said.
You might never see this happen to your GCP account in it’s lifetime.
> What if the card holder is on leave and is unreachable for three days? We would have lost everything — years of work — millions of dollars in lost revenue.
You should never be in this position. If this were to happen to us we would be able to create a new project with a different payment instrument, and provision it from the ground up with terraform, puppet and helm scripts. The only thing we would have to fix up manually are some DNS records and we could probably have everything back up in a few hours. Eventually when we have moved all of our services to k8s I would expect to be able to do this even on a different cloud provider if that were necessary.
This is one of the reasons I always implore people to have a backup of their data with another provider or at least under a different account. That protects against all kinds of accidents but also against malice.
If you are making money or if the data is important for you to lose than you should have a backup, anything else is faulty planning.
Well, you felt wrong. Of course you should back up those 100s of terabytes, in fact that it is that much information is an excellent reason on top of all the other ones to back it up, re-creating it is going to be next to impossible.
It's just that the companies I look at - not all, but definitely some - seem to be under the impression that the cloud (or their cloud provider) can be trusted. Which is wrong for many reasons, not just this article.
What bugs me about it is that there are some companies that give serious pushback because their cloud providers keep on hammering in to them how reliable their cloud is and that any back-up will surely be less reliable than their cloud solution and oh by the way we also have a backup feature that you can use.
They don't realize that even then they still have all their eggs in the one basket: their cloud account.
I only saw that part of the comment much much later.
Often the effects are more subtle, reading what you think something said rather than what it actually said, or missing a negation or some sub-clause that materially alters the meaning of a sentence.
Even in proofreading we find stuff that is so dead obvious it is embarrassing. On the whole visual input for data is rather unreliable, even when reading stuff you wrote yourself, which I find the most surprising bit of all.
Studying this is interesting, and to some extent important to us due to the nature of our business, missing critical info supplied by a party we are looking at could cause real problems so we have tried to build a process to minimize the incidence of such faults, even so I'm 100% sure that with every job we will always miss something, and I live in perpetual fear of that something being something important.
That's true, but they can do something to avoid making things worse, see the linked article.
> What if the card holder is on leave and is unreachable for three days? We would have lost everything — years of work — millions of dollars in lost revenue.
The comment suggests they are using personal GCP account instead of enterprise account.
Millions of dollars worth of work + imply no backup + non-enterprise account (but expecting enterprise support) + not having multiple forms of payment available.
Combining all these together, it seems like all sorts of things are going wrong here.
I have never used GCP (or any of the big three cloud providers), so I don't know how they are in general, but in this specific case there seems to be faulty planning on the user end.
It's a simple truth that even if you are at the millions of dollars point, there is a data size at which you are basically all-in with whatever solution you've chosen, and having a secondary site even for a billion dollar company can be exceptionally difficult and cost prohibitive to move that sort of data around, again especially when you're heavily dependent on a specific service provider.
Yes, the blame in part lies with making the decision to rely on such a provider. At the same time, there are compelling arguments for using an existing infrastructure instead of working on the upkeep of your own for data and compute time at that scale. Redundancy is built into such infrastructures, and perhaps it should take a little more evidence for the provider to decide to kill access to everything without hard and reviewed evidence.
So then you're going to have to look into realtime replication to a completely different infrastructure and if you ever lose either one then you're immediately on very thin ice.
It's like dealing with RAID5 on arrays with lots of very large hard drives.
Backup is a bit of an art in itself, everyone has a different type of backup requirement for their application, some solutions might not be even financially feasible. You might never end up using your backup ever at all, but all it needs is one very bad day. And if your data is important enough, you will need to do everything possible to avoid that possible bad day.
> Backup is a bit of an art in itself
Fully agreed on that, and what is also an art is to spot those nasty little single points of failure that can kill an otherwise viable business. Just thinking about contingency planning makes you look at a business with different eyes.
If OP happened to me, sure yes I could have my entire infra on AWS/Azure/whatever else Terraform supports in an hour, maybe more to replace some of the tiny cloud specific features we use. But if it takes me a day to me to just move the data into Azure, thats an entire lost business day of productivity.
https://aws.amazon.com/snowmobile/
Paid support tiers (as far as I know) are for deeper system level diagnostics.
Okay that sounds like a greater systemic problem.
You should be have been able to deploy your git repository on another system pretty quickly, as well as have your own backups of your database.
The most time consuming thing should be setting up the environment variables.
Let me see, what else would be tricky: if you are using google analytics that data might be gone, but your other metrics package should have had many snapshots of that data too
The only way to use their product safely is to engineer your entire business so that cloud providers are completely interchangeable.
Forcing the entire industry to pay the cost of transparently switching upfront completely commoditizes cloud providers, which means they’ll no longer be able to charge a sustainable markup for their offerings.
This is fiscally negligent. Upper management should be fired.
However, it’s great for the rest of the industry — Google nukes a few random startups from orbit, some VCs take a bath, early and mid-range adopters bleed money engineering open source workarounds, and everone else’s cloud costs drop to the marginal costs of electricity and silicon.
the shareholders should be proud that such naivete towards vendor lock is still rampant
https://cloud.google.com/support/?options=premium-support#op...
https://libcloud.apache.org/