Ask HN: Have you ever switched cloud?

>Dozen of decade+ billon+ customer contracts at this point make shutting GCP down a silly thought.

Really crappy that the demos sucked tho, sorry :(

> Interesting that you trust Google more in this regard. Given Google's terrible history of deprecating products, I would not trust any "real world" business to any of their services.

That doesn't apply to me, because I have never used any of Google's deprecated products. I assume that's because they haven't (or have they?) deprecated any of their cloud services.

> Also, at some point I was playing with Serverless in both Google and AWS. Google's Serverless examples were broken (Google cloud was returning 500 errors) while the same stuff in AWS worked smoothly. That left me with a bad taste.

Good for you!

I have used Google Cloud Run for more than a year now, and can't be more happy. Never had problems with AWS either, which means that there are at least two cloud providers providing the same-ish service that lots of people can enjoy.

I searched for exactly those 3 words on Duckduckgo.com and got https://docs.ovh.com/us/en/publiccloud/ai/training/build-use... which is a tutorial for what I use.

Probably OVHcloud AI tools. They're docker based.

https://docs.ovh.com/us/en/publiccloud/ai/

Many operations are INCREDIBLY slow. Creating a VM. Mounting a disk to a VM. All take easily 10-30x longer than I'm used to with other clouds. This is especially annoying with Kubernetes as it likes to move disks between VMs which... takes a while on Azure.

Many, many, backwards incompatible changes in their Kubernetes platform. I've had to recreate clusters about twice a year so far. Lately it's been better since they finally got node pools working (about 2 years after all of their competitors).

They can't get their network stable. Things like `kubectl port-forward` or `kubectl logs` hang after 4-30 minutes of inactivity (i.e. the tunnel is open but no packets actively being sent) which, according to Azure, is "working as expected". This makes the tooling utterly unusable. It has to do with the way Azure's load balancers deal with idle TCP connections.

Also, their support engineers are unwilling to help you unless you run Windows. They always insist on remoting into your machine using some Windows utility, even though the issue is with their cloud instead of my machine.

In my experience a lot of difficultly with Azure arises because a business is probably already using Azure to manage it's IT infrastructure.

You then get into sticky situations where IT are unhappy handing over admin access to the Ops teams: for example refusing access to AAD because it is integrated with the _corporate_ domain.

This might seem fine on the surface, it's just a people issue, but it can become very tiresome when dealing with Azure resources their permissions.

Worse still is some Azure resources use AAD almost like a data store: such as b2b and b2c. If you have write back enabled on your AAD (as most companies would - otherwise users wouldn't be able to self service forgotten passwords) you will _apparently_ clog up the on-premise domain with foreign objects from your b2b configuration.

Of course you can get around all of this by having a separate tenant in Azure for SaaS teams/Ops only. But you introduce the headache of management (should both tenants be under a super enterprise tenant?) and single sign on (security might say there is only one user login for everything with RBAC & MFA managed from one place... now you have to join the tenants somehow...)

The biggest problems: things don't work, things don't compose, and support is unknowledgeable & ineffective.

Things don't work: outages, bugs. So many I'm not sure where to start: ACR: slow as molasses. I don't know why they didn't build it on blobstore. Many queries are clearly O(images) even when they shouldn't be. Throughput it terrible and made worse by inane pagination; listing images, for example, has a throughput of something like 60kbps. And "b", as in bits. Minutes for 3MB of data. It's absurd; they think that's "working as expected". AKS: they manage the API server in AKS, and we find it is routinely non-responsive. We went through a quarter-long support ticket, which went back and forth between "you're putting too much load on the API server" -> "we don't think we are, what load is there?" "here's the top queries" and they're all queries from like, the cluster controllers — which are also managed by AKS/Azure. -> "well there's too much load!". App Gateway: normally stable, but had an outage when Let's Encrypts old root expired. (We were using a cross signed cert — i.e., our cert was valid, but App Gateway failed it, i.e., a false negative validation.) They never acknowledged the outage, and the support ticket we filed didn't get a response until like days later — missing support's SLA — meanwhile, some engineer somewhere clearly fixed the service, as it started working all on its own. ACR: we used to get 500s. IDK if these still happen or not, as we retry-looped most of the spots that were hitting them. Support response was ridiculous "what's your ISP?" "… this is inside your network, Azure, and my ISP wouldn't cause you to serve a 500." "Maybe the image is too big" "The image is ~100 KiB" … global AAD outages, the status page going down while the twitter account is like "check the status page", the Portal has issues (a few days ago listing subscriptions just returned 0 rows. Like, okay, I guess I'm not looking at those today), the activity log will occasionally just return errors, or today, it returned 0 entries where I knew there to be entries.

Composability: a cloud provider's job is to offer bricks from which I can build whatever infra my company's needs demand. But Azure constantly says "well, no, you can't put those bricks together that way." IPv6 anywhere in your vnet? You can't add a managed PostgreSQL server to that vnet, cause not only does it not support IPv6, you can't add it to an IPv4-only subnet in a dual-stack vnet. Like, the entire point of dual-stack…; also, when we attempted this, the API request took 2 hours to fail, and failed with "Internal error, please retry", which we then did, like chumps. 4 hours later, support ticket. AKS will add new features (that ought to have been there from day 1) like nodepools … but only to newly created cluster. You want to take advantage of that? Too bad! Recreate that cluster from scratch!

Support: Azure has no meaningful tooling for handling bugs in their services. The only hammer they have is support, and by god it's going to hit that nail. Support might (if you can get them to admit that yes, shit's broken) field a bug report, but then the support ticket is closed. Is the bug fixed? How will you know? IDK. Also, AFAICT, certain products you just can't open a support ticket for; notably, the portal. It isn't an "Azure service", so it isn't in the list of things to select from. Also, they override the mouse wheel on the list of services, so scrolling ~1 "detent" on a trackpad results in the list scrolling at Mach 4. Support tickets lack URLs, so they're unlinkable. Occasionally whatever the agents use to view info on tickets gets desynced from the ticket, and new replies in the portal are ...

Having plans for what to do in case of a success disaster is good. Spending in expectation of having a success disaster can be disastrous.

Funny, but honestly, a single dedicated server should easily be able to handle a million users, for most CRUD apps.

Funnily enough that was a use case for dedicated servers I encountered. Usage based pay is great if you have an unlimited budget, but most companies would rather have predictable cost and not serve every traffic spike.

Especially for internal use (CI/CD, Analytics), you'd rather want to queue a few things up than always having to consider your budget when you want to run something.

That "inevitably" is doing some really heavy lifting in your post. It practically never comes for most companies.

One phrase: right sizing. The original infra on OVH used an instance type that was suboptimal for the workload. Because re-provisioning was not an option due to the size of the cluster, especially without moving hard drives we had only one option. Move the data to S3, find the right instance type and switch over. The ability to right size your infra is grossly underrated. Decoupling data from compute too. We are talking about hundreds of nodes.

I've bounced between AWS, Azure, and GCP. Once they get entrenched enough the credits stop but right now it's GCP with the best incentives.

Also, don't sleep on Oracle. Their cloud platform is stupidly competitive price-wise but limited feature-wise. If you're just looking for basic compute and storage they can't be beat, sans credits.

Whenever I've interacted with Rackspace they've seemed almost proud to be expensive.

At the same scale? I guess if you move a whole data center, they'll quote very different prices from their public ones.

It might be different if you're a developer in SV, but I'd say for most people, that's not insignificant. That's more expensive than most hobbies, I wouldn't spend £100 on a side project if there's an alternative.

Sure, but think about this: that little side project now has about 100x the traffic it did when my bill jumped.

That still isn't much traffic (at all) in the grand scheme of things, and Cloudflare's lowest paid tier deals with around 80% of the bandwidth. Still, it's not hard to imagine that bill blowing up to several hundred pounds per month had I chosen not to act. That would translate to several thousand pounds over the course of a year. I don't know very many people for whom such an expenditure, particularly if it's unnecessary and avoidable, would be something they'd regard as insignificant.

Putting it into everyday terms: it could have grown into my second largest outgoing after my mortgage. That doesn't really seem proportionate or sensible, so why wouldn't I look for a better deal?

Absolutely. And dedicated servers at many providers (e.g. Hetzner, OVH) can be provisioned much like vms. So the only real difference is that usually there's a minimum contract of 1 month, but often at a price where it's cheaper than running a VM at a cloud provider for 30% of the time.

That's something I find so fascinating. The "cloud" will almost always be more expensive and "not worth it" if you are only using the IaaS services. I mean, look at the numbers, everyone sees that.

Cloud only ever is worth it when one uses the higher-tier services, like AWS Lambda and the likes. Even running Kubernetes in the cloud is only semi worth it, because it's not high enough in the stack and still touches too many low-level IaaS services.

Of course, higher tier means more vendor lock-in and more knowledge required and all that. But if you are not willing to pay that price, then OVH, Hetzner and the likes will have better offerings for you.

I'm still not a fan of the second way. If you develop all your software to tightly integrate with AWS, you might save time developing the software but create a huge amount of technical debt.

Managing your own infrastructure (with dedicated servers, so no hardware management) isn't too hard, even if you're a small shop. And managing a fleet of AWS services isn't necessarily less work.

Maybe there's a reason all ads for cloud tend to compare it to running your own data center. Because once you get rid of hardware management, it's not really that much easier being in the cloud, at the risk of lock-in and huge surprise bills.

Is that taking spot/reserved instances into account (for both). Afaik Fargate supports both of those

Spot/Reserved instances are used.

I probably should have clarified that the extra cost was expected as we did a lot more in AWS than we could in Heroku, we used the switch to start using bunch of stuff AWS offers like Lambda functions, CloudFront, RDS etc. Stuff that we just didn't (and couldn't) use on Heroku, thus didn't pay for it.

As the purpose of the switch was to get more control, features and out of the Herokus "black box", higher costs were expected and perfectly normal.

can you share more details, because im in teh process of doing the same since having a few terabytes of postgresql / dynamodb is stupid expensive.

for a lot big organizations its a matter of accountability. if they say AWS went down vs our dedicated servers went down, it matters a lot for insurance, clients.

what i dont get are 4 man startups paying thousands to AWS ... because everybody does it.

Very similar experience here. I work on a two person "DevOps" team. Without AWS ECS we would have to have a much higher headcount. I get to spend most of my time solving real problems for the engineers on the product team rather than sysadmin work.

> None of us are or want to be sysadmins.

It's such a huge misconception that by using a cloud provider you can avoid having "sysadmins" or don't need that kind of skills. You still need those, no matter which cloud and which service you use.

"Infrastructure is cheaper than developers(sysadmins)" all over again.

Managed container services (like Amazon ECS) are a sweet spot for me across complexity, cost, and performance. Mid size companies gain much of the modern development workflow (reproducibility, infrastructure as cattle, managed cloud, etc.) using one of these services without the need to go full blown Kubernetes.

It's lower level than functions as a service, but much cheaper, more performant, matches local developer setups more closely (allowing for local development vs. trying to debug AWS Lambda or Cloudflare FaaS using an approximation of how the platform would work).

Very much agree - due to a coworker leaving recently, I'm looking after two systems. They're both running on ECS and using Aurora Serverless.

My company takes security very seriously so if these two systems were running on bare-metal I'd probably be spending one day a week patching servers rather than trying to implement new functionality across two products.

I can bet our team is smaller than yours.

And yet… Sysadmin tasks take up maybe 2 hours a month.

Your theory is right though if no one on your team knows how to setup servers. In your case the cloud makes sense.

To the peeps running ECS? Why not just straight up AKS or GKE? Have you compared ECS to Cloud Run on GCP?

Yup. Pretty much no complaints. We do have a very boring setup though YMMV

Bump! This sounds very interesting.

Unfortunately not, but it's surprisingly straight-forward, apart from the database bit, but here's a bit more detail from memory. There are many ways of doing this and some will depend strongly on which tools you're comfortable with (e.g. nginx vs. haproxy vs. some other reverse proxy is largely down to which one you know best and/or already have in the mix) [Today I might have considered K8s, but this was before that was even a realistic option, but frankly even with K8s I'm not sure -- the setup in question was very simple to maintain]:

* Set up haproxy, nginx or similar as reverse proxy and carefully decide if you can handle retries on failed queries. If you want true zero-downtime migration there's a challenge here in making sure you have a setup that lets you add and remove backends transparently. There are many ways of doing this of various complexity. I've tended to favour using dynamic dns updates for this; in this specific instance we used Hashicorp's Consul to keep dns updated w/services. I've also used ngx_mruby for instances where I needed more complex backend selection (allows writing Ruby code to execute within nginx)

* Set up a VPN (or more depending on your networking setup) between the locations so that the reverse proxy can reach backends in both/all locations, and so that the backends can reach databases both places.

* Replicate the database to the new location.

* Ensure your app has a mechanism for determining which database to use as the master. Just as for the reverse proxy we used Consul to select. All backends would switch on promoting a replica to master.

* Ensure you have a fast method to promote a database replica to a master. You don't want to be in a situation of having to fiddle with this. We had fully automated scripts to do the failover.

* Ensure your app gracefully handles database failure of whatever it thinks the current master is. This is the trickiest bit in some cases, as you either need to make sure updates are idempotent, or you need to make sure updates during the switchover either reliably fail or reliably succeed. In the case I mentioned we were able to safely retry requests, but in many cases it'll be safer to just punt on true zero downtime migration assuming your setup can handle promotion of the new master fast enough (in our case the promotion of the new Postgres master took literally a couple of seconds, during which any failing updates would just translate to some page loads being slow as they retried, but if we hadn't been able to retry it'd have meant a few seconds downtime).

Once you have the new environment running and capable of handling requests (but using the database in the old environment):

* Reduce DNS record TTL.

* Ensure the new backends are added to the reverse proxy. You should start seeing requests flow through the new backends and can verify error rates aren't increasing. This should be quick to undo if you see errors.

* Update DNS to add the new environment reverse proxy. You should start seeing requests hit the new reverse proxy, and some of it should flow through the new backends. Wait to see if any issues.

* Promote the replica in the new location to master and verify everything still works. Ensure whatever replication you need from the new master works. You should now see all database requests hitting the new master.

* Drain connections from the old backends (remove them from the pool, but leave them running until they're not handling any requests). You should now have all traffic past the reverse proxy going via the new environment.

* Update DNS to remove the old environment reverse proxy. Wait for all traffic to stop hitting the old reverse proxy.

* When you're confident everything is fine, you can disable the old environment and bring DNS TTL back up.

The precise sequencing is very much a question of preference - the point is you're just switching over and testing change by change, and through most of them you can go a st...

> architect your services to be able to double-write

Can you share any details on how to achieve this?

For instance, if the first database accepts the write but the second is temporarily inaccessible or throws an error, do you roll back the transaction on the first and throw an error, or <insert_clever_thing_here> ... ?

You're right double-writes are flexible, and it's great when it works. With schema migrations I'm fine with it, because you can usually enforce consistency.

But with migrations from one database to another at different locations, I'm lukewarm to it because it means having to handle split-brain scenarios, and often that ends up resulting in a level of complexity that costs far more than it's worth. Of course your mileage may vary - there are situations where it's easy enough and/or where the value in doing it is high enough.

If it had been a one off migration we might have done the same, but since the end goal was Hetzner from the start, and that meant their architecture needed to handle the HA piece anyway, and served as nice validation we really could fail over without anything going down.

You're reasonably in the ballpark. It was complicated a bit because hosting on Hetzner was so much cheaper for them that $1 in credit was not worth $1 to them, as if/when they had to spend cash they spent substantially less than that at Hetzner.

Just a note, there are some companies that have been popping up recently to try to bridge the gap for services on clouds like Hetzner.

https://elest.io

https://nimbusws.com (I'm building this one so I'm biased for it).

> Would you say bare metal cost a lot of extra monitoring/maintenance, or is this something you did on the cloud hardware as well anyway? Do you run virtualization on the Hetzner machines?

It cost a lot of monitoring/maintenance up front, but once things are purring the costs amortize really well. Hetzner has the occasional hard drive failure[0], but you should be running in a RAIDed setup (that's the default for Hetzner-installed OSes), so you do have some time. They also replace drives very quickly.

If you really want to remove this headache, you run something like Ceph and make sure data copies are properly replicated to multiple hosts and you'll be fine if two drives on a single host die at the same time. Obviously nothing is every that easy but I know that I spend pretty much no time thinking about it these days.

I run a Kubernetes cluster and most of my downtime/outages have been self-inflicted, I'm wonderfully happy with my cluster now. Also another thing to note is that control plane downtime != workload downtime, which is another nice thing -- and you can hook up grafana etc to monitor it all.

[0]: https://vadosware.io/post/handling-your-first-dead-hetzner-h...

I would say cloud costs a lot of extra maintenance. When I did contracting, those of my clients who insisted on AWS tended to be far more profitable clients for me, because they needed help with so much more stuff.

In terms of monitoring, it boils down to picking a solution and building the appropriate monitoring agent into your deployment.

I've run basically anything I run in some virtualized env. or other since ~2006 at least, be it OpenVz (ages ago), KVM, or docker. And that goes for Hetzner too. It makes it easy to ensure that the app environment is identical no matter where you move things. I managed on environment where we had stuff running on prem and in several colo's, on dedicated servers at Hetzner, and in VMs, and even on the VMs we still containerised everything - deployment of new containers was identical no matter where you deployed. All of the environment specific details were hidden in the initial host setup.

See this older sibling comment https://news.ycombinator.com/item?id=30950842

I was the only person doing the migration work and setting up the HA setup and documenting it for them to take over. My fee for setting it up and doing it accounted for that 20x difference. Hetzner in the end was far cheaper for them to run including the devops work they contracted to me. I effectively got paid to reduce my future earnings from them. But that was fine; when I was doing contracting, that was a big part of my pitch - that I'd help drive down their costs and pay for myself many times over - if people were in doubt I'd offer to take payment as a percent of what they saved.

So, no, it wasn't free, but it saved them far more money than it cost them, both the initial transition and in ongoing total cost of operation.

In fact, my first project for them was to do extensive cost-modelling of their development and operations.

Second this question. Never heard of IBM being exceptional for k8s. Curious to know what makes OP say it’s so good.

Not sure where you're currently at and considering moving to, but GKE has been much simpler than EKS. Not sure about cost but it'll likely save some operations time (auto scaling is a single check box, no IAM, scaling controllers, etc)

I'd throw in stability, IAM, storage and the management plane.

You are correct, But as I understand, They get "RAW" file access to the S3 hard disk (or the equivalent) in their solution. So no matter what solution I might spawn, it will always be slower and more expensive.

Maybe I should have said "closed internal access features"

This is pretty misinformed. Each provider has multiple “serverless” offerings and cold start time has much to do with your specific application and what it’s doing on start up.

None of my workloads (5x Cloud Run services, 10's of Functions) have anywhere near 5s cold starts. More like 2s with network latency.

Do you mean GCP Cloud Run [1] ? I would love to have it on AWS and Azure if you have a link to share. Or do you mean it's possible, but through different services on each cloud?

[1] https://cloud.google.com/run

> Was it a business or technical decision to do multi-cloud?

> Did you run simultaneously in 3 clouds? Can you explain the setup?

The solution itself could be running on a single cloud. But we work in the finance sector and targeting highly regulated clients. And we got a tip very early on, that each client could ask for deployment on their cloud account that is monitored by them. Which will probably be AWS or Azure. Today we know only some require that. So it helped somewhat.

> were the savings/benefits of doing this actually worth the engineering costs involved in the end?

Like you said, very hard to know. In our case, we had a DevOps Cloud guy working a full time job, so, it was not noticeable. Reason being, Probably,

[1] Because although he had problems to solve on all clouds, clouds deployments eventually get stable enough, So pressure was spread.

[2] Although all clouds still need constant maintenance, it's a-synchronic (you can't plan ahead when AWS EKS K8s will force a new version), so pressure was spread out and it never stopped client feature building.

But who knows, maybe for other architectures or a bigger company, it would have become noticeable.

Always look for kickbacks if there are cases where the company ends up being out more money all-in. These decisions should not have been made in this way, and it isn't always incompetence that drives this.