Ask HN: Is Digital Ocean safe enough for production?
I am in the late stage of developing application using Firebase, Google App Engine and some Compute for my application backend, primarily because it is less pain than managing own servers. But the post yesterday "Firebase Costs Increased by 7,000%!"[1] made me rethink my decision as I can not afford any surprise bills now. So the cheaper alternative would be to use Digital Ocean, but is it production ready and if anyone using it successfully for running a large production app ?
[1]https://medium.com/@contact_16315/firebase-costs-increased-by-7-000-81dc0a27271d
49 comments
[ 3.3 ms ] story [ 119 ms ] threadNow there's only virtualization on top of that.
"Is this new make of car being manufactured safe to drive? How reliable is it?"
"Yes of course cars are safe, cars have existed for over a century. Many cars have been used since the beginning of cars being invented. You could make a car too if you had the money".
See how ridiculous and unhelpful your comment was?
https://uncrunched.com/2013/08/07/digital-ocean-v-aws-10x-pe... for an old comparison.
EC2 is, and always have been slow and expensive. its the other bits that make it compelling.
I would take a good, long, hard look at any decisions you've made on this basis.
You haven't avoided the pain, you've deferred it. Would it be better to deal with that pain now when you have no users and no critical data, or later when scheduled downtime is a major issue?
It's a calculated risk: Is the business revenue going to grow faster than the pain from these decisions?
Since it sounds like you haven't even launched yet, DO should have no issue getting you started.
I still run all my production stuff on Linode because I've never had any unplanned downtime since 2009 (when I started using them), they've had issues over the years but always stuff away from the actual core thing I value which is keep this online.
Since they underwent a massive DDoS a while back they've really hardened their systems up and put in their own links to the backbone all that jazz.
Go read https://www.digitalocean.com/business and https://status.digitalocean.com/ if that doesn't put you at ease send them an email.
However it shouldn't matter. Droplets are essentially just on demand "vps's" most of the operations work is done by your operations team(in this case I imagine this is you). Where your hosts(machines, containers, vps's whatever you want to call them) are shouldn't matter very much, what is important is that you have repeatable infrastructure, so that when you spin up a new VPS anywhere(AWS, DO, Vulcan, Your Garage) you can get up and running again fairly quickly.
[edit] Added status page
You can manage your own SSH keys, software updates, and you can enable 2FA on your DO account. It's definitely safe enough, if you're safe enough.
Happy to answer any questions, but tbh their support/docs section is some of the best on the web - I often find myself referring to some of their docs for non-DO stuff in my day job!
Your biggest concerns, maybe:
+ Bandwidth limitations (the $5 instance has 200m/s I think)
+ Lack of additional services you get in other clouds (email sending with AWS SES, for example). It's ok if you just need instances, shared storage, shared IPs, availability zones, backups and an API.
+ Response capacity. I'm not sure if it will be better or worse, but I think it's not the same team, architecture, core infrastructure, etc compared to your current providers... this may require some research to understand if it's a concern for you.
+ The "private" IP of the instances, so you can talk with instances in the same region, is in a shared network with other clients (or at least, it was the last time I did check).
+ Some services (balancers, ipv6) have been released recently, so maybe they are not so "mature" as in other providers. Didn't test any of them here, sorry.
+ Some times, the kernel version in the repos, and the kernel version available for the instance from the API or web interface, are out of sync (at least in Debian instances).
Said that, the instances and the service, perform as expected.
You also have "cloud init" and an API in digital ocean, so you can hook your configuration management system, to automate actions, scaling-up, scaling-down, etc.
They notify for planed maintenance.
As in every cloud provider, sometimes you spin up an instance, and you see lot of old traffic and requests (crawlers, bad dns's, etc). This is no different here.
I think you should make a POC with a pre-production environment... this is cheap. How big is your app?
If you're very comfortable managing servers and your goal is to keep costs very low then it go for it. If time is more important (likely) then stick to what you have, then migrate only when required and you have a real business case for the move.
I know DBs are not the hardest things to setup but in the back of my mind I am always concerned about the security of my DB setup along with any tuning.
About 3 years ago, our instance got shut down over the weekend because a spammer in Germany reported it for serving an 'infected' file while attempting to advertise their MX scanning service. Said file had a false positive in ClamAV (common for Windows EXEs) and only ClamAV. Digital Ocean shut down the production instance with no warning and then told us about it via ticket. It took hours to get a response and when we finally did, the instance could not be restarted (DO at the time had a bug where the kernel indicated in their admin had to match the kernel in the image or it could fail to start). Total downtime was 2 days if I recall correctly. The security spammer took out our instance 1 or 2 more times before DO finally either blacklisted them or correctly noted that it was a likely false positive. I got a few months credit for this issue at the time.
About 2 years ago, DO had an issue where they shutdown networking to our production instance because they detected a "Brute Force attack" originating from that instance. The detection was due to an increase in downloads because we'd released a new version of our software coupled with, yet again, a false positive in ClamAV (and only in ClamAV)... though in a separate copy of software on the same server that had been released a month prior. This also happened on a weekend and took hours and then a couple days to resolve due to the aforementioned bug at the time of DO instances failing to start due to the kernel in use within the Droplet not matching the kernel indicated in the admin.
In both of the above instances, a single support person killed the instance without pinging the customer first. This is to be expected from any mostly-automated lower cost provider. Contrast this to a Rackspace managed cloud instance where they will first contact the customer when a detection occurs or a false positive shows up in an antivirus scan. Of course, that's comparing our managed cloud instance at Rackspace to a self-serve low-end droplet at Digital Ocean. Why the difference? Likely because you'll pay between 3 and 5 times as much for the same level of server between the two providers.
As a result of the above, I kept our production servers at Rackspace but continued to use Digital Ocean for secondary services. We have backups for those services on DO and the ability for our product to fail-over to the backups automatically should an issue like this occur again. We have been considering moving more services to Digital Ocean as they appear to have ironed out the kinks of their first few years of operation now.
The biggest issue we've faced was their use of Google DNS. A couple of months back, Google DNS had an outage, and all our calls to external services semi-silently failed. Because this wasn't technically a DO service outage, it wasn't reported by them, and because we didn't know they used Google DNS we didn't trace the issue to the outage. We could have done more, but they could have also been more transparent.
Overall, we are starting to move more projects to DO from Linode, because I feel Linode's previously excellent support has slipped dramatically (customer for at least eight years). We also leverage a lot of AWS services (particularly S3 and SES).
(DO do have their own DNS API, but it is terrible to program against.)
[0] https://roots.io/trellis/ - If you have to work with Wordpress at all I strongly recommend you give Trellis and the other Roots.io projects a try. Coming from more "sophisticated" tools to Wordpress was a real struggle until I found roots.
1.1. Elastic IP - Allow you to remap IP to another instance, this is very important for HA where you can bring up another server and remap IP to new instance.
1.2. Alternative to elastic IP is load balancer
2. Bock storage - Allow you to use persistent storage with redundancy. Important to keep data safe when some instances down. You can bring up new instance and start using existing block storage.
DO support all of the above so DO is good enough for production usage. If you need latency sensitive services like RDS, maybe you can consider AWS. I know some people use RDS with compute on other provider like DO.
When I've done the calculations for my use cases, I've found it would be significantly cheaper for me to have a second, redundant copy of my entire infrastructure at another provider as a failover in the event of a prolonged DO downtime than it would be to host my side projects at Amazon or Google.
On the plus side, DO will make you jump through fewer hoops than GCE to get started. GCE had UX and logic bugs in their automated billing system when I used them last that ended up affecting uptime.
Your software is too precious and your small startup is too precious to be married to someone who just thinks of you as "a customer". DO has never done me wrong (I have used their platform for years) but having a plan is handy.
Then, if you run into issues with either DO or the competitor, you already have an account and familiarity with how things work.
Also, don't host the DNS records at either place. Put them somewhere else so that your DNS records aren't held hostage if there's an account issue.
If you know how to do all the sysadmin stuff, I would say your fine in terms of using DO for production.