I'm glad the author didn't turn this into a negative review of DO. "What will you do when your server is lost?" is exactly the right question to learn from a scenario like this.
Agreed. There may also be a customer service lesson in that refund email. Typically, your customer is not happy with you even after they receive a refund, so exclamation points and language like "Booyah!" is really not a good idea for that sort of message. It's always going to sound a little bit tone deaf -- and very tone deaf when the refund was for a major incident like the loss of an entire server.
It's the standard DO credit email, which typically is from referring friends or doing something else to acquire credit. Not getting a refund because DO accidentally deleted your droplet.
Though their should be another avenue for such things.
Hmm, maybe that's what they're thinking. But I wonder if it's really true that a refer-a-friend or similar, happy scenario is the "typical" cause for a refund. Certainly in my life I've received far more refunds due to company mistakes than than credits due to the happier reasons you mention.
And when in doubt, if you're refund system is not capable of discriminating between the two, it seems to me wiser to err on the side of a less jubilant message.
DO credits from positive events are pretty common in my experience; not just refer-a-friend, but they give out credits through ads on podcasts and the like, through promotions for students and whatnot. I think I've had something like $150 total gifted DO credit.
This seems like a non-issue to me. If you're using an IaaS provider you should be treating the network as volatile from the get-go. This is the reason AWS has things like auto-scaling groups. You should be designing for failure in "the cloud"
Agreed. This shouldn't be an issue for anyone using cloud services -- or any computer really. Except for some super duper redundant mainframes maybe, you should always be able to deal with losing a component not matter what it is. If you really care, always have multiple instances, in multiple (availability) zones, etc.
Yep, somehow cloud customers have decided that by being in the cloud everything is redundant and durable and has several other magical properties.
At one of my positions we had stupidly put too many eggs in the basket of a single physical machine. Its disk controller failed in a way that it trashed the data volumes. I was unable to convince anyone that "move to Amazon" was not a one-step solution to "how do we make sure this never happens again".
>somehow cloud customers have decided that by being in the cloud everything is redundant and durable and has several other magical properties
To be fair the point of the cloud is that they deal with redundancy, HA, distributing to multiple datacenters, etc. through their services - but you need to use and understand implications of said services to leverage that.
This is so true, I wish there was a way to make this more noticeable (flashing red lights on the order forms, regular email warnings about the single point of failure,etc.). This simple fact is taught in entry level system administration courses (RAID, Disaster Recovery Plans, Replication, High Availability, Fault Tolerance, etc.) but in reality some seem to not plan for it when moving to cloud services. I always suggest people set their apps up to be able to run out of multiple availability zones in case one goes down if it is very important to them. For those that are not able to I will suggest they at least setup replication to another server within the same data center and an offsite location in case their server has hardware failure, an account gets locked or other possible situation to help take of the just in case scenarios that occur to everyone.
If your server does something weird and corrupts the filesystem, you still have access to it. That can be a huge difference when it comes to restoring the most recent data.
Isn't "non-issue" a big of an exaggeration? If the dry cleaners lost my clothes, the bank lost my money, a valet lost my car, or gmail lost my inbox I'd be angry.
Yes, but those aren't expected outcomes when using those services. That's the difference gdgtfiend was pointing out — you should expect cloud servers to sometimes go away.
They are expected outcomes. Banks fail (think 2008). Dry cleaners do lose clothes. Cars do get stolen. Plenty of stories in the news of all 3 of these. No so sure about gmail losing all your email, but I was able to accidentally access someone else gdocs once. I just logged in as me and saw a complete stranger's docs.
If the bank lost the particular dollar bill that you deposited last week, but offered you a new dollar bill, would you get angry? If you rent a car from the airport every time you visit a city, and it's always been the same car, but you show up one night and they they tell you the car was in an accident and they'll get you another car, would you be angry?
The expectation with these services you mentioned, however, is reliability.
The implicit (and explicit) volatility of cloud hosting should change that expectation with such services.
"Non-issue" is an exaggeration because it was potentially a catastrophe. But this is sort of the way these services "work." Servers/droplets/instances are ephemeral and replaceable, and their underlying data is not guaranteed in a failure.
While my first intuition was to agree with you, there's certainly an upcoming generation of developers who have never operated their own root servers and the abstraction level in the cloud nowadays is so high it makes you easily forget that "Droplets" are just VMs are just servers running software. On the other hand, hardware reliability has increased in recent years due to RAID, fully redundant networking & power adaptors, you name it.
These facts combined certainly make it easy for the younger generation to forget that redundancy and backup still don't replace each other (never did).
This seems like the wrong lesson is being taught though somewhere - the delete button is right there. It should be much more obvious that is very easy to make a server go away.
That's what AWS's "Elastic Block Storage" is. You can turn it off and just use instance storage (and I personally prefer to, for truly ephemeral nodes), but it increases spawn time since your disk image actually has to get copied over to the VM host machine in that case, rather than just "attached" over EBS.
Then why EBS failure rate is several orders of magnitude higher than in SAN deployments? A SAN provider would be quickly out of business with 0.1-0.5% annual failure rate.
Just because it's a SAN doesn't mean a given abstract block device from it is backed by RAID. It's literally just a multiplexed and QoSed network-attached storage cluster.
I actually prefer the lower-level abstraction: if you want a lower failure rate (or higher speed), you can RAID together attached EBS volumes yourself on the client side and work with the resultant logical volume.
On AWS, an EBS volume is only usable from one availability zone. You still need to use application-level replication to get geographic redundancy for important data, and when you have that, EBS just lets you be lazy rather than eager about copying a snapshot to local instances.
I guess I was thinking in terms of using EBS for ephemeral business-tier nodes, rather than as the backing store of your custom database-tier. (I usually use AWS's RDS Postgres for my database.)
For ephemeral business-tier nodes, EBS gives you a few advantages, but none of them are that astounding:
• the ability to "scale hot" by "pausing" (i.e. powering off) the instances you aren't using rather than terminating them, then un-pausing them when you need them again;
• the ability for EC2 to move your instances between VM hosts when Xen maintenance needs to be done, rather than forcibly terminating them. (Which only really matters if you've got circuit-switched connections without auto-reconnect—the same kind of systems where you'd be forced into doing e.g. Erlang hot-upgrades.)
• the ability to RAID0 EBS volumes together to get more IOPS, unlike instance storage. (But that isn't an inherent property of EBS being network-attached; it's just a property of EBS providing bus bandwidth that scales with the number of volumes attached, where the instance storage is just regular logical volumes that all probably sit on the same local VM host disk. A different host could get the same effect by allocating users isolated local physical disks per instance, such that attaching two volumes gives you two real PVs to RAID.)
• the ability to quickly attach and detach volumes containing large datasets, allowing you to zero-copy "pass" a data set between instances. Anything that can be done with Docker "data volumes" can be done with EBS volumes too. You can create a processing pipeline where each stage is represented as a pre-made AMI, where each VM is spawned in turn with the same "working state" EBS volume attached; modifies it; and then terminates. Alternately, you can have an EC2 instance that attaches, modifies, and detaches a thousand EBS volumes in turn. (I think this is how Amazon expected people would use AWS originally—the AMI+EBS abstractions, as designed, are extremely amenable to being used in the way most people use Docker images and data-volumes. The "AMI marketplace" makes perfect sense when you imagine Docker images in place of AMIs, too. Amazon just didn't consider that the cost for running complete OS VMs, and storing complete OS boot volumes, might be too high to facilitate that approach very well. Unikernels might bring this back, though.)
Not a huge company like DO, but iwStack [1] provides a SAN backed cloud, with selectable KVM/XEN instances, custom ISO and virtual network support. The prices are similar to DO.
[1] http://iwstack.com/
I think one of the reasons is that they only have a small number of datacenters, have a small number of (very friendly) staff and are not going after the mass market like DO, and I guess they don't spend anything on marketing.
> DO should do what's written in their terms of service
Agreed.
> the customer should read them carefully.
Sure, but they shouldn't need to, as Digital Ocean should set expectations clearly.
Keep in mind Digital Ocean's <title>: Simple Cloud Infrastructure. Being surprised because there's an unsafe default hidden in a document somewhere didn't work out for MongoDB and it won't work out for DO.
I worked for a company that believed that, for a while.
They bought a hugely expensive SAN solution (HP I think). One of my questions was: What happens when the SAN fails? Well you see, because it has redundancy, that can't happen. Clever uh?.
First time it failed, everything gridded to a halt for two days. The second time they did better and had it running within eight hours.
I suggest Digital Ocean should be using any mechanism that stops a single server from affecting data. A SAN backed block store allows you to provide rules for the level of redundancy required, as well as replication mechanisms for things like firmware failures, to a much greater degree than a host backed RAID array.
That's completely different from suggesting 'SANs are magical boxes incapable of losing data'. Have you considered apologising?
Servers affect data. That's kind of the whole point of servers. Maybe a multi-SAN setup with synchronous replication would have prevented the data loss. Maybe not. Not enough information has been provided to know if the cause of the data loss was the storage or the server.
But if DO had a multi-SAN setup with synchronous replication this thread wouldn't exist because DO's business model of "super cheap VPS w/ fast SSD storage" would have failed due to costs. Everybody wants Five Nines until they have to pay for it...
Digital Ocean sell 'Simple Cloud Infrastructure'. Server data might be transient, but most web developers aren't infrastructure engineers and will have no idea of this - certainly it seems to have come as a surprise to some people. Digital Ocean need to manage their customer's expectations - and upsell for people who want permanent storage - as a rude shock doesn't do well for their brand.
CAP theorem applies everywhere, but most software practices still do not assume that you must code for your app to be able to handle performance variation and partial failure state both of which are common scenarios of cloud infra.
I have to disagree - DO's "cloud servers" are equivalent to virtual private servers, which you would never expect to loose in this manner from other providers.
The lack of explanation is what worries me most - it leads me to think this might have been a case of "we forgot to replace a bad drive, then the second in the pair failed".
I remember a ~36 hour down time with a serious uk provider (in my case just a shell account, but they also housed managed and unmanaged servers). They had redundant fiber links to the NOC, from competing providers. But turns out they both ran through the same box under a high way. And then someone blew that box up in order to knock out alarms while they committed a robbery...
My family business website got killed once by a flood.
And many years ago it suffered a truck crash ( the truck crashed on the lamp post right outside the server company building and destroyed the telecom-related wires)
They regularly are unable to create new droplets [or other control issues where you can't perform normal functions with normal latency] and/or have full DC outages.
Pretty much any use case where I'd use something like DO being unable to create new VMs, etc. is the same as an outage.
I asked why as I work for DO's operations team and I wanted to know what your concerns were. Do you have a specific instance of a problem? What you're describing would be considered a MAJOR outage for us and we have not had one in quite some time.
That is annoying enough I'd want to be able to failover if it exceeded 15 minutes. I get it might not be an issue for anyone else but failing over is less complex than mitigating latency issues with droplet creation or whatever.
I get "regularly" to me might mean something different to you but if the 2 DO DCs I was using are hit literally every month with a problem of some kind...it seems regular to me.
I definitely wouldn't recommend DO's built in backup service in production based on my experience. It brings my site down every week since all I/O freezes during the backup. Sometimes the droplet doesn't recover and needs a hard reset. Apparently it's a known issue but hard to fix.
Anecdotally: Every company I've worked for that uses hardware raid controllers, it feels like the controllers break about as often as one of the disks. Sure, still an improvement over no RAID, but still ridiculous compared to any decent software raid (md or ZFS).
You lose a RAID card as often as spinning metal disks? Sure that's not a wild exaggeration? Sure sounds like one. In 15+ years of doing this, I've only been a part of a RAID card swap three times. Once was just to be safe with an overly paranoid customer.
That said, I'm a fan of moving away from RAID. It's just more complexity in the face of a movement towards simpler architectures. The one benefit of a good RAID card: stupid fast, safe writes with a BBWC.
What if you need to scale up to more servers? What if your server gets hacked and you have to recreate it? What if you accidentally delete the server or perform an upgrade that completely breaks it?
None of these would be an issue if you've scripted server recreation from scratch and permanent data is stored in high availability shared databases and services like S3. If you're relying on weekly backups to save your server state and customer data you're just asking for trouble. I like that Heroku recreates your server once a day to force you to do this properly.
How many servers? 1, 100, 1000,10,000? Where are you hosting them, in your home or in a datacenter? How much are you paying for bandwidth and power? What SLAs do you have in place for network and power? How much do you pay for your remote hands?
I hope you see what I'm getting at. While I love ZFS and think self-hosting can make a lot of economic sense, without knowing all the variables you can't really make a fair comparison.
Not surprising. DO is an epic scam. Their systems were designed by incompetent engineers and the CEO and his brother are both Coke heads. I can't even begin to tell you how many single points of failure there are in their platform.
Probably due to optimization of the conversion funnel, it's VERY easy to build things on DigitalOcean without understanding that - unlike many other *-as-a-service - droplets do not include backups and that is your responsibility. As a sysadmin, this is fine since I don't trust a single provider for both prod and backups anyhow, but I fear for those who are just getting started and don't realize this.
I agree that DO handled this situation appropriately, but they should warn people a little more deliberately when they sign up and create a droplet. A checkbox along the lines of "I, user, understand that I'm responsible for backing up my server", with a few links to some how-to articles of the excellent quality that DO is known for, would suffice and empower without reducing the conversion rate.
I would really hope that the presence of a "Backups" checkbox on the droplet creation page is enough to tell the (presumably tech-savvy) user that backups don't happen unless you check that box.
That's fine, but they didn't introduce this functionality until recently. One day, I logged into my DO console (I don't have a reason to do this very often), and saw a new backup tab. Older customers might not even be aware.
I don't host anything I couldn't live without though, so I'm not really worried.
The functionality has been there for a long time. It's possible the checkbox on droplet creation is new, and it used to require activating backups afterwards, I don't recall.
That's probably wise, I really should do the same. My important stuff is in git with a remote repository, but losing all my configuration and setup would still not be great.
SolusVM — software that powers the majority of VPS range (I would love to argue how DO isn't anything cloud) provides providers with backups out-of-the-box. It's a matter of them setting it up, and usually almost always is. If it's not, well, you're probably with someone who hasn't an idea what they're doing.
The weekly backup deal is something I really do not like with Digital Ocean, I think Linode does it better, I've been planning to move for a while now, I've just been super lazy, but this might be the push I need. 7 days of lost data is unacceptable to me, especially when I'm paying for backups.
> Three backup slots are executed and rotated automatically: a daily backup, a 2-7 day old backup, and an 8-14 day old backup. A fourth backup slot is available for on-demand backups.
I see DO's weekly backup as a convenience feature for if you need to restore your server, not as your primary backup method. Use the DO snapshot to restore most of the way, then run an up-to-date differential restore from your other (real) backup solution (that is hosted somewhere else).
But wouldn't it be much more convenient (largely because of the complexity) to have the complete backup solution at the same place (i.e. DO)?
Obviously, you want some offsite backups rather than completely relying on DO but that should be for disaster recovery (massive failures in DO's DCs, etc) rather than a more routine one.
Yes exactly besides that. What makes you feel your data is public that its better to host on a public tracker?
Because all those earlier breaches(which you are referring to) revealed customer info not their data. If this was your point of user info, I wouldn't have asked.
Agreed. When you actually need to restore something from backup, often you need a fairly recent state. Hours or days ago (and not 7 days ago).
Having something 7 days old might be useful in some cases (especially e.g. DB dump) but it is far from 'good enough'.
I am not sure what's the reason why they do not provide more recent backups (for additional fee, obviously).
Would it be so complicated that the engineering effort required wouldn't make it worth it? Or such a service wouldn't be popular enough among DO users?
I had two DO machines with a unrepairable file system after a crash and reboot. Shit happens. Never rely on a _any_ VPS. I treat them as throw-away boxes that can fail anytime.
Although I have backups of my critical files, I don't want to try to rebuild from those files if I can help it.
After learning that Digital Ocean does no backups of their own (even for critical hardware failure on their own side) I've enabled weekly backups for an additional $1 per month.
Glad you posted this so that I know that option is available and really necessary. And who wouldn't pay $1?
Edit: It costs 20% but I only pay $5 for my personal server.
If you're running in "the cloud", you should _always_ be able to destroy your instances and have them auto-rebuild from configuration management and a persistent object storage system (git repo + S3 or Google Nearline).
do you have any guides or tutorials I could possibly follow to have a setup like that? I'm working on configuring a site and would like the ability to 1) scale as quickly as possibly or 2) rebuild in event of total failure.
I just don't even know where to begin. Is this something I could stand up locally and push to like s3, spin up a new server and install 1 thing and have it pull the configs and software installs down?
Its a semi-religious topic, so you will get varied and loud opinions.
My opinion is ansible. Basically python + yml, you list out steps to set up your server. Takes a few minutes longer round 1, but you can remake the server in 60 seconds in 6 months. Lots of ansible tutorials around, try it out..
> I just don't even know where to begin. Is this something I could stand up locally and push to like s3, spin up a new server and install 1 thing and have it pull the configs and software installs down?
Yes. You could use shell scripts, Ansible, Salt, Puppet, whatever. I'm fond of Ansible or shell scripts, depending on complexity.
If you're relying on backups for servers other than your database then you're keeping state on your servers and that's a Bad Thing. You should regularly destroy your own servers and recreate them using your configuration / deployment scripts if the prospect of this happening worries you. Do it before your business starts to rely on it.
For database servers you need to have procedures in place to quickly switch the production application over to a database you just spun up and migrated the data from a recent backup to in order to test your data backups. You want this to happen as smoothly as possible in the case of a failure. Keep data backups in three different places and test your procedure on all of them.
On a production system, do not keep the database on the same server as the application.
It's so insanely easy to get started with puppet or ansible I wouldn't start rolling out a product without using them from the get-go at this point. At the very least properly package your software (read: DEB or RPM, not docker containers) so it's easy to reinstall quickly if need be.
Personally, I like platform libraries rather than OS packages. I have several Ruby projects set up as gems. What's neat about that is it allows code reuse across projects.
So go ahead and vendor them in, outside of inclusion in distribution software packages (with exceptions) there's nothing stopping you from including your own dependencies. Packaging your application properly makes deployment a hell of a lot easier and less error-prone.
Because state left on servers is inevitably unmanaged and will get lost eventually. At my job, I have cron jobs running on the production system from years ago that I have no clue what they do and no time to try to figure it out. I find out when they fail and someone, usually customer service, complains.
If the server goes away, then even once I redeploy, I've lost all those cron jobs. Who knows what will happen then.
On systems I build, cron jobs are added as part of the deployment process and managed as code, in the git repository.
That's just one example. Others include iptables rules, FTP server configuration, startup scripts, and such. If it's not in your codebase, it's unmanaged state on the server that you will lose if your cloud provider takes a shit. All of that needs to be managed as code or as configuration and deploy scripts should contain idempotent commands that add them if they are not present. I consider shelling into a server for any reason to be unideal and will think of ways to avoid it.
If you are asking about database being on the same server as the application, it's because it makes infrastructure management more difficult. Application servers are set up quickly, torn down quickly, and work as soon as you deploy. Database servers are fortresses, you don't build new ones and tear them down half as quickly, unless you've got scaling requirements.
It is tempting to want to put data on the same server as the application early on in order to save on hosting costs, with the understanding that you'll separate them later, but that's a rookie move. It's much less work to separate them now, when there's nothing riding on it, much less headache.
I think the other point of contention was your term "database". There are lots and lots of systems out there that rely on data that is not stored in a database. The data is the same kinds of things you would store in a databases but they exist as files on a filesystem instead (or in another data storage devices that is usually not referred to as a database).
That said you could easily use the same advice for that data. But in some cases the application itself needs to be on the same server as the database. Think if you were building postgres for instance...
> There are lots and lots of systems out there that rely on data that is not stored in a database. The data is the same kinds of things you would store in a databases but they exist as files on a filesystem instead (or in another data storage devices that is usually not referred to as a database).
If you can't afford to lose it, then it needs to be appropriately managed so it can be recreated in the case that it's lost. If you can afford to lose it, then it's not really state and you can forget about it.
I think people will understand more easily if instead of saying "database" specifically, you say "database tier."
I'm pretty sure the usual understanding of the terms "business tier" and "database tier" is that the "business tier" is a bunch of ephemeral nodes, and the "database tier" is where-ever the data from the "business tier" goes to be persisted.
The other reason to keep the DB separate is that they have different IO patterns that are not always complementary, and it's easier to gather diagnostic metrics if the data isn't munged. In the days of physical servers you'd also choose different hardware for the DB machine.
Configuration capture is a big thing, and it separates the adults from the kids. I've had hardware failure of my dev box twice in my career, and when I tell certain people that I have to rebuild my machine you can see their faces blanche. These are the people I know I haven't converted yet.
Everything important, except for the handful of things I'm actively investigating, is always stored on shared machines. Version control. Wiki. Some sort of artifact repository. The instructions for getting an environment set up with version X of our software are also stored in one of those places, and have been vetted by every new developer and some of the QA team. I can have my machine back up and running in a couple of hours, and most of that is waiting for downloads, if we didn't have the presence to store copies locally.
Why do I care about this stuff? Seems sort of OCD on the face of it, and maybe you're right. But sooner or later you're going to have a high severity issue in production that should be an all-hands-on-deck affair, and if you haven't done this work, losing a hard drive on a dev laptop will be the least of your worries. Everyone busy doing work on X+1 needs to be able to get back to version X and all of its dependencies in under an hour, and by themselves, because the people who could help them are most likely on the front lines of fixing the bug as fast as possible.
What's more, someone probably needs to get versions X-1 and X-2 running to figure out if you need to warn people using older versions. So that's getting people running 3 or 4 versions of your software autonomously, so that you can identify repro steps, long and short term mitigation strategies, verify that they work, formulate a bulletin and provide patches for people. Not only do you need to get your configuration captured/documented, you need it captured 2-3 versions of your software ago, which means you need to start thinking about this stuff pretty early in your project.
And the nice thing about DO and some of their cohort is that they charge by the hour for servers. Fire up new droplets and install your upgrades on them, smoke test the system, swap your routing over to the new machines, back up the old droplets (just in case you missed something) and and then destroy them.
If that takes 4 hours you've hardly spent any money at all, and you're running your cluster the way a lot of people already do.
218 comments
[ 3.0 ms ] story [ 174 ms ] threadForewarned is fore armed!
Keep it professional.
Though their should be another avenue for such things.
And when in doubt, if you're refund system is not capable of discriminating between the two, it seems to me wiser to err on the side of a less jubilant message.
At one of my positions we had stupidly put too many eggs in the basket of a single physical machine. Its disk controller failed in a way that it trashed the data volumes. I was unable to convince anyone that "move to Amazon" was not a one-step solution to "how do we make sure this never happens again".
To be fair the point of the cloud is that they deal with redundancy, HA, distributing to multiple datacenters, etc. through their services - but you need to use and understand implications of said services to leverage that.
But that doesn't stop the misconception.
The implicit (and explicit) volatility of cloud hosting should change that expectation with such services.
"Non-issue" is an exaggeration because it was potentially a catastrophe. But this is sort of the way these services "work." Servers/droplets/instances are ephemeral and replaceable, and their underlying data is not guaranteed in a failure.
These facts combined certainly make it easy for the younger generation to forget that redundancy and backup still don't replace each other (never did).
(Could be worse. Their first lessons could be like mine, in C! If anything will make you paranoid, writing C will.)
DO should be designing for failure: VM storage should be on a SAN. A single physical server failing should not cause data loss. This is basic stuff.
Does any cloud provider have SAN backed VMs? If any do, at what price?
SAN reliability ratings start at 99.999%.
I actually prefer the lower-level abstraction: if you want a lower failure rate (or higher speed), you can RAID together attached EBS volumes yourself on the client side and work with the resultant logical volume.
For ephemeral business-tier nodes, EBS gives you a few advantages, but none of them are that astounding:
• the ability to "scale hot" by "pausing" (i.e. powering off) the instances you aren't using rather than terminating them, then un-pausing them when you need them again;
• the ability for EC2 to move your instances between VM hosts when Xen maintenance needs to be done, rather than forcibly terminating them. (Which only really matters if you've got circuit-switched connections without auto-reconnect—the same kind of systems where you'd be forced into doing e.g. Erlang hot-upgrades.)
• the ability to RAID0 EBS volumes together to get more IOPS, unlike instance storage. (But that isn't an inherent property of EBS being network-attached; it's just a property of EBS providing bus bandwidth that scales with the number of volumes attached, where the instance storage is just regular logical volumes that all probably sit on the same local VM host disk. A different host could get the same effect by allocating users isolated local physical disks per instance, such that attaching two volumes gives you two real PVs to RAID.)
• the ability to quickly attach and detach volumes containing large datasets, allowing you to zero-copy "pass" a data set between instances. Anything that can be done with Docker "data volumes" can be done with EBS volumes too. You can create a processing pipeline where each stage is represented as a pre-made AMI, where each VM is spawned in turn with the same "working state" EBS volume attached; modifies it; and then terminates. Alternately, you can have an EC2 instance that attaches, modifies, and detaches a thousand EBS volumes in turn. (I think this is how Amazon expected people would use AWS originally—the AMI+EBS abstractions, as designed, are extremely amenable to being used in the way most people use Docker images and data-volumes. The "AMI marketplace" makes perfect sense when you imagine Docker images in place of AMIs, too. Amazon just didn't consider that the cost for running complete OS VMs, and storing complete OS boot volumes, might be too high to facilitate that approach very well. Unikernels might bring this back, though.)
Seems like they sell 50 GB of real SAN backed storage for just 3.6€ per month.
How can they afford to be so cheap?
Disclaimer: happy customer
Except that is not SAN backed storage.
From the website:
> We only use RAID-60 drive arrays with a minimum of 16 drives
That's somewhat scary. RAID-6 might be almost ok, but striped? No thank you. I bet they don't also have block level checksums.
Agreed.
> the customer should read them carefully.
Sure, but they shouldn't need to, as Digital Ocean should set expectations clearly.
Keep in mind Digital Ocean's <title>: Simple Cloud Infrastructure. Being surprised because there's an unsafe default hidden in a document somewhere didn't work out for MongoDB and it won't work out for DO.
They bought a hugely expensive SAN solution (HP I think). One of my questions was: What happens when the SAN fails? Well you see, because it has redundancy, that can't happen. Clever uh?.
First time it failed, everything gridded to a halt for two days. The second time they did better and had it running within eight hours.
That's completely different from suggesting 'SANs are magical boxes incapable of losing data'. Have you considered apologising?
But if DO had a multi-SAN setup with synchronous replication this thread wouldn't exist because DO's business model of "super cheap VPS w/ fast SSD storage" would have failed due to costs. Everybody wants Five Nines until they have to pay for it...
The lack of explanation is what worries me most - it leads me to think this might have been a case of "we forgot to replace a bad drive, then the second in the pair failed".
Fire happens, floods happen, electrical faults happen, mistakes happen. I can't really expect any human or humans to be perfect.
And many years ago it suffered a truck crash ( the truck crashed on the lamp post right outside the server company building and destroyed the telecom-related wires)
Server seems down :-/
I award OP the "Most Accurate Title Of The Day" Award.
Backup servers no matter where hosted, because two is one and one is none.
Pretty much any use case where I'd use something like DO being unable to create new VMs, etc. is the same as an outage.
I asked why as I work for DO's operations team and I wanted to know what your concerns were. Do you have a specific instance of a problem? What you're describing would be considered a MAJOR outage for us and we have not had one in quite some time.
* Jan 6th
* Dec 11th
* Dec 4th
* Nov 27th
* Oct 10th
* Oct 2nd
That is annoying enough I'd want to be able to failover if it exceeded 15 minutes. I get it might not be an issue for anyone else but failing over is less complex than mitigating latency issues with droplet creation or whatever.
I get "regularly" to me might mean something different to you but if the 2 DO DCs I was using are hit literally every month with a problem of some kind...it seems regular to me.
That said, I'm a fan of moving away from RAID. It's just more complexity in the face of a movement towards simpler architectures. The one benefit of a good RAID card: stupid fast, safe writes with a BBWC.
None of these would be an issue if you've scripted server recreation from scratch and permanent data is stored in high availability shared databases and services like S3. If you're relying on weekly backups to save your server state and customer data you're just asking for trouble. I like that Heroku recreates your server once a day to force you to do this properly.
I run Redundant disk controllers and hard drives on ZFS. And it's cheaper than a decent RAID system.
First thing I do with a new server is disabling RAID.
I hope you see what I'm getting at. While I love ZFS and think self-hosting can make a lot of economic sense, without knowing all the variables you can't really make a fair comparison.
I agree that DO handled this situation appropriately, but they should warn people a little more deliberately when they sign up and create a droplet. A checkbox along the lines of "I, user, understand that I'm responsible for backing up my server", with a few links to some how-to articles of the excellent quality that DO is known for, would suffice and empower without reducing the conversion rate.
I don't host anything I couldn't live without though, so I'm not really worried.
ie, it could suffer a catastrophic failure at any point in time?
http://arstechnica.com/civis/viewtopic.php?f=25&t=238085
> Three backup slots are executed and rotated automatically: a daily backup, a 2-7 day old backup, and an 8-14 day old backup. A fourth backup slot is available for on-demand backups.
Obviously, you want some offsite backups rather than completely relying on DO but that should be for disaster recovery (massive failures in DO's DCs, etc) rather than a more routine one.
Because all those earlier breaches(which you are referring to) revealed customer info not their data. If this was your point of user info, I wouldn't have asked.
Having something 7 days old might be useful in some cases (especially e.g. DB dump) but it is far from 'good enough'.
I am not sure what's the reason why they do not provide more recent backups (for additional fee, obviously).
Would it be so complicated that the engineering effort required wouldn't make it worth it? Or such a service wouldn't be popular enough among DO users?
After learning that Digital Ocean does no backups of their own (even for critical hardware failure on their own side) I've enabled weekly backups for an additional $1 per month.
Glad you posted this so that I know that option is available and really necessary. And who wouldn't pay $1?
Edit: It costs 20% but I only pay $5 for my personal server.
I just don't even know where to begin. Is this something I could stand up locally and push to like s3, spin up a new server and install 1 thing and have it pull the configs and software installs down?
Any suggestions?
My opinion is ansible. Basically python + yml, you list out steps to set up your server. Takes a few minutes longer round 1, but you can remake the server in 60 seconds in 6 months. Lots of ansible tutorials around, try it out..
user data lives in redundant databases / storage service
server setup lives in a repo via bash scripts or ansible etc (and bake that into an ami or container)
setup your launch-configuration for your servers to setup all of the above and you can destroy and spin up servers on a whim
Yes. You could use shell scripts, Ansible, Salt, Puppet, whatever. I'm fond of Ansible or shell scripts, depending on complexity.
Start here: http://www.opsschool.org/en/latest/config_management.html
EMC and Netapp are of course another matter, they're great. If you can afford them.
I'm a fan of ZFS, but this is the first time I've heard this claim. Do you have any links to articles/stories talking about it?
For database servers you need to have procedures in place to quickly switch the production application over to a database you just spun up and migrated the data from a recent backup to in order to test your data backups. You want this to happen as smoothly as possible in the case of a failure. Keep data backups in three different places and test your procedure on all of them.
On a production system, do not keep the database on the same server as the application.
If the server goes away, then even once I redeploy, I've lost all those cron jobs. Who knows what will happen then.
On systems I build, cron jobs are added as part of the deployment process and managed as code, in the git repository.
That's just one example. Others include iptables rules, FTP server configuration, startup scripts, and such. If it's not in your codebase, it's unmanaged state on the server that you will lose if your cloud provider takes a shit. All of that needs to be managed as code or as configuration and deploy scripts should contain idempotent commands that add them if they are not present. I consider shelling into a server for any reason to be unideal and will think of ways to avoid it.
If you are asking about database being on the same server as the application, it's because it makes infrastructure management more difficult. Application servers are set up quickly, torn down quickly, and work as soon as you deploy. Database servers are fortresses, you don't build new ones and tear them down half as quickly, unless you've got scaling requirements.
It is tempting to want to put data on the same server as the application early on in order to save on hosting costs, with the understanding that you'll separate them later, but that's a rookie move. It's much less work to separate them now, when there's nothing riding on it, much less headache.
That said you could easily use the same advice for that data. But in some cases the application itself needs to be on the same server as the database. Think if you were building postgres for instance...
If you can't afford to lose it, then it needs to be appropriately managed so it can be recreated in the case that it's lost. If you can afford to lose it, then it's not really state and you can forget about it.
I'm pretty sure the usual understanding of the terms "business tier" and "database tier" is that the "business tier" is a bunch of ephemeral nodes, and the "database tier" is where-ever the data from the "business tier" goes to be persisted.
Configuration capture is a big thing, and it separates the adults from the kids. I've had hardware failure of my dev box twice in my career, and when I tell certain people that I have to rebuild my machine you can see their faces blanche. These are the people I know I haven't converted yet.
Everything important, except for the handful of things I'm actively investigating, is always stored on shared machines. Version control. Wiki. Some sort of artifact repository. The instructions for getting an environment set up with version X of our software are also stored in one of those places, and have been vetted by every new developer and some of the QA team. I can have my machine back up and running in a couple of hours, and most of that is waiting for downloads, if we didn't have the presence to store copies locally.
Why do I care about this stuff? Seems sort of OCD on the face of it, and maybe you're right. But sooner or later you're going to have a high severity issue in production that should be an all-hands-on-deck affair, and if you haven't done this work, losing a hard drive on a dev laptop will be the least of your worries. Everyone busy doing work on X+1 needs to be able to get back to version X and all of its dependencies in under an hour, and by themselves, because the people who could help them are most likely on the front lines of fixing the bug as fast as possible.
What's more, someone probably needs to get versions X-1 and X-2 running to figure out if you need to warn people using older versions. So that's getting people running 3 or 4 versions of your software autonomously, so that you can identify repro steps, long and short term mitigation strategies, verify that they work, formulate a bulletin and provide patches for people. Not only do you need to get your configuration captured/documented, you need it captured 2-3 versions of your software ago, which means you need to start thinking about this stuff pretty early in your project.
If that takes 4 hours you've hardly spent any money at all, and you're running your cluster the way a lot of people already do.