They might limit IO speed (based on usage of previous 24h for example). Like how most cheaper AWS EC2 instances get their CPUs throttled after you're running CPU heavy stuff.
Unless I missed it, I saw "no need for complicated formulas to determine the overall cost for transactions or IOPs limit," which I do not read as "no IOPS limit." I was looking but could not find any word on performance.
Google doesn't charge for IOPS either, but they do explain how things are limited. For example, speed in MB/s is limited by the instance size so that you don't use the entire network link for your connection to the disk. IOPS are limited by the size of the disk. If you're sharing the disk with other people, you're going to be limited in your IOPS. For example, if you get 10GB of SSD storage, you'll get 300 random IOPS, but another user with 50GB of storage will get 1,500 IOPS because they're renting a larger portion of the disk.
If DigitalOcean isn't doing something limiting, they'll have to in order to prevent the situation that you're describing. You don't have to charge for it in order to prevent bad neighbors.
There are limits - they are marketing their simplicity by avoiding complicated pricing schemes. This is typical DO, and I see that as a good thing especially if you're just dipping your feet into the concept. The past year or two that we've had block storage at Codero we've been marketing the ability to set your own IOPS with even our smallest blocks (1000/2000/3000 iops). That said I'd be interested to know exactly what performance they offer because they are definitely cheaper.
Um. Can you provide context? What virtio setup? How much faster? How are you accessing each? This is the reason benchmarks are often worse than useless.
Edit3: Remountable/movable flexible storage for DO instances is what this gives you and it's kind of pricey. The comparison to B2 is not valid. Leaving the original mess for posterity.
---
TWENTY times (edit) the price as B2 from Backblaze ($.10 vs $.005 per GB per month). It is one of the more expensive ones. But that gets you two things:
* (moot See edit 2) SSD! (significant iops improvement)
* (moot See edit 2) No transactional costs! (not sure if just between digital ocean instances, but they say none)
Improved performance and no transaction costs MAY (edit) be worth it for some applications.
Edit1: made it an order of magnitude cheaper in my head after looking at backblaze. It is no where close to the same price. Thank you for the catch, scq!
Edit2: and I'm just all kinds of off on this. Block not object storage. Essentially storage you can mount and move between digitalocean instances. That makes no transaction costs moot. You still have to get data out of the instance.
Thank you all for quickly catching how backwards this post was! I need coffee.
Note that b2 charges 10x their cost/gb for downloads. Which means it's cheap only if you don't ever read your data back (at least in my use-case which was for ~1tb of backup data - makes it expensive to test your backups...).
We'll, "expensive" is obviously a relative term, but why store data and not access it?
Don't be confused: the article makes the mistake of comparing DOs new block storage service with other companies object stores. EBS is the competitor to this, not S3. Same for gce persistent disks and azure drives.
Unfortunately this means the pricing comparison is just wrong.
A bit pricey for the long term but great if you just need to add some disk space to your vm and don't need the other improvements more expensive vms give you.
I use DO mostly to compile stuff on Linux when i don't have access to a physical server, and storage size is always a problem.
The purpose of the block storage in this instance isn't about giving your vm/droplet more space. It's separation. That way any data that's on that device can be attached to another vm/droplet. It probably would be more cost effective just to upgrade the vm/droplet if space were a concern. It's at least how we've marketed the same feature for the past year or two via Codero's portal. Not to say I don't like how DO has entered the space: keeping it simple.
Exactly! It's about having an easy upgrade path, generally reducing the amount of work for common operations, and for more flexibility engineering your cloud architecture. It's nice that it happens to be SSD, but highest performance or lowest cost per GB isn't necessarily the only cost saving factor.
This a main reason I still use Amazon AWS: I can create an instance and if it doesn't perform, upgrade it until it does. Then when I'm finished, kill the instance and save the volume. Next time I need it, just create the instance for the job, perhaps at spot pricing, then kill it again.
This helps me with a nicer deployment setup. I was always keen on 'rebuild from scratch' rather than 'update stuff and hope you're idempotent and have captured all changes' but transitory data was always the problem. Now I can start building a new updated droplet and the only downtime will be that needed to detach and reattach the block storage containing the db etc.
Anyone see a flaw in this? (I know there are other ways to achieve similar benefits - my files could be on S3 and the database could be a separate droplet etc but these introduced various drawbacks and added complexity)
Perhaps not a flaw, but some issues with your setup are implied.
If you're rebuilding from scratch because you're not sure that you can update things, then you're probably in need of a configuration management tool (I'm a big fan of saltstack[1], mostly because I don't like Ruby or DSL's, but there's lots of options out there[2])
If you're worried you're going to lose transitory data, it sounds like you don't have a trusted and tested backup/archival/recovery process in place. So having it stored on a single EBS / DO BS / etc means you're still exposed. If you're rebuilding and rolling data over, in this scenario, I'd be copying, rather than relocating, any precarious data repositories.
I tend to avoid those config management tools other than for basic bootstrapping exactly because while you can use those too to recreate from scratch, when you don't do that, you leave the door open for undocumented, unknown state, since most of them basically take a system in an unknown-but-hopefully-mostly-consistent state and try to bring them to a known state.
But they'll only be in a known state in that case if your setup is extremely comprehensive.
In practice I've seen too many config management tools where long running servers have ended up in unknown states because changes have been applied, and subsequently changes have either been made outside of the toolchain, or changes have been done to the config in ways that doesn't let the tool know what has changed, or the tools simply doesn't have a way of comparing machine states without comprehensively enumerating everything on the server (e.g. people running Ansible playbooks that adds X, subsequently removing the requirement X from the playbook, and going about without considering whether or not X will interact with Y which they've added later).
As a result, I see rebuilding from scratch as largely orthogonal to whether or not you use a configuration management tool or e.g. build VM images that you replace wholesale, or whatever you do: You should rebuild from scratch regularly, as coupled with a test-suite it's the only realistic way of knowing whether or not you've left anything out of your build process.
My biggest caveat with config management systems is that they tend to end up encouraging live changes to a setup, instead of a build-test-deploy cycle. Sometimes that's necessary, but to me that's a last resort.
I actually agree with your overall point but generally speaking if someone is making changes outside of your standardized toolchain you have a human problem. Emergencies aside, you use those tools for a reason. Straddling the fence is almost the worst of both worlds.
Agreed. But in my experience, the easier you make going outside the process the easier it becomes to invent excuses for why it is ok. A lot of my job involves making the right thing to do the path of least resistance, because when it isn't, because humans overall tend to be the cause of a whole lot more of the problems than the servers.
1. I am already using Ansible - but I want to run my "build from scratch" playbook every time I push a change instead of my "git pull etc." playbook.
2. I'm not scared of losing data because of backups or similar - it's because I want to start from a clean droplet every time I push a change.
I do know in theory that playbooks are idempotent and running the 'build from scratch' playbook on a running instance should result in the same state as a genuine 'build from scratch' but there are many places this can fail:
1. Someone has inadvertently modified the environment at some point. A quick-fix or accidental change or an alteration in my playbook that isn't clearly reflected in the running instance
2. Not all ansible modules are perfectly idempotent. It's a fairly leaky abstraction.
3. If you never build from scratch apart from the first time then you're not really testing your automated deployment. The next time you really need to build from scratch you might get a shock.
> 1. Someone has inadvertently modified the environment at some point.
> A quick-fix or accidental change or an alteration in my playbook that
> isn't clearly reflected in the running instance
This will cause you comparable problems whether you're building from scratch or upgrading in place (iff you're using a tool like ansible). As noted by another responder, you've got a human problem there, not necessarily a technical one.
> 2. Not all ansible modules are perfectly idempotent. It's a
> fairly leaky abstraction.
I've not used ansible, but have used a few other tools, so I understand how this is (easily) possible. As in other cases, the trick, of course, is to fix this. ; )
> 3. If you never build from scratch apart from the first time then
> you're not really testing your automated deployment. The next time
> you really need to build from scratch you might get a shock.
Totally agreed.
Your original post suggested the problem you had was with transitory data - evidently your concern is more to do with the ability to consistently and reliably rebuild a known and trusted environment.
Persistent block storage won't help you with that.
What I was hoping it would do is enable me to use a single droplet but get a fast, minimal down-time 'from scratch' deployment that retains user data.
The reason I want a single droplet is mainly cost. I use one droplet per client rather than a multitenant setup. I could use S3 and a separate hosted db but for various reasons I'd prefer not to.
* Easy to add one when creating a droplet; by default they let you create volumes with these sizes: 100GB, 250GB, 500GB, 1000GB, 1.95TB; it's also really easy to create your own size.
* You can resize in any increments; took about 4 seconds to go from 100GB to 110GB with no downtime; you obviously need to resize/manage the mounted volume yourself.
* [Edit 1] Deleting the droplet does NOT destroy the volume. Worth keeping in mind when you spin them up/down.
* [Edit 2] Remounting an existing volume to a new droplet was quick and painless.
I can't help but I note that you benchmark the drive with a block size of 1024. Minimum block size of a modern SSD is 4096, anything lower would just cause unnecessary load. Also if you want to benchmark a drive/block storage I highly recommend using fio.
This is my "dd" test for any storage system. Filename can be a regular file; you can also pass --directory instead. I would love to see numbers for both this test as well as with --bs=4M.
Also experiment with --rw=rw|randrw|read :)
(and make sure your device is large enough to hold numjobs*size)
I should rephrase that, it's not "really bad" but the way that most people use it is not good. The main issue is caching from your OS, and caching from the hypervisor ( the one that runs your VM ).
There is also the problem about i/o syncing. By default OS buffer the data before writing it to disk.
That's true of all I/O benchmarks, even those which try to do things like call fsync since the lower layers often lie about whether something was committed to durable storage. The only way to do benchmarks reliably is still to make sure that you're reading/writing data which is much larger than the available cache.
In the case of dd, the real reason why it's often a misleading benchmark is that it's the best-case scenario: full-size streaming I/O where each block is only accessed once. Some applications do have that kind of operations pattern but most don't and can see very different performance characteristics because they depend on things like file creation/deletion overhead, random I/O within an open file, read and write contention if the same file is being accessed by multiple processes, etc.
probably because it can produce meaningless results. it's extremely hard to tell where which bits actually ended up, ram, or disk, or some other intermediate buffer or cache on a controller somewhere.
we use fio with a standard test config across systems for meaningful results.
Are you sure? Bonnie++ has byte-wise and block-wise tests. Yes, the byte-wise tests are CPU bound (as expected), but I have not seen that for the block-wise tests on any machine so far?
It's fundamentally objects, but not S3-style objects. It's more NASD/T10 objects, with rich semantics including partial writes and strong consistency. Implementing block storage on top of this layer (called RADOS) is trivial. Implementing S3-style objects is a bit more work, and implementing a filesystem was downright hard, but both have been done.
Disclaimer: I'm a Gluster developer, which isn't quite the same as being a Ceph developer but we do talk to one another. ;)
Really? Hrmm. Someone should let all those VPS hosters know that they're doing something impossible. Ceph has block, POSIX file, and S3/object interfaces.
Block storage (RBD) is probably the most widely used interface (judging by mailing list and IRC activity).
Since I asked what the "backend" is, my question is still valid.
Well except being 4 years late to the block storage game? Seems they have an uphill fight. Aws and gce match them on low end droplet price and offer much more. No local ssd but not sure what % of apps really need local ssd. DO effectively forces you to pay for local ssd for all of your servers.
This. I'm always a little confused by how people even evaluate the big three clouds, with prices per request and all bandwidth "not included". It's so strange coming from cheap dedicated servers with typically 10tb bandwidth/month "included". I mean, what do people do in the cloud that requires 100gb of storage, and insignificant transfer?
Sure, and certain classes of problems are cheaper on DO (webscraper, botnet, etc).
When they launched they just felt so far ahead of everyone else (1/4 the price of linode for SSD!). But price has been static for 5? years, so feeling lessing exciting these days.
Reminder that just a few weeks ago DigitalOcean rolled over on one of their customers and took down 38,000 websites after receiving a claim of infringement from the NRA against a parody site hosted on surge.sh:
Remember the old joke that I can't be a Democrat because I want to spend my own money, and I can't be a Republican because of what I want to spend my money on?
If you chuckled at it, you should check Gary Johnson out. For real.
And, assuming you don't like Hillary or Trump, the most positive thing you could do with your vote this cycle is help a 3rd party get federal funding and a national stage to promote their platform.
This is exactly why I have been voting for the Canadian green party for my entire adult life. We finally got an MP (equivalent to a senator, I think) in a few years ago, Elizabeth May. You should hear her rip into the mainstream parties in the middle of parliament, it's awesome. Harper tried passing a bunch of new laws to shut her up, but he lost an election before he could pull it off.
Reminder that you have to act on abuse notifications sharpish. You're providing a service, it's on you if you ignore abuse notifies.
"We received notice on behalf of a trademark holder that a customer of DigitalOcean was hosting infringing content on our network. DigitalOcean immediately notified our customer of the infringement, and the customer was given a five day period to resolve the issue. The infringing content was not removed within the specified period even though several notifications were issued. Per DigitalOcean’s terms of service, a final reminder was issued to our customer and, when no action was taken, access to the content was disabled. The infringing content was subsequently removed by the customer and all services were restored in less than two hours."
> The infringing content was not removed within the specified period even though several notifications were issued.
You don't have to remove content under the DMCA, you can also file a counter-notice which gets the content host off the hook and then the matter goes to court[0].
But that also assumes DMCA which, if memory serves, was not in play here. It was a trademark complaint, which DigitalOcean has no responsibility to resolve.
Ultimately DigitalOcean's response, even with that statement, seems at odds with how the law is actually written. The other party also claimed they did respond to DigitalOcean, they just never removed the legal parody material which is their right.
DigitalOcean's understanding of the NRA's rights is more expansive than the law itself. Effectively their trademark policy is to automatically side with the trademark holder, irrespective of fair use[1] (see page 9+).
It's against DO's terms of service to "use the Services in violation of the copyrights, trademarks, patents or trade secrets of third parties", which appears to be the issue here.
> Effectively their trademark policy is to automatically side with the trademark holder, irrespective of fair use[1] (see page 9+).
Did you miss that? It's not infringement if it falls under fair use. They were not following their TOS because they did not confirm the content was infringing a copyright
I don't expect that, but I do expect that a company not immediately kowtow to an infringement request if there is some ambiguity as to whether it's infringing.
That said, the original reporting on this and the statement from DigitalOcean are at odds (Motherboard's update with DO's statement), and since I haven't verified either, I'll retract any specific support for either side of this particular instance.
> I don't expect that, but I do expect that a company not immediately kowtow to an infringement request if there is some ambiguity as to whether it's infringing.
That requires money for a lawyer to evaluate it. If the customer has their own legal staff that does this and relays that opinion to the host, as well as being large enough to cover any legal costs DO might incur, DO would be fine with it.
You are basically saying you are entitled to using DO's legal staff and financial resources in addition to the hosting you've paid for.
> You are basically saying you are entitled to using DO's legal staff and financial resources in addition to the hosting you've paid for.
No, what I'm saying is that DO must already do this to some degree if they are handling requests, as otherwise I could send letters claiming trademark/copyright infringement for any number of things and get many customers shut down. If they have internal guidelines for what they do in cases when trademark/copyright infringement, I expect they follow those. I also expect that those policies do the minimum legally required of them. That's not because it's cheaper and garners good will from customers (it does), but because to do otherwise is taking sides in a legal situation without being an appointed arbiter of the law. Not only is this excessive, but it's anti-customer.
If DO is doing what they think they must by law, I have no problem with that, as long as that is clearly explained. In the case we were previously talking about, the statement from DO (at the motherboard article) is somewhat ambiguous as to why they did what they did. Per DigitalOcean’s terms of service, a final reminder was issued to our customer and, when no action was taken, access to the content was disabled. Was the take down required by law, or was DO overly aggressive in handling it? Without a statement as to why, (and I think that given some people's assertion that they went beyond what was legally required of them), their reasoning is somewhat ambiguous, and harder to call into question. If they clearly define they enforced their TOS based on what they believe is a legally required of them, then we can look at the law and their actions and evaluate whether that's true, and if it's not, DO can learn from the experience or be called out as a company that is capricious in their execution of the law.
What it boils down to is that "We received a complaint infringement. We enforced our TOS and shut down access to the content in question." leaves a lot open for assumption. I would be much happier if it was "We received a complaint infringement and as we believe is legally required of us we enforced our TOS and shut down access to the content in question." It's a small change, but it allows customers (and critics) a much clearer view on how DO handles situations like this, and allows for the public to make an informed choice on whether they think DO was correct in their actions (whether they really were legally required to do so). It's subtle, but I think it's a very, very important distinction.
I agree, but why not just take action based on the only things you have to? Namely court orders and DMCA notifications. It seems like a more complex balancing act to assume that any notification you receive is in good faith.
The primary thing commentators on HN don't acknowledge is that dealing with legal issues, whether in court or phone calls/letters, costs money and time. That takes a company away from their core objective of delivering service to their customers (in the round) and making a profit. Most companies would rather spend their money on product/customer development than hire more lawyers. But, there's a lot of low-level 'rights holders' hassle for many digital companies. A steady and constant stream of Copyright and (separately) DMCA take-down notices. When I was in a service provider we didn't think it was our job to resolve those issues (it's for the end-user site to defend whether it is or is not copyright infringing) because we were just providing a service: as a SP you're just caught in the cross-fire.
It comes down to them making a risk assessment on whether they're more likely to incur cost from taking down versus ignoring. In some cases the balancing act is due to the nature of the outcomes: on the one side if you do nothing it's unlikely (low probability) but you could land-up in court and incur millions, versus on the other side refunding a customer their annual subscription and taking a reputation hit - it's the asteroid crash problem. It's notable that they asked their customer for a response and didn't get (what they consider) a satisfactory one. Who knows the specifics in this situation, but they won't have done it just on a whim!
hence when a DMCA takedown is received, i don't see how DO is in a position to assess that something is or is not a violation. they simply follow the process outlined in the legislation.
An that process has a provision for the customer to file a dispute and this obsolves the host and sends the matter to court. Ot did you
Miss that comment?
11:02am - "you need to remove the content or fill out a counter claim"
12:47pm - "we were wrong, no counter claim can be filed and you now have 39minutes to remove the content or we are disabling the network"
"""
Then in the comments:
"""
5 days ago we got forwarded a forward of a takedown request that was sent to CloudFlare from a representative of the trademark holder. We responded to that forward within 22 minutes of first receiving it and have been in communication with Digital Ocean ever since right up until about 1.5h prior to be shut down.
As evident in the comments beneath your link, they had been in contact with DO for ~5 days at that point. Your quote leaves out important information and is misleading at best
This is EXACTLY the thing that I need for one of my droplets! I love how there is nothing "special" about it - it's just a disk that you can attach to a droplet. I'm sure that under the hood there's some kind of magic going on, but it looks like it's nicely abstracted away. This is what I hoped block storage would turn out to be - here's a block device, use it like one.
As soon as this rolls out to the region I've got that droplet in, I'm going to pull the trigger on it. I might even spend the effort to migrate my droplet to a supported region just to get this.
Same here. My $5/month droplet is sufficient for all my needs, except it's a bit limiting on storage. $2/month for an extra 20GB, doubling my storage, would be great. I don't want to migrate anything and I don't care about more CPU or RAM, I just want more space!
At the cost of significantly worse networking (VPC is fantastic), no object storage mechanism, no equivalent to IAM for instance permissioning, and no autoscaling (seriously? in 2016? I had to go look to make sure I wasn't misremembering--that they recommend something like DOProxy should make the lot of them faintly embarrassed).
DigitalOcean may have some value at the basement level of compute, but as a professional in this area there is literally no situation where I would use DigitalOcean right now because I value my time. AWS is already laughably cheap, the tooling is overwhelmingly superior, the resources available are better, and you aren't duct-taping together half-solutions and reinventing every wheel. (This is less an endorsement of AWS and more an endorsement of Not DigitalOcean; GCE is more than fine, Azure has some ugly bits around autoscaling that I don't like but you can get by.)
I have been asking for non-SSD on DO for a long time now. My heart jumped when I saw the HN title, only to be dashed on the rocks.
What are us data nerds supposed to do? We want to take 10 terabytes, run a batch process on it, keep the 20TB, then continue with about 5GB of working data until the next month's terabyte comes in, then we want to batch through the 21TB. Right now the price slider doesn't even go up to 21TB, and clicking on the "need more storage button" doesn't go anywhere, but I'm assuming it would be $2100 / month which is more than 3x as expensive than vanilla S3.
At Bytemark we've done network as standard since 2012, so we can do 100GB of higher-capacity storage for £2/month, and all discs are expandible / migrateable online: bigv.io/prices
Not being snarky, but at that scale why would you still be using Digital Ocean? Are you processing this dataset with just a single droplet?
Are you going to keep each months 10 TiB forever? Something like Nearline is going to be a much better fit even if you process the data and have to pay the $.01/GB retrieval fee.
Disclaimer: I work on compute engine (so of course I want your money), but I'm honestly curious.
All SSD all the time, we may make that our new tagline.
But yeah, we basically look to SSD for everything as that continues to take over market share from spinning disk and when you have to support a product 5-10 years, it's necessary to look ahead, so as SSD prices continue to drop we will eventually switch to SSD for everything and spinning disk will become the equivalent of tape drives in the old days.
At Hetzner, you can rent a dedicated server with 2x3TB drives for about $25/month ($0.005/GB, to scale to multiple servers, use Ceph) and a few larger machines for under $100. At OVH you can buy object storage for around 1c/GB and rent a few dedicated servers for less than $100/month, or use their cloud.
If you go to the _really_ low end, time4vps will give you a 1TB VPS for 2EUR/month, if you pay for 2 years (and they give you 4x the storage as bandwidth).
I don't work for any of these companies but I have services with all of them.
Actually because I use the DO API and lots of small instances that work together, not that long. Right now I use S3, which blows because I hate the S3 cli and API, so I would love a better solution.
You can actually create multiple volumes and then stripe them together so you should be able to get above 16TB with a bit of work.
The reason we focus on SSD exclusively for most of our services is that there are obvious performance benefits and as we look into the future the prevalence of spinning disk will continue to decrease over time.
That being said, if we are only providing SSD there are going to be customer use cases where it's not going to make sense, but that's also part of looking into the future.
So you should be able to create a larger striped volume to run through your batch processing and then depending on how you plan to access that data afterwards you can ship that off to an object store if it's going to be accessed infrequently.
If you have a lot of data that you want to ingest and store, without keeping it in an active instance, you might want to take a look at Joyent Manta[1]. We allow you to run compute jobs directly on the storage servers themselves -- basically anything you can do in a UNIX environment with a script you can run against an object in the store, from "grep" up to "python" and beyond. If you were to select "single copy" storage, the public service pricing[2] might be more interesting for you; if you grow too big or want to move onto your own hardware, the stack is open source[3] and/or we can sell you support!
I think this might be a mistake. Ever since Joyent's commentary on one of the big Amazon EBS failures in 2011 [1] [2] [3], I've been suspicious of all network-attached block storage. Then again, I haven't heard of any big EBS failures recently; I wonder what changed.
Network block storage isn't inherently broken, the initial EBS implementation was frankly just unreliable.
We've not had anything like those dark days with Persistent Disk. It's still true that having your storage across the network opens you to networking failures taking out your storage, but the gain in durability and maintenance pays for it (in our case, live migration would just be crazy with local spinning disks, we tried it didn't work).
Disclaimer: I work on GCE, and we want your business ;)
Then they'll run into the same problem anyone does doing that: migrating reasonably-sized block devices across a reasonable network takes an unreasonable length of time.
Been there, done that, don't want to sit staring at consoles waiting for live disc migrations ever again.
My guess is that they live migrate VMs only before a planned maintenance. Let's say they use a 10 Gbps network, and they use only 1 Gbps of bandwidth for migrating data, then migrating a 200 GB disk would take something like 1 hour, which sounds OK. Can you share some details about your experience?
You're missing a subtlety: it's a live migration. You've also got to migrate the data the VM's writing while the migration is going on. Depending on how your network's set up, it might well be possible for the VM to saturate out the migration process. Plus this won't only be planned maintenance, it'll be to get off dodgy hardware, too, including spannered RAID controllers where you really don't want to risk hanging around. Add in that you're likely going to be moving sets of discs at once (so possibly a couple of TB at a time), rather than individually, and you're very quickly looking at spending a day (or a night) at a time watching in case there's a network blip which means you need to restart any of them. This does not lend itself to a peaceful, happy existence.
It doesn't take doing this very often to make you realise that this is fundamentally backwards: you want the disc data already present on more than one storage server so that if one goes pop, you're not stuffed. Once you've done that, you can make the observation that hardware RAID is no longer necessary, and save yourself a layer of complexity.
Thanks a lot for your detailed answer. I understand your arguments, but Google Compute Engine Local SSDs support live migration [1], which seems to prove it's possible, despite the difficulties. Any advice about this?
Yeah, it's possible. Just not pleasant. What they're doing (and how unpleasant it is to operate) will depend on exactly what they mean by:
> Your instance might experience a short period of decreased performance
and how tolerant they are of unplanned reboots. I suspect what it means is that they'll throttle your VM (or maybe pause your IO) so it can't interfere with finalising the migration.
That being said, this is Google. They've probably thrown more man-hours at this than anyone else would think sane. I'll note that this is in the context of automatic migration away from "maintenance events" (whatever that covers) - it sounds like they think they've automated away a lot of the reasons we were having to keep an eye on things, but they're still vulnerable to hardware failure (obviously).
Too late to edit, so I'll put that up here: we do migrate your local SSD but it's best effort and it takes a while (something like 30 minutes maybe?). But for comparison, our SSDs have crazy amounts of read/write bandwidth (1 gigabyte per second) compared to a single spinning disk. And while your VM is still going, it's not that disruptive to be re-syncing an SSD, randomly seeking around a hard drive while someone is trying to use it for say streaming reads will take their reasonable performance and tank it.
tl;dr: Yes, we do this for our scratch disk SSDs, but you wouldn't want to rely on this for a guaranteed durable storage (as it'll take a really really long time to move 100 GiB of HDD).
Thank god! Highly Available Block Storage. From Digital Ocean. Great! Now I can finally store all the 300mb/s streaming in on my server. Oh wait. I cannot, because DO cancelled the service again. Bummer.
144 comments
[ 2.1 ms ] story [ 204 ms ] threadThat said, how do you prevent a rogue droplet from going crazy and hogging up all of the SSD I/O?
If DigitalOcean isn't doing something limiting, they'll have to in order to prevent the situation that you're describing. You don't have to charge for it in order to prevent bad neighbors.
https://cloud.google.com/compute/docs/disks/performance
---
TWENTY times (edit) the price as B2 from Backblaze ($.10 vs $.005 per GB per month). It is one of the more expensive ones. But that gets you two things:
* (moot See edit 2) SSD! (significant iops improvement)
* (moot See edit 2) No transactional costs! (not sure if just between digital ocean instances, but they say none)
Improved performance and no transaction costs MAY (edit) be worth it for some applications.
Edit1: made it an order of magnitude cheaper in my head after looking at backblaze. It is no where close to the same price. Thank you for the catch, scq!
Edit2: and I'm just all kinds of off on this. Block not object storage. Essentially storage you can mount and move between digitalocean instances. That makes no transaction costs moot. You still have to get data out of the instance.
Thank you all for quickly catching how backwards this post was! I need coffee.
I believe B2 is an object store, not a block store (like this). The AWS analogy is block store is EBS, object store is S3.
(The TechCrunch article also got this confused.)
Not more like amazon EFS?
We'll, "expensive" is obviously a relative term, but why store data and not access it?
https://www.backblaze.com/b2/cloud-storage-providers.html
Unfortunately this means the pricing comparison is just wrong.
"He was a bold man that first ran a production database on a brand new block storage service!"
I use DO mostly to compile stuff on Linux when i don't have access to a physical server, and storage size is always a problem.
This a main reason I still use Amazon AWS: I can create an instance and if it doesn't perform, upgrade it until it does. Then when I'm finished, kill the instance and save the volume. Next time I need it, just create the instance for the job, perhaps at spot pricing, then kill it again.
Anyone see a flaw in this? (I know there are other ways to achieve similar benefits - my files could be on S3 and the database could be a separate droplet etc but these introduced various drawbacks and added complexity)
If you're rebuilding from scratch because you're not sure that you can update things, then you're probably in need of a configuration management tool (I'm a big fan of saltstack[1], mostly because I don't like Ruby or DSL's, but there's lots of options out there[2])
If you're worried you're going to lose transitory data, it sounds like you don't have a trusted and tested backup/archival/recovery process in place. So having it stored on a single EBS / DO BS / etc means you're still exposed. If you're rebuilding and rolling data over, in this scenario, I'd be copying, rather than relocating, any precarious data repositories.
[1] https://saltstack.com/ [2] https://en.wikipedia.org/wiki/Comparison_of_open-source_conf...
But they'll only be in a known state in that case if your setup is extremely comprehensive.
In practice I've seen too many config management tools where long running servers have ended up in unknown states because changes have been applied, and subsequently changes have either been made outside of the toolchain, or changes have been done to the config in ways that doesn't let the tool know what has changed, or the tools simply doesn't have a way of comparing machine states without comprehensively enumerating everything on the server (e.g. people running Ansible playbooks that adds X, subsequently removing the requirement X from the playbook, and going about without considering whether or not X will interact with Y which they've added later).
As a result, I see rebuilding from scratch as largely orthogonal to whether or not you use a configuration management tool or e.g. build VM images that you replace wholesale, or whatever you do: You should rebuild from scratch regularly, as coupled with a test-suite it's the only realistic way of knowing whether or not you've left anything out of your build process.
My biggest caveat with config management systems is that they tend to end up encouraging live changes to a setup, instead of a build-test-deploy cycle. Sometimes that's necessary, but to me that's a last resort.
1. I am already using Ansible - but I want to run my "build from scratch" playbook every time I push a change instead of my "git pull etc." playbook.
2. I'm not scared of losing data because of backups or similar - it's because I want to start from a clean droplet every time I push a change.
I do know in theory that playbooks are idempotent and running the 'build from scratch' playbook on a running instance should result in the same state as a genuine 'build from scratch' but there are many places this can fail:
1. Someone has inadvertently modified the environment at some point. A quick-fix or accidental change or an alteration in my playbook that isn't clearly reflected in the running instance
2. Not all ansible modules are perfectly idempotent. It's a fairly leaky abstraction.
3. If you never build from scratch apart from the first time then you're not really testing your automated deployment. The next time you really need to build from scratch you might get a shock.
Your original post suggested the problem you had was with transitory data - evidently your concern is more to do with the ability to consistently and reliably rebuild a known and trusted environment.
Persistent block storage won't help you with that.
The reason I want a single droplet is mainly cost. I use one droplet per client rather than a multitenant setup. I could use S3 and a separate hosted db but for various reasons I'd prefer not to.
https://www.digitalocean.com/community/tutorials/how-to-use-...
root@ubuntu-1gb-nyc1-01:~# time dd if=/dev/disk/by-id/scsi-0DO_Volume_volume-nyc1-01 of=test.dat bs=1024 count=10000000 10000000+0 records in
10000000+0 records out
10240000000 bytes (10 GB) copied, 58.0655 s, 176 MB/s
real 0m58.248s
user 0m2.608s
sys 0m41.604s
Some quick observations:
* Easy to add one when creating a droplet; by default they let you create volumes with these sizes: 100GB, 250GB, 500GB, 1000GB, 1.95TB; it's also really easy to create your own size.
* You can resize in any increments; took about 4 seconds to go from 100GB to 110GB with no downtime; you obviously need to resize/manage the mounted volume yourself.
* [Edit 1] Deleting the droplet does NOT destroy the volume. Worth keeping in mind when you spin them up/down.
* [Edit 2] Remounting an existing volume to a new droplet was quick and painless.
I can't help but I note that you benchmark the drive with a block size of 1024. Minimum block size of a modern SSD is 4096, anything lower would just cause unnecessary load. Also if you want to benchmark a drive/block storage I highly recommend using fio.
Also experiment with --rw=rw|randrw|read :)
(and make sure your device is large enough to hold numjobs*size)
There is also the problem about i/o syncing. By default OS buffer the data before writing it to disk.
In the case of dd, the real reason why it's often a misleading benchmark is that it's the best-case scenario: full-size streaming I/O where each block is only accessed once. Some applications do have that kind of operations pattern but most don't and can see very different performance characteristics because they depend on things like file creation/deletion overhead, random I/O within an open file, read and write contention if the same file is being accessed by multiple processes, etc.
we use fio with a standard test config across systems for meaningful results.
I did not get good speeds and I am wondering why that may be...
I think you might have misread the units: locally attached 50105 us (50 ms) vs block 546 ms.
The throughput numbers are on par with AWS and GCP block storage. This seems reasonable aside from the high latency.
Disclaimer: I'm a Gluster developer, which isn't quite the same as being a Ceph developer but we do talk to one another. ;)
Block storage (RBD) is probably the most widely used interface (judging by mailing list and IRC activity).
Since I asked what the "backend" is, my question is still valid.
If I had to pick two, those would be them!
With DO, at least there is an additional option available for users.
With GCE and AWS, the outbound bandwidth is expensive. 1TB = $90 (AWS, GCE) vs 1TB included (Linode, DO).
When they launched they just felt so far ahead of everyone else (1/4 the price of linode for SSD!). But price has been static for 5? years, so feeling lessing exciting these days.
http://motherboard.vice.com/read/nra-complaint-takes-down-38...
Get out and vote for the right people this November.
If you chuckled at it, you should check Gary Johnson out. For real.
"We received notice on behalf of a trademark holder that a customer of DigitalOcean was hosting infringing content on our network. DigitalOcean immediately notified our customer of the infringement, and the customer was given a five day period to resolve the issue. The infringing content was not removed within the specified period even though several notifications were issued. Per DigitalOcean’s terms of service, a final reminder was issued to our customer and, when no action was taken, access to the content was disabled. The infringing content was subsequently removed by the customer and all services were restored in less than two hours."
> The infringing content was not removed within the specified period even though several notifications were issued.
You don't have to remove content under the DMCA, you can also file a counter-notice which gets the content host off the hook and then the matter goes to court[0].
But that also assumes DMCA which, if memory serves, was not in play here. It was a trademark complaint, which DigitalOcean has no responsibility to resolve.
Ultimately DigitalOcean's response, even with that statement, seems at odds with how the law is actually written. The other party also claimed they did respond to DigitalOcean, they just never removed the legal parody material which is their right.
DigitalOcean's understanding of the NRA's rights is more expansive than the law itself. Effectively their trademark policy is to automatically side with the trademark holder, irrespective of fair use[1] (see page 9+).
[0] https://en.wikipedia.org/wiki/Digital_Millennium_Copyright_A...
[1] https://apps.americanbar.org/litigation/committees/intellect...
https://www.digitalocean.com/legal/terms/
Did you miss that? It's not infringement if it falls under fair use. They were not following their TOS because they did not confirm the content was infringing a copyright
The vast majority will not unless you are a large customer with your own legal staff on retainer to provide the appropriate legalese/notices/etc.
That said, the original reporting on this and the statement from DigitalOcean are at odds (Motherboard's update with DO's statement), and since I haven't verified either, I'll retract any specific support for either side of this particular instance.
That requires money for a lawyer to evaluate it. If the customer has their own legal staff that does this and relays that opinion to the host, as well as being large enough to cover any legal costs DO might incur, DO would be fine with it.
You are basically saying you are entitled to using DO's legal staff and financial resources in addition to the hosting you've paid for.
No, what I'm saying is that DO must already do this to some degree if they are handling requests, as otherwise I could send letters claiming trademark/copyright infringement for any number of things and get many customers shut down. If they have internal guidelines for what they do in cases when trademark/copyright infringement, I expect they follow those. I also expect that those policies do the minimum legally required of them. That's not because it's cheaper and garners good will from customers (it does), but because to do otherwise is taking sides in a legal situation without being an appointed arbiter of the law. Not only is this excessive, but it's anti-customer.
If DO is doing what they think they must by law, I have no problem with that, as long as that is clearly explained. In the case we were previously talking about, the statement from DO (at the motherboard article) is somewhat ambiguous as to why they did what they did. Per DigitalOcean’s terms of service, a final reminder was issued to our customer and, when no action was taken, access to the content was disabled. Was the take down required by law, or was DO overly aggressive in handling it? Without a statement as to why, (and I think that given some people's assertion that they went beyond what was legally required of them), their reasoning is somewhat ambiguous, and harder to call into question. If they clearly define they enforced their TOS based on what they believe is a legally required of them, then we can look at the law and their actions and evaluate whether that's true, and if it's not, DO can learn from the experience or be called out as a company that is capricious in their execution of the law.
What it boils down to is that "We received a complaint infringement. We enforced our TOS and shut down access to the content in question." leaves a lot open for assumption. I would be much happier if it was "We received a complaint infringement and as we believe is legally required of us we enforced our TOS and shut down access to the content in question." It's a small change, but it allows customers (and critics) a much clearer view on how DO handles situations like this, and allows for the public to make an informed choice on whether they think DO was correct in their actions (whether they really were legally required to do so). It's subtle, but I think it's a very, very important distinction.
It comes down to them making a risk assessment on whether they're more likely to incur cost from taking down versus ignoring. In some cases the balancing act is due to the nature of the outcomes: on the one side if you do nothing it's unlikely (low probability) but you could land-up in court and incur millions, versus on the other side refunding a customer their annual subscription and taking a reputation hit - it's the asteroid crash problem. It's notable that they asked their customer for a response and didn't get (what they consider) a satisfactory one. Who knows the specifics in this situation, but they won't have done it just on a whim!
i'm under the impression that "fair use" is a defense and/or judge's assessment, rather than something "civilians" can factually declare.
https://news.ycombinator.com/item?id=12010521
"""
11:02am - "you need to remove the content or fill out a counter claim"
12:47pm - "we were wrong, no counter claim can be filed and you now have 39minutes to remove the content or we are disabling the network"
"""
Then in the comments:
"""
5 days ago we got forwarded a forward of a takedown request that was sent to CloudFlare from a representative of the trademark holder. We responded to that forward within 22 minutes of first receiving it and have been in communication with Digital Ocean ever since right up until about 1.5h prior to be shut down.
"""
Believe who you will.
Edit: added 2nd quote from comments
As evident in the comments beneath your link, they had been in contact with DO for ~5 days at that point. Your quote leaves out important information and is misleading at best
I also haven't forgotten about their private blog censorship debacle.
DO is not a serious player and should not be trusted.
As soon as this rolls out to the region I've got that droplet in, I'm going to pull the trigger on it. I might even spend the effort to migrate my droplet to a supported region just to get this.
https://aws.amazon.com/ebs/pricing/
DigitalOcean may have some value at the basement level of compute, but as a professional in this area there is literally no situation where I would use DigitalOcean right now because I value my time. AWS is already laughably cheap, the tooling is overwhelmingly superior, the resources available are better, and you aren't duct-taping together half-solutions and reinventing every wheel. (This is less an endorsement of AWS and more an endorsement of Not DigitalOcean; GCE is more than fine, Azure has some ugly bits around autoscaling that I don't like but you can get by.)
https://www.online.net/en/dedicated-server/rpn-san
There are several places you can get ~5 TB for +/-10% of the 1TB price at DO.
DO is offering a SAN at Object Store prices. :/
What are us data nerds supposed to do? We want to take 10 terabytes, run a batch process on it, keep the 20TB, then continue with about 5GB of working data until the next month's terabyte comes in, then we want to batch through the 21TB. Right now the price slider doesn't even go up to 21TB, and clicking on the "need more storage button" doesn't go anywhere, but I'm assuming it would be $2100 / month which is more than 3x as expensive than vanilla S3.
Are you going to keep each months 10 TiB forever? Something like Nearline is going to be a much better fit even if you process the data and have to pay the $.01/GB retrieval fee.
Disclaimer: I work on compute engine (so of course I want your money), but I'm honestly curious.
But yeah, we basically look to SSD for everything as that continues to take over market share from spinning disk and when you have to support a product 5-10 years, it's necessary to look ahead, so as SSD prices continue to drop we will eventually switch to SSD for everything and spinning disk will become the equivalent of tape drives in the old days.
At Hetzner, you can rent a dedicated server with 2x3TB drives for about $25/month ($0.005/GB, to scale to multiple servers, use Ceph) and a few larger machines for under $100. At OVH you can buy object storage for around 1c/GB and rent a few dedicated servers for less than $100/month, or use their cloud.
If you go to the _really_ low end, time4vps will give you a 1TB VPS for 2EUR/month, if you pay for 2 years (and they give you 4x the storage as bandwidth).
I don't work for any of these companies but I have services with all of them.
Price calculation is straightforward:
- $0.005/GB/month for storage
- $0.05/GB download
- $0.004 per 10k downloads
There is a base free tier as well.
https://www.backblaze.com/b2/
The reason we focus on SSD exclusively for most of our services is that there are obvious performance benefits and as we look into the future the prevalence of spinning disk will continue to decrease over time.
That being said, if we are only providing SSD there are going to be customer use cases where it's not going to make sense, but that's also part of looking into the future.
So you should be able to create a larger striped volume to run through your batch processing and then depending on how you plan to access that data afterwards you can ship that off to an object store if it's going to be accessed infrequently.
You've got to remember that block storage is >> faster than s3/http. Its also has significantly lower latency.
Also if you want fast processing you need to start looking at shared filesystems/NFS
Disclaimer: I work at Joyent.
[1]: https://www.joyent.com/manta
[2]: https://www.joyent.com/pricing/manta
[3]: https://github.com/joyent/manta
This is EXACTLY the thing I need for some stuff I'm working on!
[1]: https://www.joyent.com/blog/on-cascading-failures-and-amazon...
[2]: https://www.joyent.com/blog/magical-block-store-when-abstrac...
[3]: https://www.joyent.com/blog/network-storage-in-the-cloud-del...
We've not had anything like those dark days with Persistent Disk. It's still true that having your storage across the network opens you to networking failures taking out your storage, but the gain in durability and maintenance pays for it (in our case, live migration would just be crazy with local spinning disks, we tried it didn't work).
Disclaimer: I work on GCE, and we want your business ;)
It looks like Exoscale does live migrations with locally attached SSDs.
Been there, done that, don't want to sit staring at consoles waiting for live disc migrations ever again.
It doesn't take doing this very often to make you realise that this is fundamentally backwards: you want the disc data already present on more than one storage server so that if one goes pop, you're not stuffed. Once you've done that, you can make the observation that hardware RAID is no longer necessary, and save yourself a layer of complexity.
[1] https://cloud.google.com/compute/docs/disks/#data_persistenc...
> Your instance might experience a short period of decreased performance
and how tolerant they are of unplanned reboots. I suspect what it means is that they'll throttle your VM (or maybe pause your IO) so it can't interfere with finalising the migration.
That being said, this is Google. They've probably thrown more man-hours at this than anyone else would think sane. I'll note that this is in the context of automatic migration away from "maintenance events" (whatever that covers) - it sounds like they think they've automated away a lot of the reasons we were having to keep an eye on things, but they're still vulnerable to hardware failure (obviously).
tl;dr: Yes, we do this for our scratch disk SSDs, but you wouldn't want to rely on this for a guaranteed durable storage (as it'll take a really really long time to move 100 GiB of HDD).
Will post results.