We have always tried to squeeze costs out of storage. That required a ton of tech/ops work. Bandwidth it turns out isn't that expensive - seems to just typically be overpriced. Hopefully lowering the price will enable people to do more with their data.
I'd love to hear what you haven't done in the past due to bandwidth/download pricing that you'd have wanted to...?
No feedback from me at the moment, other than to say that what y'all are doing with B2 is awesome. It's great to see more competition in this space, and I love the open-source ethos you've cultivated at Backblaze.
What software do you use to actually do the backups/manage them? (Wondering if we'd need to work with someone else to integrate to make that easier for you.)
Pretty sure any decent B2 CLI client supports Linux.
But for regular backup, this bit hard. I ended up switching to Crashplan for not only Linux support but also support for external hard drives. Then again the official Crashplan client is GUI-only and AFAIK closed-source, so you can't run it on a headless server
As a not incredibly impressed customer of Crashplan, I switched to rclone/duplicati and Amazon Cloud Drive for these and other reasons. The price is the same, and you can use whatever client you want.
Can't speak for the others but I use duplicati and send the backups to my unlimited google drive. I like that it encrypts and gives you the key vs some app saying they encrypt it.
* borg: deduplicating backup repository. I used this recently. It's cool, but the performance is disappointing. It's single-threaded. I got into the GitHub a little bit and they seem committed to their current way of doing things, which makes it hard for an external multithreaded compressor to take over (everything is processed in 2 MB chunks).
* rclone: Meant to provide an rsync-like interface to any cloud backend
* duplicity: Automatically performs encrypted, compressed delta-enabled backups, but requires some upfront configuration. Been using this for years.
* rdiff-backup: Haven't used, but it seems more like an ad-hoc backup utility for "I want to keep an old version of this folder". Duplicity seems like a better choice for disk-scale backups.
Backblaze supports external hard drives. You just have to keep them plugged in. If they aren't plugged in for 30 days, Backblaze will erase the backup - but gives you a warning.
I also ended up switching to CrashPlan. My reasons were Linux support, and the ability to back up to another computer running CrashPlan instead of cloud only. I hate their GUI.
However, we have a sizeable Elasticsearch snapshot in S3, and it's getting expensive. We'd love to use B2 instead, but there's currently no Elasticsearch plugin for B2.
Running Elasticsearch would require compute, which we don't have. However, how often do you run analysis on the data? Would it make sense to move the data to B2 and put back on s3 when it's needed?
Not sure what you mean that running Elasticsearch requires compute. I'm talking about the snapshot/restore functionality in Elasticsearch. We use it mostly for backups. As far as I can tell, it should be quite possible to implement a B2 plugin for that (Azure, GCS, HDFS, and S3 are available currently).
Got it. Let me look into that. Agree that if you're just taking a snapshot from Elasticsearch and putting into B2, that should be an implementalbe plug-in.
I'm a backup customer of Backblaze (have been for years) and I backup five of my families computers.
I'd really like it if your online restore feature would let me buy multiple backups on a single hard drive as an option. I've considered buying a hard drive with my backups from your company multiple times but I wouldn't pay the premium you ask without being able to have all five computers on the drive.
I feel like gp wants to keep a hdd as a third copy of the backup somewhere safe under their physical control.
This is prohibitively expensive to do for multiple computers at once, and gp would prefer a single hdd that has a copy of the backup for multiple computers, which would reduce the cost.
Thanks for the suggestion (and for being a customer.) We've contemplated adding functionality to send the Backblaze laptop/desktop restore to your Backblaze B2 account. Then you could put multiple restores in a 'snapshot' and get that delivered on a drive. Useful?
I'd like to permanently, easily offload all image storage & delivery (consumer oriented services) - without being mangled by the obscene bandwidth costs of AWS. I don't want to roll my own solution to get cheap bandwidth, I want a dependable service that hosts files (images in this case) & can serve them up, and do so at very cheap bandwidth prices.
B2 references Vintage Aerial as a success case for storing their large image collection. So I checked out their site, pretty cool service. So I right click -> view image, on the large & thumbnail images on their site and they're on AWS of course (archival storage on B2 I assume).
Hey Backblaze, AWS's bandwidth margin is your opportunity.
Both AWS and Backblaze have had (and will have more) outages. No single vendor is a good choice for a service that absolutely positively must stay up no matter what.
I want to use your cloud storage instead of S3. My main reason for using S3 is the DURABILITY (not reliability). S3 offers 99.999999%, which means it is replicated numerous times. Which means I DON'T HAVE TO WORRY ABOUT taking extra customer backups, it is "baked" into S3.
...but in a single day datacenter, which realistically drops several "9"s from that guarantee when you consider tail events.
I like b2 and am using it for backup in some projects, but unfortunately it's still not an appropriate choice as a primary data store.
The common counterargument I've heard is that you should have multiple redundant storage providers that you manage at the application layer anyway. Problem with that is, there are a near-infinite number of things I "should" do, but very finite resources to do them. S3 backs up across multiple availability zones by default, so using it means one less thing I have to spend brain cycles worrying about.
Yep - I agree with you 100%, just pointing out what their page says. I'd love to see B2 get more data centers around the world - when we have to do a 500GB restore on a server in Germany, it's already slow even pulling from S3 - with B2 it would be even slower.
Depends. But if there are rules about keeping it in-country, you should consider storing it not at a storage provider.
Mostly, it is important to keep the data in the EER. So germany is fine. Moving data out of the EU is a big issue, although I have been using B2 but encrypt my data first myself.
I was suitable impressed with their description of the singular (I believe) data center they own. Even though it is in an area that is at risk for a major earthquake it seems to be at the upper end of engineering specifications.
Having said that, if your data is absolutely critical then I agree that even further geographic diversity would be desirable. Most likely you want to avoid the Pacific Rim, and volatile countries. A central European site/company would be good for this. Obviously you should expect to pay more to store your data in more than one place.
> if your data is absolutely critical then I agree that even further geographic diversity would be desirable
I think geographic distribution is absolutely critical if you need "uptime". But for pure durability I think multiple geographic regions is a red herring. What you need much more than multiple geographic regions is having your data stored in two profoundly different formats with two profoundly different vendors with source code written by two separate teams. Furthermore, I would pay for those with two separate credit cards.
The same software bug (or lack of payment resulting in Backblaze B2 or Amazon S3 deleting ALL copies of your data) will delete your data flawlessly across geographic regions. Every time.
Backblaze carefully chose our main datacenter to not be in an earthquake zone, and on a hill (flood proof), and not in tornado zone, and no hurricanes. I suppose that in the case of a meteor strike on a datacenter or terrorist attack some geographic regions would help, but that is really diminishing returns compared with software or human error.
If you truly value your data, two regions won't save you -> two vendors will.
The thing I haven't done due to the download pricing is back up very much data into B2. Since the B2 API doesn't support renames, you have to download and then re-upload a file to rename it. I would be storing a lot more there if renames were available! [Note: they may be available now; I haven't checked for a few months. Please tell me they are.]
I was an early adopter of B2 and uploaded a bunch of files with naming that's incompatible with the "virtual folders" of the web UI (wrote my own script to handle this since Duplicity et al did not yet have support). Now I'm stuck and the only solution offered is just "pay us to download everything again".
Also, the web interface does not have any pagination. Since virtual folders are based on naming, and since rename support is not available, this can flood the browser. I have a bucket that crashes the tab every time I try to open because it's not paged. I could write a little Greasemonkey injector to page it myself, but meh, I don't really care enough to do that. Instead, I just don't really use B2.
When I discussed these issues with support and suggested that at least pagination on the web interface was a basic thing that engineering should be able to accomplish quickly, they did not seem to agree.
Interesting. "rename" is generally not supported by object storage, but an API to "copy with a new name & delete" often is, which allows the same result. I presume that would work? (We're planning to add that in the future.) Pagination is on the list as well...
Both of these things seem like such weird things to avoid adding, especially pagination. I can't be the only user to have a bad experience with the web UI because of this. How hard is it to add a simple pager? It doesn't have to be fancy.
I'd be interested to know, have you looked into reducing/eliminating bandwidth prices further for clients close to your network? It seems like you're with Unwired, which has heavy inbound traffic and peers at a number of facilities in the Bay Area. It'd be really interesting to know if you could, for example, offer free traffic to Linode, DigitalOcean or Vultr, all of which have offerings in the area and could help balance out Unwired's traffic flow (I think that's a thing ISPs care about).
The big thing missing from B2 for me now, relative to the other big players, is the ability to process my data once it's stored with you. If I store my images in S3 for example, I can use EC2 to thumbnail them without paying anything for retrieval. At the moment I can't do that with B2 and it seems unlikely to make sense for you to build your own compute cloud but perhaps you can integrate with others.
B2 is a competitor of long-term storage solutions like Glacier or TarSnap, not virtualized storage like S3. It's meant to be almost 100% write-only. Basically, you pull your backups down in case of disaster.
Backblaze has a backup service for laptops/desktops, but our B2 service is actually more like S3 - the data is available without delays or penalties for access. (Unlike Glacier.)
Yeah, it's similar to Google Nearline or TarSnap in that unlike Glacier, the data can be read out instantly instead of waiting for Amazon to restore it off of a tape or whatever they do. The "penalty for access" is the download cost.
However, my understanding is that B2 was intended for long-term storage and not everyday cloud storage. Is this understanding incorrect? Backblaze does take the position that B2 should be considered as a competitor for S3?
Up to this point, my experience with B2 has been that they don't want to be the authoritative storage source for the data. I seem to recall reading that B2 should not be used as primary storage (though that was in beta days). In support discussions, a representative expressed surprise when it was implied that I may not have local copies of files that already exist on B2 (I do indeed have local copies, but it was beside the point) before confirming that yes, I would need to redownload and reupload to complete a rename.
> my understanding is that B2 was intended for long-term storage...
Long term storage is a great use of B2, but it was never "intended" as anything other than the ability to reliably store blobs of data and retrieve them really fast whenever and how ever often you like. I suppose there are different use cases that work well for B2 and others that don't work well, but that is up for customers to decide, not Backblaze.
> ... they [Backblaze?] don't want to be the authoritative storage source for the data
Who told you that? Backblaze wants people to use B2 for whatever they want. Backblaze makes an AMAZINGLY GOOD authoritative storage source.
> I would need to redownload and reupload to complete a rename
Right now the B2 service does not include a "rename" function. If that would be useful to customers, we can add it? We collect customer requests and add them in the order that we have time and what gets the most customer votes.
>Long term storage is a great use of B2, but it was never "intended" as anything other than the ability to reliably store blobs of data and retrieve them really fast whenever and how ever often you like. I suppose there are different use cases that work well for B2 and others that don't work well, but that is up for customers to decide, not Backblaze.
I guess that the pricing structure of free uploads and expensive downloads is the main thing that caused it to be compared to Glacier and other long-term storage solutions like Nearline. Reducing the download cost is an attempt to get more people to see B2 as an S3 replacement, right?
>Who told you that? Backblaze wants people to use B2 for whatever they want. Backblaze makes an AMAZINGLY GOOD authoritative storage source.
Like I said, I must be recalling discussion from the time the beta was announced, and that such restrictions were shed post-beta. Did Backblaze ever expand into its promised second datacenter?
>Right now the B2 service does not include a "rename" function. If that would be useful to customers, we can add it? We collect customer requests and add them in the order that we have time and what gets the most customer votes.
Yes, this made me stop using B2 for all practical intents and purposes. It's such an easy thing that it doesn't seem like there's any reason it shouldn't be supported. Between the web UI's lack of pagination and the inability to rename a file to comply with virtual folders, the web interface is not usable.
> Reducing the download cost is an attempt to get more people to see B2 as an S3 replacement, right?
B2's download cost was already cheaper than AWS for everyone but the largest customers but I expect a little guy has to try harder.
I think something that has contributed to seeing B2 as mainly for backup, aside from it being a service of a backup company, is greater emphasis on durability rather than availability. And looking at web pages now, B2's availability SLA is 99.9% [0] while S3's is 99.99% [1] (S3 Infrequent Access is 99.9%).
> Reducing the download cost is an attempt to get more people to see B2 as an S3 replacement, right?
A price drop MIGHT open new markets to us. What those markets are are kind of an internal debate at Backblaze. What we agree on is that we priced outbound bandwidth incorrectly when we launched B2. We know this for two reasons:
A) The price Backblaze pays bandwidth is WAAAAAY lower than 5 cents per GByte.
B) Almost nobody is choosing to use Backblaze as a CDN or website server or a large source of outbound data.
One new market might be making B2 more attractive as a CDN or website server. It is my understanding that CDN customers use the CDN because of two things: 1) cheap download prices and 2) fault tolerance and reliability and distribution all over the world.
Most CDNs charge about 2 cents/GByte or maybe a little more.
If B2 outbound bandwidth prices drop below what CDNs charge, then SOME of their customers might consider B2 as one part of their solution.
In my humble opinion we are still overpriced, but the reason we're dropping prices slowly is because we're slightly terrified of becoming too attractive too fast and not being able to handle the onslaught as the world abandons all the other CDNs and goes with B2 since we are so much cheaper. So we're going slowly at first while we ramp up, and then we can drop prices much more in the future as we gain comfort with serving up gigantic amounts of cached data.
> Did Backblaze ever expand into its promised second datacenter?
In the process of. Servers are sitting in the new datacenter powered up and talking with each other (and even have customer data on them), but we do not yet let customers choose which datacenter their data lands in (or choose to duplicate their data between).
I would love to do this. Cross-connects aren't free, but they are certainly less expensive than going out across the Internet. And enabling compute/storage use cases by (possibly) partnering with other providers would be excellent. I'll reach out to a few of those compute providers and explore.
Can you tell me more about how you use/would use cloud storage? (with or without compute)
Currently I don't really use cloud storage, it's cheaper for me to colocate a 2U stuffed with disks and buy transit.
Primarily though I store media. I have (for a home user) large amounts of high resolution (lossless audio, RAW images, high bitrate (even ProRes) video) media that I like to keep for archival and transcoding but don't need for playback. I want to store the original media somewhere safe and cheap but have the ability to transcode it to a format suitable for playback/streaming on demand.
Actually it just occurred to me that it might be worth you talking to the folks at Plex (https://plex.tv) about this too, as they're working on their Cloud product and need more places for their users to store media.
If I was a storage company, I would consider Plex a hot potato and would stay far, far away. There is a reason Plex isn't building out their own storage and the reason isn't cost. It's because they know that 99% of all the media their customers handle is pirated content. At some point, Plex will become large enough to get on the radar of the MPAA and at that point, I assume they will go after the cloud storage providers.
I feel like it'd be the opposite. DMCA protects the storage provider as long as they're unaware that they're storing infringing content and Plex limits sharing enough that any infringement is unlikely to reach the MPAA/RIAA.
I think inking a deal with Plex would be pretty obvious. I'm not sure where the line is for knowing that you are storing infringing content, but MEGA claims they didn't know what users were uploading.
Plex itself is a US company and were it "obvious" that its existing Cloud product was infringing on copyrights, it would have already been the target of a lawsuit.
Yeah, I'm not sure about that. Plex has nothing to do with how users are getting the content, it just offers playback and cataloging after they've already retrieved it.
I'd agree if they allowed something like library sharing between users, but I don't see them as a likely target just for allowing playback of consumer media.
I use a (local) Plex installation for home videos, downloaded clips, and various other non-infringing things. It's the most convenient way to share the library out to a Chromecast/phone/Roku.
Ya, I don't think Plex would be necessarily at fault, although they could be susceptible to DMCA takedown notices. But a storage provider could get into deep trouble if they know tons of pirated content was being uploaded. Kim Dotcom claims MEGA had no idea if the content being uploaded was infringing.
Megaupload (which I assume you mean because afaik MEGA has not been in legal trouble) was a distribution platform; the purpose was to upload something and give the link to others. If it was a personal file locker that couldn't link out or be shared, as Plex intends to be, it probably would not have had issues.
I can give you my example -- I have an open source backup tool called Snebu which uses an sqlite DB to store the backup catalog (file names, file hash, metadata etc), whereas the individual files are stored in a vault area (currently just a disk volume, with files named after the sha1 hash). Backup data is sent to it by piping in tar format to the main process, which updates the catalog DB and deposits files into the storage vault.
I'd like to add cloud capabilities, but for that I'd have to run the backend process somewhere that can easily update the DB, and easily read/write the data in the vault, along with syncing the full DB to the vault location after a backup.
My other option is to run everything locally, and compute a delta file between versions of the DB and send that delta over to the vault (i.e., do D2D2C backups).
Edit: Another possible requirement -- I'd like to pack multiple smaller files into an archive file, and deposit that into a bucket. But as files age out of the archive, I'd like to be able to move active ones out of it and into a new one. Right now, I'd have to re-create the still-valid backup data out of on-hand information, upload the new archive, then send a command to delete the old one.
Speaking for myself, I'm excited about using B2 as a replacement for more expensive local disk in not-heavy-iops-role machines. It's not the most exotic use, but if you look at every vm provider (linode, digital ocean, even amazon) the prices of local disk are huge in comparison to the prices of cloud storage like B2 & S3.
By combining them I can have a cheap box (Linode for instance) with a local write cache, with B2 providing a terabyte of raw storage, and get a slow but effective VM with a 1TB "local disk" for under $20 month. Which is great.
Unless you're going to be doing pretty much entirely sequential reads of entire files, I don't think this is likely to work well for you. B2 (and other cloud storage) isn't just priced by bandwidth but by "transactions" (requests).
It's true, there is a price for some API requests, but it's actually quite low if you're using the cloud storage to populate a local cache! Backblaze categorizes download as a "class B" transaction with 2500 free per day and $0.004/10k. I fully admit it's not a suitable solution for many workloads, but I've been playing with a VM using B2 as slow additional disk (as an additional volume, not suggesting putting / or /usr..) and am yet to exceed the free tier.
It doesn't need to be a specific provider, those were just examples. The idea is to have free traffic between Backblaze and someone that provides compute, so you can transform the data without fetching all of it. I suggested those in particular because I believe they're all in the SF bay area and should be present at one of Unwired's existing peering points.
The reason I can't see traffic just being straight free for everyone is that for Backblaze to make traffic free, it needs to make arrangements with the providers to reduce costs (for example by connecting directly rather than through the internet).
Backblaze does not charge anything (zero) for the bandwidth of the b2_list_file_names response. What we mean for the bandwidth is "file contents you download". So compressing or not compressing b2_list_file_names won't make any difference in your bill.
We do change a TINY "per transaction" charge on api calls like b2_list_file_names. The reason is to break even on the farm of servers (we call them "API Servers" internally) that are needed to support the HTTPS load generated from those calls. But that is really really tiny, like you get 1,000 calls for 4/10ths of one penny.
I like B2. And thanks for "absorbing" the cost of the list() function.
My sincere question to you is:
Sending this much redundant data in the response can cost YOU a lot, it is not "trivial" when you add millions of requests. How hard is it to allow the user to specify which fields he wants in the response?
Also, please answer my other question about the authorize() token headache.
> How hard is it to allow the user to specify which fields he wants in the response?
Pretty easy, but like all things we prioritize features based on customer requests. Seriously, you could probably game it by creating 20 different gmail addresses and emailing us once every three days from a new email address requesting the same feature. :-) If we hear from something like 20 different customers the same request we prioritize it higher and get it done. The OTHER way is if a "potentially large" customer approaches us and says they require some feature or they cannot integrate or won't use B2. I won't name the customer for privacy reasons, but we added the ability to put the SHA1 at the END of the upload instead of at the start in the HTTP headers because one "important" customer required it. It also makes sense, the headers come FIRST and some streaming applications don't have the SHA1 until all the data streams through them, thus at the end of the stream.
> redundant data in the response can cost YOU a lot
Much to my surprise (totally being honest), bandwidth only costs Backblaze about 2% or 3% of our total operating budget (including salaries and datacenter). Modern bandwidth is CHEAP! But even more interesting is that most of our Online Backup customers and B2 customers pretty much only UPLOAD data, but we are forced to purchase symmetric bandwidth (same upload as download). So hilariously enough, outbound bandwidth is utterly free to us until it comes up to match the inbound bandwidth. Free. I've even tried to come up with a scheme where we make it free to our customers up until the moment we actually have to increase our capacity to support outbound, but I can't figure out a scheme to make that happen. So we are pricing it for the situation when we exceed the "shadow", even though it is nowhere near (and probably never will) exceed the shadow.
> please answer my other question about the authorize() token headache
I looked back and found one question you asked about 24 hours vs weeks vs forever. If that is the question, the 24 hours is a "guideline" because so many customers kept asking what to expect so we made up a number. In reality, you have to follow the protocol which is: if a call is ever rejected for whatever reason with a "expired_auth_token" or "bad_auth_token" then your code should re-authorize. It's source code that you have to write and it doesn't matter how often it is triggered, after you write it (correctly) then it will always occur when it needs to occur, and never occur when it doesn't need to occur. To give you an insight as to why, if we happen to reboot a particular server (or it crashes, or the power supply dies), then all the auth codes will be incorrect and all currently connected customers that used that authentication server need to re-authorize which will come from one of the OTHER authentication servers. We do weekly code updates on Thursdays at 2pm-ish so if you get an authentication token on Friday it will probably work for 6 days, but if you get one Thursday morning it might fail once in the middle of the day. This is all about cost reduction for Backblaze - we simply don't purchase ANY high end load balancers, it is all cheap computers that are expected to fail and break, but because there are hundreds (or thousands of them), even if half of them have lost power or died there are still plenty up and running (now we have some in a different datacenter!) to respond to your software's requests. We don't own a single load balancer (like F5), and we pass the savings on to you. Unfortunately we also pass along a small amount of additional complexity.
So it is really straight-forward code and you write it once and it just executes every time it needs to and never executes when it doesn't need to. Your code should not have ANY assumptions of time in it. Unfortunately it isn't something where you c...
* each file name is guaranteed to map to a unique content (content-addressable)
* some of the files to upload might already be in the bucket
It's actually quite expensive to list all of the bucket files to find out if they haven't been uploaded. Especially when the file names are hashes it's hard to filter by prefix as well. (same issue on S3)
It would be great if there was a way to just upload the files and get an early 303 response if the entity already exists and has the same SHA1. That way the client can just push all the files without having to build the "new files" list.
I care less about the download pricing for backups, and more about the storage pricing. I wish you had a comparison for your storage versus Glacier storage, because all of the claims that it is "cheaper than S3" just seem non-credible when I factor Glacier into the picture. It is true that Glacier is a different service from classic S3, but you can't just ignore it. Or at least some of us can't. It does exist. I wonder how it compares. If you don't have a chart, can you tell us here?
I understand download pricing can get complicated. But again I'm not asking about that. I realize download pricing also is important, but right now I'm just asking about storage pricing. Not sure if you are answering questions here but if yes, please share what you know.
In short, unless you only upload very large individual files, do nothing with them, and don't access any of them for at least 10 years...B2 is lower cost. (And, of course, doesn't required you to wait to get them out.)
Edit: BTW it's really a breath of fresh air to see someone whose product is covered hanging around for a while answering questions. So many people in your position either don't show up on their HN stories at all, or do so only for an hour or so and then leave a ton of questions unanswered.
Probably my number one wish list feature for hacker news would be if they could somehow coax people at companies that get linked to return more, including beyond the first hour or first day.
I hope you check back on this story later and answer more questions that come in from different time zones.
Yev from Backblaze Here -> My #1 feature request would be notifications or something a-la reddit that would tell me if someone replied to me. Scrolling through these is tough!
Our primary colo is SunGard in Rancho Cordova, California. I believe we are currently provisioned for about 220 Gbits/sec (in this one facility alone). I'm not completely sure what this would top out at if we asked for more, but it is a colocation facility with a bunch of network providers such as Cogent, GTT, Century Link, etc. I kind of expect they can provide anything we ask for?
> what is my likely MB/s at the top-performing location
I have three answers, you can choose one. :-)
1) If you want to download a file once after a very long time of not accessing it (meaning it is not in the "B2 cache"), and with one thread, it must be fetched from the vaults to the "caching layer". Spooling a single file off of the vaults with one thread is relatively slow at about 15 Mbits/sec.
2) If the file is already in the caching layer, then it is fetched from local SSDs already inside the caching servers. Each individual caching server has a 10 Gbit/sec network card so if you only have one thread then you are absolutely limited to 10 Gbit/sec (and realistically the caching servers are not completely idle so a realistic number might be only 3 or 4 Gbit/sec).
3) If you want to fetch many files with many threads, fetch one file at a time with each thread. I'm not sure how many HTTPS threads an EC2 node can handle, maybe 200 safely? If it isn't in the cache then you get 200 * 15 Mbits/sec = 3,000 Mbits/sec = 375 MBytes/sec. If it is coming out the cache even faster?
This is unlike S3, which simply needs a STATIC API key, and boom, you can access the API. I don't know why you did this, but it is bad developer UX. Is it because tokens are "cool". I see no need to have this concept of tokens for your API. Why I am annoyed:
1) An API operation now requires one ADDITIONAL call. B2 already requires 2 calls to upload a file, and this now means 3 calls!
2) Related to above, this means bad latency. The roundtrip can be significant, especially since you only have 1 data center in California. My users will NOT be happy.
3) My code is much more complex now.
4) It is more work FOR YOU. This means more processing. More resources wasted FOR YOU. More cost. Why would you do this? You are supposed to be lean. Imagine millions of extra calls EVERY day. Adds up.
Is there a solution to this? Can the token be cached? If so, for how long.
Also, would love to see a reasonable rationale for this? If there is no rationale, then please provide a STATIC approach to access your API.
I REALLY like B2, and I wish they could think of developer UX a little more.
If you look at their API you'll see that the token can be cached for up to a week. I doubt that having a check if the token is up to date will add much complexity.
"-if an API call failed, I have to check for an additional error: a invalid token"
In general it's considered good practice to always check return values and error codes. Not for all code, but just for the code you would like to work.
@loxias. I do not understand your response. Of course one should check error codes. My point is: now there's one more thing that can go wrong and needs to be checked.
> -I have to do date arithmetic, what if my server clock is incorrect,
How is your clock inaccuracy in any way the responsibility of a service provider (Backblaze or other)? From certificate expiry to inaccurate logging, and so on, complaining about "what if my clock is wrong, I have to date math in my app!" is precisely "solving the wrong problem".
@FireBeyond Did you even read my reasoning? All I'm saying is that there is ADDITIONAL complexity know in relation to the other, simpler static key approach.
Not shifting blame on Backblaze for my server clock being potentially inaccurate.
Uh, what's your complaint? Why in the world would you assume that it has to have the same API as S3? I also don't understand your comments about "cool". You need an auth token so that the subsequent API calls, well, know you're authentic... Makes perfect sense to me.
Yes, the first API call requires authenticating first but every additional call reuses cached credentials, so no change to latency.
I've been using Backblaze to write a for-fun app, using c++ and libcurl, for the last few weeks and I've found it to be wonderful from a developer perspective.
Their documentation is short and sweet. They have capabilities to help you stress test your implementation to make sure you handle edge cases! (you can pass an extra flag and it will randomly send errors) That is so amazingly helpful! And to top it off, their support staff is quick and responsive even with technical details. _And I don't even have a paid account_
If you find dealing with this to make your code "much more complex", then as they say "my consulting rates are very reasonable, my email is in my profile". :P
(Disclaimer: I do not work for Backblaze, have never given them a dime of my money. So far.)
Somewhere on their site, they address this issue. It is part of the reason they are able to offer cheaper prices. By having to request the token first, then upload, this allows them to avoid using load balancers. The less equipment they have to have, the cheaper your storage can be.
> Backblaze requires an authorize() call to obtain a token before calling any API function.... this now means 3 calls to upload a file
That is correct if you only plan to upload one file. The "authorize" can be reused for weeks, and so can the upload URL. So if you want to upload 100,000 files, it is 100,002 calls to upload the 100,000 files (give or take).
It is true this is slightly more complex than it could be, but in turn this means we don't need to have any load balancers in our datacenter (which in turn makes the cost of the service cheaper and at the same time enormously more easily scalable). The idea here is that your application authenticates, then you ask for a URL to a vault that has spare space on it. Then your application "hangs up" (stops talking with that server) and contacts the vault directly. This means there is zero "chatter" between our vaults and that you are talking directly (no hops) to the vault you are storing your data on.
If you use b2_list_file_versions (or any of the b2_list functions) there are checksums included in that list (SHA1).
When you ask for that file list, the results are actually coming out of a large Cassandra cluster (database with many computers and fast SSDs).
Ok, so even if you do not request a file list of any kind, Backblaze itself automatically is always walking over the entire data farm checking every single last file's SHA1, plus checking if the files match the database's SHA1. A full sweep completes in less than a month then begins again.
If any bit error is detected (which happens semi-regularly since we have over 72,000 hard drives and bits get flipped) then we rebuild the files from the other copies (Reed-Solomon encoded across many separate computers in many separate locations in our datacenter).
We love and take care of your data so you don't have to.
Thanks for the information! Though I'd like to be able to initiate the sweeping process for my files more often, maybe you can charge for such requests?
@Gleb: I'm from Brazil and if you don't know already bandwidth here is priced like gold, so I just can't ignore the price you are offering.
The service I provide has, currently, an average of 60k http requests daily, approximately 10Gb. Those requests are images stored on Google Cloud Storage. I'm going to offer a new service for photographer to create online galleries, by some calculations I had made the average daily requests will increase around 10x, growing up to 600K daily.
My main questions about B2 are:
1) Is there any use case you know of someone uploading files to B2 directly from the browser? I know this is possible, but as per I know I would have to open tokens and url uploads on JS code, which, obviously, is not cool (and I won't do that). I can do this on S3 and GCS, their API offers the possibility to create a key on the server and use it on the webpage, but I didn't see anything like that on B2.
2) About http requests, is it ok to have this amount of requests (600k) in a daily basis? I mean I know B2 is excellent for backups but I'm not sure if it will be good serving websites that have a big number of images.
I tried it a while back, maybe it was just me being stupid but I found no CLI, or decent tool for the API. Ended up giving up because the barrier to entry was too high. I'd suggest they work on tooling.
Here is a sample from that web page:
CloudBerry, Rclone (mount B2 as a hard drive on your desktop), Synology (this is a NAS drive), Retrospect, Cantemo, Cubix, Cyberduck, Duplicacy, Duplicity Linux, HashBackup, NyNAS, NeoFinder, Odrive, Qbackup, SmartFTP
I've tried it but ultimately the cost of storage outweighs the benefit to bandwidth assuming infrequent data access.
OVH is also entirely located in Canada and France so performance between much of the US and their services isn't fantastic.
Also notable is that unlike many of the other cloud providers, OVH doesn't seem to offer free bandwidth between their services. You can't spin up an OVH server to thumbnail a bunch of images for example as far as I can tell.
Have they added any non-credit/debit card payment options (e.g. PayPal) yet? I'm from the UK, and my bank charges me a flat 'non-sterling transaction fee', which overshadows the actual usage costs. This means, for smaller projects, B2 is actually much more expensive than the competition that allows me to pay in my native currency either directly, or through a service like PayPal.
Another option worth considering is a pre-paid debit card like Monzo's [1]. This doesn't have any fees when used outside of the UK.
I think in the not too distant future you'll be able to get a regular (non-prepaid) debit card from them now that they have a unrestricted UK banking license [2].
Ah, thanks - the Halifax Clarity card seems like it would be the best for me, especially as it's from a company I recognise.
I previously tried Revolut (https://revolut.com) to avoid fees, and from having to apply for another credit card. Whilst great in theory, and it works fine for in-person purchases abroad, too many online services decline it.
Is this the cheapest place for storing another copy of your photos? I have 150GB worth atm with 3 copies and I don't mind another copy if it's cheap enough.
You can get full quality on google but you have to be on an unlimited plan for it to make sense. Despite their marketing, a single account gsuite account doesn't appear to have a cap.
Brian from Backblaze here -> I don't quite understand your math?
$0.004/GByte/month - Amazon Glacier
$0.005/GByte/month - Backblaze B2
Furthermore, I think if you price it out, Backblaze will be cheaper than Glacier in most examples. For example, Glacier charges for uploads (5 cents per 1,000 uploads). If you ever download the data, it costs 9 cents/GByte to retrieve it from Glacier(!!!)
Use anything you like, but just make sure you consider all the charges and give Backblaze a fair consideration.
I'm not entirely sure I'm reading this correctly (because Amazon pricing is so ridiculously confusing) but I think you are reading the prices to get data out of Glacier and into an EC2 instance (which is still inside Amazon). THEN you get charged an additional 9 cents/GByte to get it out of EC2 and downloaded onto your laptop.
At Backblaze, we cannot figure out the business case of why Amazon prices things so that the small customers get really expensive prices (9 cents/GByte for less than 10 TBytes), and as the volume increases the price drops to 5 cents/GByte (for 250 TBytes). It makes no business sense to drive away small companies. A byte is a byte, the WHOLE POINT of a large cloud storage farm like Backblaze B2 or Amazon S3 is that we bundle together 200,000 small customers and together they get the economy of scale of a large customer! That's the whole point, the whole strategy! And what Backblaze has seen over and over again is that over time, some of our small customers became really gigantic customers. So driving away the small customers by gouging them on price means the small customers go to some other provider, then they get large with the other provider. But hey, I'm just thankful Amazon thinks this is good for them because it leaves the market wide open for Backblaze to gobble up all the small customers (which we are really super happy to have as customers).
Backblaze is doing to S3 what I dream about doing to Amazon Retail after several years working for them.
The amazon philosophy is quite a bit like the underpants gnomes: 1) grow, 2) ..., 3) profit. Where there are natural monopolies with obvious economies of scale, this is an incredible strategy. But outside of that (retail, for example), it just doesn't work. Economies of scale come coupled with diseconomies of scale, and #2 needs to be an intentional and deliberate approach to managing those diseconomies of scale. If you look at their financials, it is obvious that Amazon retail has had a miserable time managing this...over the last 15 years they have steadily lower inventory turns, steadily increasing logistics costs, availability inconsistency, etc. Anybody who works there now or worked there in the past can see how inefficiently they run things and how efficiency improvements one day are ameliorated by political and speculative decisions the next day.
Combine this with their typical acquisition strategy, as exemplified by the Quidsi approach. Lowball, and when they don't accept, spend money subsidizing your own product to undercut their market and make them more desperate to sell.
The Backblaze approach is genius for taking on Amazon because it inverts the priorities to 1) meticulously manage costs, and then 2) directly engage amazon by competing on price. Backblaze can compete on price because their cost structure is lower than Amazon. Amazon can always subsidize their product, but by doing so Amazon is in the weaker position in the endurance race. Backblaze can ride out the attempts at undercutting them because they have a lower cost structure and prices need to drop a lot before they affect their cash flow safety. It's also a resilient strategy; Amazon can know that's exactly what they're doing and they can't do anything to stop it apart from beating them at their own game of cost management. It's a slow and steady road, but it can work.
Now, if only they would start running more than an S3 competitor :)
> Now, if only they would start running more than an S3 competitor
But then Backblaze don't have the cost advantage! Amazon uses their profits from S3/EC2 to subsidise all the new AWS services (and feature development for the existing services).
I don't know much about Amazon's retail business, but you're spot-on about us focusing on meticulously managing costs. Sometimes people ask how we do it and the hardest part to explain is that the culture of the company is built around that.
In a nutshell, any one byte of data is spread across 20 different computers in 20 different physical locations in our datacenter through Reed-Solomon encoding. If we lose 3 full computers in 3 separate locations your data is still fine.
We have been doing this for 10 years now, and we take it very seriously, and we have 73,000 hard drives in our datacenter. We are the only company on earth to publish our drive failure rates: https://www.backblaze.com/blog/hard-drive-benchmark-stats-20...
I'm looking for the cheapest gigabit based NAS solution that supports B2. Any suggestions?
I'm thinking about replacing my aging Core 2 Duo laptop running Plex with an attached USB hard drive and BackBlaze with a combination NAS with Gig E and an NVidia Shield. I would use B2 for backups.
I have gigabit internet up and down and a house wired for Gig e throughout.
If you're looking for a NAS, Synology boxes support Backblaze B2 directly via Cloudsync. I know some of their models support Gig E; not sure if all do.
Gleb: Understanding that it's not a CDN per se, would using this primarily for large file distribution (meaning, not images on web sites, etc.) instead of backup be a bad idea?
Should be a fine use case. B2 is fine for backup, but designed to be used for all sorts of storage use cases. Would be happy to have someone on our team explore with you the specifics of your use case (performance, etc.); You can shoot me an email at gleb.budman at backblaze.
I asked if you guys can offer a similar price to OVH's prices and got rejected. After 3 weeks I moved literally 48 million objects to OVH, you guys decide to announce this. Cool. I hope all goes well.
I've been begging the pricing team at Backblaze to drop this price for about a year. I'm just glad it finally percolated through.
As far as we can tell, all the cloud storage providers are gouging their customers on outbound bandwidth prices. I'm sorry we couldn't get this pushed through sooner.
155 comments
[ 223 ms ] story [ 2862 ms ] threadI'd love to hear what you haven't done in the past due to bandwidth/download pricing that you'd have wanted to...?
Gleb @ Backblaze
Maybe now I can consider it for my 10tb offline backups.
But for regular backup, this bit hard. I ended up switching to Crashplan for not only Linux support but also support for external hard drives. Then again the official Crashplan client is GUI-only and AFAIK closed-source, so you can't run it on a headless server
* zbackup: Appears to be unmaintained, superseded by attic/borg.
* attic: deduplicating backup repository. Doesn't seem actively maintained anymore either.
* borg: deduplicating backup repository. I used this recently. It's cool, but the performance is disappointing. It's single-threaded. I got into the GitHub a little bit and they seem committed to their current way of doing things, which makes it hard for an external multithreaded compressor to take over (everything is processed in 2 MB chunks).
* rclone: Meant to provide an rsync-like interface to any cloud backend
* duplicity: Automatically performs encrypted, compressed delta-enabled backups, but requires some upfront configuration. Been using this for years.
* rdiff-backup: Haven't used, but it seems more like an ad-hoc backup utility for "I want to keep an old version of this folder". Duplicity seems like a better choice for disk-scale backups.
However, we have a sizeable Elasticsearch snapshot in S3, and it's getting expensive. We'd love to use B2 instead, but there's currently no Elasticsearch plugin for B2.
I'm a backup customer of Backblaze (have been for years) and I backup five of my families computers.
I'd really like it if your online restore feature would let me buy multiple backups on a single hard drive as an option. I've considered buying a hard drive with my backups from your company multiple times but I wouldn't pay the premium you ask without being able to have all five computers on the drive.
This is prohibitively expensive to do for multiple computers at once, and gp would prefer a single hdd that has a copy of the backup for multiple computers, which would reduce the cost.
B2 references Vintage Aerial as a success case for storing their large image collection. So I checked out their site, pretty cool service. So I right click -> view image, on the large & thumbnail images on their site and they're on AWS of course (archival storage on B2 I assume).
Hey Backblaze, AWS's bandwidth margin is your opportunity.
AWS picked the first two. B2 picked the second two. I don't think they would make a great CDN host.
> AWS picked highly available, redundant > Backblaze picked redundant, and cheap
Both AWS and Backblaze have had (and will have more) outages. No single vendor is a good choice for a service that absolutely positively must stay up no matter what.
We liked this article that shows how to configure Amazon S3, Microsoft Azure, or Backblaze B2 so you are not susceptible to Amazon outages: https://cloudrail.com/avoid-next-aws-s3-outage-failover-scen...
I want to use your cloud storage instead of S3. My main reason for using S3 is the DURABILITY (not reliability). S3 offers 99.999999%, which means it is replicated numerous times. Which means I DON'T HAVE TO WORRY ABOUT taking extra customer backups, it is "baked" into S3.
How is the durability for B2?
https://www.backblaze.com/b2/cloud-storage.html
"Reliability: 99.999999% durability."
I like b2 and am using it for backup in some projects, but unfortunately it's still not an appropriate choice as a primary data store.
The common counterargument I've heard is that you should have multiple redundant storage providers that you manage at the application layer anyway. Problem with that is, there are a near-infinite number of things I "should" do, but very finite resources to do them. S3 backs up across multiple availability zones by default, so using it means one less thing I have to spend brain cycles worrying about.
Restricting it to Germany would seem anti-competitive on an EU level, which is usually not allowed. But I don't know the laws.
Their main disadvantage is that they only have one datacenter, unless they have gotten a second one recently.
Having said that, if your data is absolutely critical then I agree that even further geographic diversity would be desirable. Most likely you want to avoid the Pacific Rim, and volatile countries. A central European site/company would be good for this. Obviously you should expect to pay more to store your data in more than one place.
> if your data is absolutely critical then I agree that even further geographic diversity would be desirable
I think geographic distribution is absolutely critical if you need "uptime". But for pure durability I think multiple geographic regions is a red herring. What you need much more than multiple geographic regions is having your data stored in two profoundly different formats with two profoundly different vendors with source code written by two separate teams. Furthermore, I would pay for those with two separate credit cards.
The same software bug (or lack of payment resulting in Backblaze B2 or Amazon S3 deleting ALL copies of your data) will delete your data flawlessly across geographic regions. Every time.
Backblaze carefully chose our main datacenter to not be in an earthquake zone, and on a hill (flood proof), and not in tornado zone, and no hurricanes. I suppose that in the case of a meteor strike on a datacenter or terrorist attack some geographic regions would help, but that is really diminishing returns compared with software or human error.
If you truly value your data, two regions won't save you -> two vendors will.
Never thought of that. It is costly but still, great advice!
The thing I haven't done due to the download pricing is back up very much data into B2. Since the B2 API doesn't support renames, you have to download and then re-upload a file to rename it. I would be storing a lot more there if renames were available! [Note: they may be available now; I haven't checked for a few months. Please tell me they are.]
I was an early adopter of B2 and uploaded a bunch of files with naming that's incompatible with the "virtual folders" of the web UI (wrote my own script to handle this since Duplicity et al did not yet have support). Now I'm stuck and the only solution offered is just "pay us to download everything again".
Also, the web interface does not have any pagination. Since virtual folders are based on naming, and since rename support is not available, this can flood the browser. I have a bucket that crashes the tab every time I try to open because it's not paged. I could write a little Greasemonkey injector to page it myself, but meh, I don't really care enough to do that. Instead, I just don't really use B2.
When I discussed these issues with support and suggested that at least pagination on the web interface was a basic thing that engineering should be able to accomplish quickly, they did not seem to agree.
The big thing missing from B2 for me now, relative to the other big players, is the ability to process my data once it's stored with you. If I store my images in S3 for example, I can use EC2 to thumbnail them without paying anything for retrieval. At the moment I can't do that with B2 and it seems unlikely to make sense for you to build your own compute cloud but perhaps you can integrate with others.
However, my understanding is that B2 was intended for long-term storage and not everyday cloud storage. Is this understanding incorrect? Backblaze does take the position that B2 should be considered as a competitor for S3?
Up to this point, my experience with B2 has been that they don't want to be the authoritative storage source for the data. I seem to recall reading that B2 should not be used as primary storage (though that was in beta days). In support discussions, a representative expressed surprise when it was implied that I may not have local copies of files that already exist on B2 (I do indeed have local copies, but it was beside the point) before confirming that yes, I would need to redownload and reupload to complete a rename.
> my understanding is that B2 was intended for long-term storage...
Long term storage is a great use of B2, but it was never "intended" as anything other than the ability to reliably store blobs of data and retrieve them really fast whenever and how ever often you like. I suppose there are different use cases that work well for B2 and others that don't work well, but that is up for customers to decide, not Backblaze.
> ... they [Backblaze?] don't want to be the authoritative storage source for the data
Who told you that? Backblaze wants people to use B2 for whatever they want. Backblaze makes an AMAZINGLY GOOD authoritative storage source.
> I would need to redownload and reupload to complete a rename
Right now the B2 service does not include a "rename" function. If that would be useful to customers, we can add it? We collect customer requests and add them in the order that we have time and what gets the most customer votes.
>Long term storage is a great use of B2, but it was never "intended" as anything other than the ability to reliably store blobs of data and retrieve them really fast whenever and how ever often you like. I suppose there are different use cases that work well for B2 and others that don't work well, but that is up for customers to decide, not Backblaze.
I guess that the pricing structure of free uploads and expensive downloads is the main thing that caused it to be compared to Glacier and other long-term storage solutions like Nearline. Reducing the download cost is an attempt to get more people to see B2 as an S3 replacement, right?
>Who told you that? Backblaze wants people to use B2 for whatever they want. Backblaze makes an AMAZINGLY GOOD authoritative storage source.
Like I said, I must be recalling discussion from the time the beta was announced, and that such restrictions were shed post-beta. Did Backblaze ever expand into its promised second datacenter?
>Right now the B2 service does not include a "rename" function. If that would be useful to customers, we can add it? We collect customer requests and add them in the order that we have time and what gets the most customer votes.
Yes, this made me stop using B2 for all practical intents and purposes. It's such an easy thing that it doesn't seem like there's any reason it shouldn't be supported. Between the web UI's lack of pagination and the inability to rename a file to comply with virtual folders, the web interface is not usable.
B2's download cost was already cheaper than AWS for everyone but the largest customers but I expect a little guy has to try harder.
I think something that has contributed to seeing B2 as mainly for backup, aside from it being a service of a backup company, is greater emphasis on durability rather than availability. And looking at web pages now, B2's availability SLA is 99.9% [0] while S3's is 99.99% [1] (S3 Infrequent Access is 99.9%).
[0] https://www.backblaze.com/b2/cloud-storage.html
[1] https://aws.amazon.com/s3/
A price drop MIGHT open new markets to us. What those markets are are kind of an internal debate at Backblaze. What we agree on is that we priced outbound bandwidth incorrectly when we launched B2. We know this for two reasons:
A) The price Backblaze pays bandwidth is WAAAAAY lower than 5 cents per GByte.
B) Almost nobody is choosing to use Backblaze as a CDN or website server or a large source of outbound data.
One new market might be making B2 more attractive as a CDN or website server. It is my understanding that CDN customers use the CDN because of two things: 1) cheap download prices and 2) fault tolerance and reliability and distribution all over the world.
Most CDNs charge about 2 cents/GByte or maybe a little more.
If B2 outbound bandwidth prices drop below what CDNs charge, then SOME of their customers might consider B2 as one part of their solution.
In my humble opinion we are still overpriced, but the reason we're dropping prices slowly is because we're slightly terrified of becoming too attractive too fast and not being able to handle the onslaught as the world abandons all the other CDNs and goes with B2 since we are so much cheaper. So we're going slowly at first while we ramp up, and then we can drop prices much more in the future as we gain comfort with serving up gigantic amounts of cached data.
> Did Backblaze ever expand into its promised second datacenter?
In the process of. Servers are sitting in the new datacenter powered up and talking with each other (and even have customer data on them), but we do not yet let customers choose which datacenter their data lands in (or choose to duplicate their data between).
Can you tell me more about how you use/would use cloud storage? (with or without compute)
Primarily though I store media. I have (for a home user) large amounts of high resolution (lossless audio, RAW images, high bitrate (even ProRes) video) media that I like to keep for archival and transcoding but don't need for playback. I want to store the original media somewhere safe and cheap but have the ability to transcode it to a format suitable for playback/streaming on demand.
Actually it just occurred to me that it might be worth you talking to the folks at Plex (https://plex.tv) about this too, as they're working on their Cloud product and need more places for their users to store media.
I'd agree if they allowed something like library sharing between users, but I don't see them as a likely target just for allowing playback of consumer media.
I use a (local) Plex installation for home videos, downloaded clips, and various other non-infringing things. It's the most convenient way to share the library out to a Chromecast/phone/Roku.
I'd like to add cloud capabilities, but for that I'd have to run the backend process somewhere that can easily update the DB, and easily read/write the data in the vault, along with syncing the full DB to the vault location after a backup.
My other option is to run everything locally, and compute a delta file between versions of the DB and send that delta over to the vault (i.e., do D2D2C backups).
Edit: Another possible requirement -- I'd like to pack multiple smaller files into an archive file, and deposit that into a bucket. But as files age out of the archive, I'd like to be able to move active ones out of it and into a new one. Right now, I'd have to re-create the still-valid backup data out of on-hand information, upload the new archive, then send a command to delete the old one.
By combining them I can have a cheap box (Linode for instance) with a local write cache, with B2 providing a terabyte of raw storage, and get a slow but effective VM with a 1TB "local disk" for under $20 month. Which is great.
The reason I can't see traffic just being straight free for everyone is that for Backblaze to make traffic free, it needs to make arrangements with the providers to reduce costs (for example by connecting directly rather than through the internet).
A crosspromotion deal between Linode and Backblaze would be fantastic.
I'm about to sign up for the B2 Cloud Storage, just noticed a tiny typo in the "Keep costs down" section on https://www.backblaze.com/b2/cloud-storage.html:
"...the fist 1 GB of downloads per day are free."
If I have a 10000 files, this will be a LOT of bandwidth. I JUST want the list of file names, NOT the other redundant fat. Solution:
1) let me specify which fields I want OR 2) compress the response
Backblaze does not charge anything (zero) for the bandwidth of the b2_list_file_names response. What we mean for the bandwidth is "file contents you download". So compressing or not compressing b2_list_file_names won't make any difference in your bill.
We do change a TINY "per transaction" charge on api calls like b2_list_file_names. The reason is to break even on the farm of servers (we call them "API Servers" internally) that are needed to support the HTTPS load generated from those calls. But that is really really tiny, like you get 1,000 calls for 4/10ths of one penny.
I like B2. And thanks for "absorbing" the cost of the list() function.
My sincere question to you is:
Sending this much redundant data in the response can cost YOU a lot, it is not "trivial" when you add millions of requests. How hard is it to allow the user to specify which fields he wants in the response?
Also, please answer my other question about the authorize() token headache.
> How hard is it to allow the user to specify which fields he wants in the response?
Pretty easy, but like all things we prioritize features based on customer requests. Seriously, you could probably game it by creating 20 different gmail addresses and emailing us once every three days from a new email address requesting the same feature. :-) If we hear from something like 20 different customers the same request we prioritize it higher and get it done. The OTHER way is if a "potentially large" customer approaches us and says they require some feature or they cannot integrate or won't use B2. I won't name the customer for privacy reasons, but we added the ability to put the SHA1 at the END of the upload instead of at the start in the HTTP headers because one "important" customer required it. It also makes sense, the headers come FIRST and some streaming applications don't have the SHA1 until all the data streams through them, thus at the end of the stream.
> redundant data in the response can cost YOU a lot
Much to my surprise (totally being honest), bandwidth only costs Backblaze about 2% or 3% of our total operating budget (including salaries and datacenter). Modern bandwidth is CHEAP! But even more interesting is that most of our Online Backup customers and B2 customers pretty much only UPLOAD data, but we are forced to purchase symmetric bandwidth (same upload as download). So hilariously enough, outbound bandwidth is utterly free to us until it comes up to match the inbound bandwidth. Free. I've even tried to come up with a scheme where we make it free to our customers up until the moment we actually have to increase our capacity to support outbound, but I can't figure out a scheme to make that happen. So we are pricing it for the situation when we exceed the "shadow", even though it is nowhere near (and probably never will) exceed the shadow.
> please answer my other question about the authorize() token headache
I looked back and found one question you asked about 24 hours vs weeks vs forever. If that is the question, the 24 hours is a "guideline" because so many customers kept asking what to expect so we made up a number. In reality, you have to follow the protocol which is: if a call is ever rejected for whatever reason with a "expired_auth_token" or "bad_auth_token" then your code should re-authorize. It's source code that you have to write and it doesn't matter how often it is triggered, after you write it (correctly) then it will always occur when it needs to occur, and never occur when it doesn't need to occur. To give you an insight as to why, if we happen to reboot a particular server (or it crashes, or the power supply dies), then all the auth codes will be incorrect and all currently connected customers that used that authentication server need to re-authorize which will come from one of the OTHER authentication servers. We do weekly code updates on Thursdays at 2pm-ish so if you get an authentication token on Friday it will probably work for 6 days, but if you get one Thursday morning it might fail once in the middle of the day. This is all about cost reduction for Backblaze - we simply don't purchase ANY high end load balancers, it is all cheap computers that are expected to fail and break, but because there are hundreds (or thousands of them), even if half of them have lost power or died there are still plenty up and running (now we have some in a different datacenter!) to respond to your software's requests. We don't own a single load balancer (like F5), and we pass the savings on to you. Unfortunately we also pass along a small amount of additional complexity.
So it is really straight-forward code and you write it once and it just executes every time it needs to and never executes when it doesn't need to. Your code should not have ANY assumptions of time in it. Unfortunately it isn't something where you c...
* a bucket has 1M files in it
* uploads are generally of 200 files at a time
* each file name is guaranteed to map to a unique content (content-addressable)
* some of the files to upload might already be in the bucket
It's actually quite expensive to list all of the bucket files to find out if they haven't been uploaded. Especially when the file names are hashes it's hard to filter by prefix as well. (same issue on S3)
It would be great if there was a way to just upload the files and get an early 303 response if the entity already exists and has the same SHA1. That way the client can just push all the files without having to build the "new files" list.
I understand download pricing can get complicated. But again I'm not asking about that. I realize download pricing also is important, but right now I'm just asking about storage pricing. Not sure if you are answering questions here but if yes, please share what you know.
Upload: Free === Free + $0.05/1000 transactions
Download: $0.02/GB === $0.05-$0.09/GB
Retrieval Fee: Free === $0.003-$0.03/GB + $0.003-$0.03/1000 requests
Retrieval Time: Milliseconds === 1 min - 12 hours
In short, unless you only upload very large individual files, do nothing with them, and don't access any of them for at least 10 years...B2 is lower cost. (And, of course, doesn't required you to wait to get them out.)
Edit: BTW it's really a breath of fresh air to see someone whose product is covered hanging around for a while answering questions. So many people in your position either don't show up on their HN stories at all, or do so only for an hour or so and then leave a ton of questions unanswered.
Probably my number one wish list feature for hacker news would be if they could somehow coax people at companies that get linked to return more, including beyond the first hour or first day.
I hope you check back on this story later and answer more questions that come in from different time zones.
S3 can deliver 500 MB/s to a single EC2 node pretty consistently (same region, multiple streams) which keeps me there.
> what colo facilities have max bandwidth to you
Our primary colo is SunGard in Rancho Cordova, California. I believe we are currently provisioned for about 220 Gbits/sec (in this one facility alone). I'm not completely sure what this would top out at if we asked for more, but it is a colocation facility with a bunch of network providers such as Cogent, GTT, Century Link, etc. I kind of expect they can provide anything we ask for?
> what is my likely MB/s at the top-performing location
I have three answers, you can choose one. :-)
1) If you want to download a file once after a very long time of not accessing it (meaning it is not in the "B2 cache"), and with one thread, it must be fetched from the vaults to the "caching layer". Spooling a single file off of the vaults with one thread is relatively slow at about 15 Mbits/sec.
2) If the file is already in the caching layer, then it is fetched from local SSDs already inside the caching servers. Each individual caching server has a 10 Gbit/sec network card so if you only have one thread then you are absolutely limited to 10 Gbit/sec (and realistically the caching servers are not completely idle so a realistic number might be only 3 or 4 Gbit/sec).
3) If you want to fetch many files with many threads, fetch one file at a time with each thread. I'm not sure how many HTTPS threads an EC2 node can handle, maybe 200 safely? If it isn't in the cache then you get 200 * 15 Mbits/sec = 3,000 Mbits/sec = 375 MBytes/sec. If it is coming out the cache even faster?
Hope that helps!
This is unlike S3, which simply needs a STATIC API key, and boom, you can access the API. I don't know why you did this, but it is bad developer UX. Is it because tokens are "cool". I see no need to have this concept of tokens for your API. Why I am annoyed:
1) An API operation now requires one ADDITIONAL call. B2 already requires 2 calls to upload a file, and this now means 3 calls!
2) Related to above, this means bad latency. The roundtrip can be significant, especially since you only have 1 data center in California. My users will NOT be happy.
3) My code is much more complex now.
4) It is more work FOR YOU. This means more processing. More resources wasted FOR YOU. More cost. Why would you do this? You are supposed to be lean. Imagine millions of extra calls EVERY day. Adds up.
Is there a solution to this? Can the token be cached? If so, for how long.
Also, would love to see a reasonable rationale for this? If there is no rationale, then please provide a STATIC approach to access your API.
I REALLY like B2, and I wish they could think of developer UX a little more.
-I have to store the cached token somewhere:
-I have to do date arithmetic, what if my server clock is incorrect,
-if an API call failed, I have to check for an additional error: a invalid token
I don't mind complexity when it is needed, but I do not prefer complexity for no good reason.
That being said, can it really be cached up to a week? I see "at most 24 hours", which means it could be even less than 24 hours, so there's no guarantee. Please see: https://www.backblaze.com/b2/docs/b2_authorize_account.html
In general it's considered good practice to always check return values and error codes. Not for all code, but just for the code you would like to work.
How is your clock inaccuracy in any way the responsibility of a service provider (Backblaze or other)? From certificate expiry to inaccurate logging, and so on, complaining about "what if my clock is wrong, I have to date math in my app!" is precisely "solving the wrong problem".
Not shifting blame on Backblaze for my server clock being potentially inaccurate.
Yes, the first API call requires authenticating first but every additional call reuses cached credentials, so no change to latency.
I've been using Backblaze to write a for-fun app, using c++ and libcurl, for the last few weeks and I've found it to be wonderful from a developer perspective.
Their documentation is short and sweet. They have capabilities to help you stress test your implementation to make sure you handle edge cases! (you can pass an extra flag and it will randomly send errors) That is so amazingly helpful! And to top it off, their support staff is quick and responsive even with technical details. _And I don't even have a paid account_
If you find dealing with this to make your code "much more complex", then as they say "my consulting rates are very reasonable, my email is in my profile". :P
(Disclaimer: I do not work for Backblaze, have never given them a dime of my money. So far.)
I like there docs, short and sweet. It fits in their lean approach. But, using tokens is much heavier than the easier static key approach.
https://help.backblaze.com/hc/en-us/articles/218513487-Is-th...
> Backblaze requires an authorize() call to obtain a token before calling any API function.... this now means 3 calls to upload a file
That is correct if you only plan to upload one file. The "authorize" can be reused for weeks, and so can the upload URL. So if you want to upload 100,000 files, it is 100,002 calls to upload the 100,000 files (give or take).
It is true this is slightly more complex than it could be, but in turn this means we don't need to have any load balancers in our datacenter (which in turn makes the cost of the service cheaper and at the same time enormously more easily scalable). The idea here is that your application authenticates, then you ask for a URL to a vault that has spare space on it. Then your application "hangs up" (stops talking with that server) and contacts the vault directly. This means there is zero "chatter" between our vaults and that you are talking directly (no hops) to the vault you are storing your data on.
If you use b2_list_file_versions (or any of the b2_list functions) there are checksums included in that list (SHA1).
When you ask for that file list, the results are actually coming out of a large Cassandra cluster (database with many computers and fast SSDs).
Ok, so even if you do not request a file list of any kind, Backblaze itself automatically is always walking over the entire data farm checking every single last file's SHA1, plus checking if the files match the database's SHA1. A full sweep completes in less than a month then begins again.
If any bit error is detected (which happens semi-regularly since we have over 72,000 hard drives and bits get flipped) then we rebuild the files from the other copies (Reed-Solomon encoded across many separate computers in many separate locations in our datacenter).
We love and take care of your data so you don't have to.
The service I provide has, currently, an average of 60k http requests daily, approximately 10Gb. Those requests are images stored on Google Cloud Storage. I'm going to offer a new service for photographer to create online galleries, by some calculations I had made the average daily requests will increase around 10x, growing up to 600K daily.
My main questions about B2 are:
1) Is there any use case you know of someone uploading files to B2 directly from the browser? I know this is possible, but as per I know I would have to open tokens and url uploads on JS code, which, obviously, is not cool (and I won't do that). I can do this on S3 and GCS, their API offers the possibility to create a key on the server and use it on the webpage, but I didn't see anything like that on B2.
2) About http requests, is it ok to have this amount of requests (600k) in a daily basis? I mean I know B2 is excellent for backups but I'm not sure if it will be good serving websites that have a big number of images.
Thanks!
https://www.ovh.com/us/public-cloud/storage/object-storage/
It seems to be more expensive for storage but nearly half the cost for transfer.
EDIT: And Backblaze B2 invented their own API, so no tooling at all.
At first that was true, but now we have a list of 3rd party tools you can use. Check the list out here: https://www.backblaze.com/b2/integrations.html
Here is a sample from that web page: CloudBerry, Rclone (mount B2 as a hard drive on your desktop), Synology (this is a NAS drive), Retrospect, Cantemo, Cubix, Cyberduck, Duplicacy, Duplicity Linux, HashBackup, NyNAS, NeoFinder, Odrive, Qbackup, SmartFTP
OVH is also entirely located in Canada and France so performance between much of the US and their services isn't fantastic.
Also notable is that unlike many of the other cloud providers, OVH doesn't seem to offer free bandwidth between their services. You can't spin up an OVH server to thumbnail a bunch of images for example as far as I can tell.
Or full support for hosting a static website: https://help.backblaze.com/hc/en-us/articles/230383208-Hosti...
Great prices though!
https://github.com/wal-e/wal-e
At some point Arq could even support B2, though the most recent record I could find of this suggests that it's not planned right noW: https://twitter.com/arqbackup/status/662037695879880704?lang...
I don't know if any of those banks also offer a similar deal for business accounts.
I think in the not too distant future you'll be able to get a regular (non-prepaid) debit card from them now that they have a unrestricted UK banking license [2].
[1] https://monzo.com/
[2] https://monzo.com/blog/2017/04/05/banking-licence/
I previously tried Revolut (https://revolut.com) to avoid fees, and from having to apply for another credit card. Whilst great in theory, and it works fine for in-person purchases abroad, too many online services decline it.
https://aws.amazon.com/glacier/pricing/
GCE Coldline is .007c / GB
https://cloud.google.com/storage/archival/
$0.004/GByte/month - Amazon Glacier
$0.005/GByte/month - Backblaze B2
Furthermore, I think if you price it out, Backblaze will be cheaper than Glacier in most examples. For example, Glacier charges for uploads (5 cents per 1,000 uploads). If you ever download the data, it costs 9 cents/GByte to retrieve it from Glacier(!!!)
Use anything you like, but just make sure you consider all the charges and give Backblaze a fair consideration.
https://aws.amazon.com/glacier/pricing/
At Backblaze, we cannot figure out the business case of why Amazon prices things so that the small customers get really expensive prices (9 cents/GByte for less than 10 TBytes), and as the volume increases the price drops to 5 cents/GByte (for 250 TBytes). It makes no business sense to drive away small companies. A byte is a byte, the WHOLE POINT of a large cloud storage farm like Backblaze B2 or Amazon S3 is that we bundle together 200,000 small customers and together they get the economy of scale of a large customer! That's the whole point, the whole strategy! And what Backblaze has seen over and over again is that over time, some of our small customers became really gigantic customers. So driving away the small customers by gouging them on price means the small customers go to some other provider, then they get large with the other provider. But hey, I'm just thankful Amazon thinks this is good for them because it leaves the market wide open for Backblaze to gobble up all the small customers (which we are really super happy to have as customers).
The amazon philosophy is quite a bit like the underpants gnomes: 1) grow, 2) ..., 3) profit. Where there are natural monopolies with obvious economies of scale, this is an incredible strategy. But outside of that (retail, for example), it just doesn't work. Economies of scale come coupled with diseconomies of scale, and #2 needs to be an intentional and deliberate approach to managing those diseconomies of scale. If you look at their financials, it is obvious that Amazon retail has had a miserable time managing this...over the last 15 years they have steadily lower inventory turns, steadily increasing logistics costs, availability inconsistency, etc. Anybody who works there now or worked there in the past can see how inefficiently they run things and how efficiency improvements one day are ameliorated by political and speculative decisions the next day.
Combine this with their typical acquisition strategy, as exemplified by the Quidsi approach. Lowball, and when they don't accept, spend money subsidizing your own product to undercut their market and make them more desperate to sell.
The Backblaze approach is genius for taking on Amazon because it inverts the priorities to 1) meticulously manage costs, and then 2) directly engage amazon by competing on price. Backblaze can compete on price because their cost structure is lower than Amazon. Amazon can always subsidize their product, but by doing so Amazon is in the weaker position in the endurance race. Backblaze can ride out the attempts at undercutting them because they have a lower cost structure and prices need to drop a lot before they affect their cash flow safety. It's also a resilient strategy; Amazon can know that's exactly what they're doing and they can't do anything to stop it apart from beating them at their own game of cost management. It's a slow and steady road, but it can work.
Now, if only they would start running more than an S3 competitor :)
But then Backblaze don't have the cost advantage! Amazon uses their profits from S3/EC2 to subsidise all the new AWS services (and feature development for the existing services).
I really wonder if you'll get the same level of reliability as the leading providers.
> They only have 50 employees
We are larger than that now, but yes, still in that ballpark.
> same level of reliability as the leading providers
How is reliability related to number of salespeople? (That was a joke.)
But seriously, we published the EXACT reliability and the design in our blog post here: https://www.backblaze.com/blog/vault-cloud-storage-architect...
In a nutshell, any one byte of data is spread across 20 different computers in 20 different physical locations in our datacenter through Reed-Solomon encoding. If we lose 3 full computers in 3 separate locations your data is still fine.
We have been doing this for 10 years now, and we take it very seriously, and we have 73,000 hard drives in our datacenter. We are the only company on earth to publish our drive failure rates: https://www.backblaze.com/blog/hard-drive-benchmark-stats-20...
I'm thinking about replacing my aging Core 2 Duo laptop running Plex with an attached USB hard drive and BackBlaze with a combination NAS with Gig E and an NVidia Shield. I would use B2 for backups.
I have gigabit internet up and down and a house wired for Gig e throughout.
I've been begging the pricing team at Backblaze to drop this price for about a year. I'm just glad it finally percolated through.
As far as we can tell, all the cloud storage providers are gouging their customers on outbound bandwidth prices. I'm sorry we couldn't get this pushed through sooner.