Nice, and it seems to be quite cheap as well.
It's unfortunate, tho, they don't talk about data residency: where are servers located? Where will my data be copied?
We plan to add support for data residency requirements on an object-by-object basis. In other words, if you need one object to stay in the EU and another to stay in the US and a third to stay in India you tag each object using the same API and we handle ensuring the residency requirements behind the scenes.
we just got this big grant that works with healthcare and genomic data for a new type of therapy in Australia, so we have jurisdiction requirements (data needs to stay within Australia, some data needs to stay within New South Wales). We're currently talking with your run of the mill providers, but I'd pretty excited to try this out with Cloudflare instead... esp. when previous similar projects have been hit with some nasty egress costs
Interesting pricing considering Backblaze is another Bandwidth Alliance member and they only charge $0.005/GBmonth (vs. $0.015/GBmonth). B2 + CloudFlare gives you a similar deal at a third the cost.
I'm excited because while B2 + Cloudflare is great, the speed+latency isn't the greatest for some applications. So there's definitely a place for R2 here to compete more with AWS S3 than B2.
I'm a fan of B2 as well, but for some use-cases they seriously need to up their game. They only have three datacenters (CA, AZ, and Amsterdam), they still don't have a public status page, their admin UI is lacking lots of features (like proper invoices and different 2FA options), their permission system is very basic and inflexible, and they are not integrating compute with storage like AWS already does and Cloudflare will eventually be able to. However they are impossible to beat on cost, and for me their latency has recently improved significantly and has become much more stable, so I'm not going to move anytime soon.
Latency (time to first byte) in serving infrequently accessed images was a big problem with me and B2. The cost was low enough that I've stuck with it though and coded on the front end of the site to use placeholder images until the real media can be retrieved.
I recently started keeping about 30TB of ElasticSearch and Postgres backups in Backblaze B2. The price is great, but getting data in is not particularly easy as the Backblaze S3 API seems to fail a high proportion of requests when under load.
If R2 can be approximately as reliable on ingest as AWS/GCS/Azure is, but without the egress fees of the other major providers, then $0.015/GB-month seems like a pretty good deal.
B2 really is an exercise in making sure your code is robust with respect to external APIs but I'll be damned if it isn't cheap. We ended up building a whole queueing system for it because ad-hoc retry logic stopped being good enough.
Yes but you can only use B2 via CloudFlare for web pages. Using it as a data storage platform isn't allowed. Unless of course you're willing to pay handsomely via an enterprise contract, but then the pricing changes.
Use of the Services for serving video or a disproportionate percentage of pictures, audio files, or other non-HTML content is prohibited, unless purchased separately as part of a Paid Service or expressly allowed under our Supplemental Terms for a specific Service. [1]
You may use Cloudflare Pages and Workers (whether in conjunction with a storage offering such as Cloudflare Workers KV and Durable Objects or not) to serve HTML content as well as non-HTML content (e.g., image files, audio files) other than video files.
The said limitation should apply however to their tranditional service with orange on whether it is B2 or not. I am not sure if being a Bandwidth Alliance partner makes a difference.
So the gray area comes from an exception being granted from R2 not specified in that linked page. R2, like B2 is part of Cloudflare's bandwidth alliance, so is the unwritten exception for R2 or for the bandwidth alliance?
Very much depends on what you're utilizing Backblaze for. It's unusable for volume image hosting for example. It has severe performance problems with small files, as does much of the object storage field (including DigitalOcean, they directly warn customers not to bother using their object storage for small files if performance is a concern). The CDN doesn't help much, Backblaze chokes if you try to shovel a large number of small files at it (their infrastructure was not designed for that and they admit it in their docs), and or attempt to pull a lot of small files out of it. AWS is pretty great when it comes to that by contrast, and with the price to go with it. I'm looking forward to finding out what R2 can do.
This is true - most of the object storage providers are really bad at serving data fast, they more focused on a cold storage. Tebi.io works really fast for small files, plus it is a geo-distributed object storage meaning that data is physically stored in different locations around the world, greatly reducing latency.
Backblaze, DO Spaces simply were not designed for this in the first place.
Really interesting, will R2 support lifecycle rules like S3 does? We write around 90 million files per month to S3, if we could replace that with R2 and have the files automatically expire after 30 days that'd be a pretty amazing price reduction for us.
They appear to be focused on an automatic version:
> Behind the scenes, R2 automatically and intelligently manages the tiering of data to drive both performance at peak load and low-cost for infrequently requested objects. We’ve gotten rid of complex, manual tiering policies in favor of what developers have always wanted out of object storage: limitless scale at the lowest possible cost.
The full lifecycle support s3 has is really powerful. One use case we have for ugc content is disallowing online serving systems the ability to permanently delete an object. Instead the object is marked as deleted and then automatically cleaned up after 30 days.
Very exciting. Object storage is getting really competitive and i love the naming scheme alliance - S3, B2 (Backblaze) and now R2, who will do one with a “1”?
On a serious note i’m wondering about the signed urls and ACL capabilities of the cloudflare offering cause this is something we use.
I’m also interested does R2 replace S3 and CloudFront at the same time? That’d be nice and one headache less.
The Cloudflare offering supports their serverless Workers compute which would make signed URLs and ACLs trivial. Cloudflare certainly would also be replacing CloudFront.
> Our vision for R2 includes multi-region storage that automatically replicates objects to the locations they’re frequently requested from.
Seems like they are going to automatically replicate data to other regions. Something like tebi.io is doing for a long time already, it is a geo-distributed S3-compatible storage that is replicating data across the world to reduce read/write latency and increase speed.
If it is done right, this might increase download speeds by a lot especially for big infrequently accessed files.
Really curious to see how this goes. If they live up to the following paragraph, that's pretty game-changing:
>> This cheaper price doesn’t come with reduced scalability. Behind the scenes, R2 automatically and intelligently manages the tiering of data to drive both performance at peak load and low-cost for infrequently requested objects. We’ve gotten rid of complex, manual tiering policies in favor of what developers have always wanted out of object storage: limitless scale at the lowest possible cost.
The amount of effort it takes to understand and account for S3 Intelligent Tiering is somewhat mind-blowing so to get rid of all of that (and the corresponding fees) would be really nice and TheWayThingsShouldBe™ for the customer -- on top of that most users just don't even know S3 Intelligent Tiering exists so it'll be great if Cloudflare just handles that automatically.
We at https://vantage.sh/ (disclosure, I'm the Co-Founder and CEO) recently launched a cross-provider cost recommendation for CloudFront Egress to Cloudflare which was really popular and I can imagine doing something similar for S3 -> R2 once it is live and we are able to vet it.
Does Vantage offer comparisons for Backblaze B2, OVH etc?
When looking at object storage, tail latency is probably the single most overlooked metric, and the most material differentiator between providers after bandwidth costs. Don't sweat the cent spent on storing an object, worry about the cost of the 6,000,0000 copies of it you'll ship after it's stored.
As for bandwidth, CloudFlare becomes uninteresting the moment your account starts to see any real consumption, even AWS are easier to negotiate with.
We are working a more holistic approach to ingesting and providing recommendations from companies like Backblaze, OVH, etc. in addition to many, many more providers for other AWS services. The goal being that we can give customers the tools they need to get visibility on what options exist and take action themselves from there.
Your average cable modem can't do a good job of hosting much more than 50GB/day. Would you say that 1TB/day is well into 'real' then? You seem dismissive of that much.
As someone who took up making travel videos as a hobby, this is definitely on my radar.
Video files are large, although ~20 cents per video streamed for a small website is manageable (S3, Cloud Storage, Azure...), it's the potential for abuse that could drive my bill up that terrifies me, which is why I decided to stick to Hetzner VMs with their 20TB of free egress.
I have, I've also taken a look at Mux. Both would be fantastic options, and I'm still considering them, but I don't have many films and I'm biased towards distributing the original high-quality encodes.
Both of these services significantly reduce the file-size with a re-encode, even if they promote an "impercievable quality loss". They seem to be more suited to high traffic on-demand streaming for websites, promotional material, etc.
What format, bitrate, and resolution are your video outputs? Or just a length and file size of one that’s handy. (I’m a streaming video engineer and curious.) Reduction in file size from re-encoding doesn’t mean there will be perceivable loss in quality. Your videos should almost certainly not require the kind of bandwidth that you mention unless your deliverables are higher-quality than those from, say, Netflix :)
FYI Mux isn’t an okayish platform for mid-level use-cases, it’s the gold standard for high-performance live streaming and VOD, created by the guys behind a bunch of other stuff pretty central to the streaming industry overall, and is used at the highest level by folks like CBS, Mandolin, etc. Of course you don’t have to use it, but it’s certainly no toy.
Just curious, does your audience expect to download it in full to watch later or on a USB stick as you describe, or is that a side effect due to the large file sizes?
EDIT: I am not a Mux employee, nor is any of this very important, I've just been working in this space for 10 years and very seldom run across something that needs these requirements and I'm curious :)
No worries, I've always been more interested by the technical aspect more than the creative aspect of filmmaking, hence my interest in programming. It's a personal project, I've been making films since the age of 11, although I haven't been able to do much since I started my Bachelor degree...
I encode H.264/AAC for compatibility, usually 2-pass at 8Mbit/s for FHD content or around 30Mbit/s for UHD, 256-320 kbps for AAC. This gives a size of 1-4 GB per video. My Dad worked in broadcast, gave me those figures 10 years ago, I generally stick by them for any work I'm proud of!
You are right, that bitrate absolutely isn't necessary :D , and there are better codecs too. I don't have more than 17 films at the moment, the whole backend costs me about 4 euros a month on Hetzner with an attached 32 GB of block storage, no bandwidth costs (300Mbit/s max), single-core running Node.js and Nginx.
I make films during travel and events, and share them with friends and family who were there and for who it has some intrinsic value. They're mostly personal, not just for consumption. Short compilations that I would like to keep for the future, like old photos. Hence why people watch (and hopefully rewatch!) them on a TV and don't mind the wait.
Buffer-less streaming is absolutely not a priority (although nowadays I think it's more expected, people have shorter attention spans, despite the hours of work that goes into a short travel film). It's a very niche case, but would have cost me at least $50 in bandwidth alone with the big three. It's not going to break the bank, but it's also unnecessary.
You don't usually notice the drop in quality on YouTube or Netflix, until you actually try working with high quality source footage (high-bitrate from a dedicated camera, not phone footage). Then it's very noticeable (purple/green flickering on even surfaces from chroma subsampling, I'm guessing), and makes you sad when you realise what everyone is missing!
If you're still curious, my website is still live. I beg you not to judge, I started making short films very young and never got very good at them either (I'm studying nanotech & engineering, no film career for me)!
I suggest to use constant quality encoding instead of a constant bitrate, this way encoder will automatically adapt bitrate for a particular scene. This is a much better approach, it will give you better quality and a smaller file at the same time.
For example, encoder might choose 1Mbps bitrate for a scene which is mostly static and 20Mbps for a part of the video with a lot of movement. Your 8Mbps constant bitrate will be an overkill for the first scene and too low for the second. Let encoder decide the optimal bitrate.
Yes, we have read after write consistency across regions today. We're considering relaxing that if we need to reduce latency (one of the things to be worked out in beta!).
That's the reason why most enterprises have 4-7 copies of their data..... No inherent geo-replication by default (and as CF shares, it's a "hotel California" problem.... Too expensive to egress completely from AWS.
The claim is that the service handles distribution across reliability, so I think the more interesting question is the odds of that _mechanism_ failing when your site would otherwise be up[1].
Similarly, 3-2-1 is a backup strategy and the pricing appears to already include multiple copies using the same mechanism so the correct calculation would be the cost of R2 plus whatever _different_ mechanism you choose for disaster recovery purposes such as on-premise storage or a completely different provider.
1. For example, if you use Cloudflare as your CDN / DNS provider and they have a catastrophic failure, the fact that your storage is inaccessible is just another facet of the same issue.
> Our object storage will be extremely inexpensive for infrequent access and yet capable of and cheaper than major incumbent providers at scale.
How frequent is infrequent? In our case it's "never unless other backups fail" and for that S3 Glacier Deep Archive is still cheaper ($0.00099 per GB).
"At launch, R2 will cost $0.015 per gigabyte of data stored on the service, about half as much as AWS charges for customers on its frequent access tier. Infrequent access to R2 will be free, the company said.
"
So.... I still need to test, but if that is true can be a game changer. Of course... $0.00099 is almost free, but if everything is done automagically will be awesome.
Cloudflare Pages locks you into git-based deployment, which isn't always practical, especially for sites that are heavy with images and other non-text static assets. I don't want to use git to manage the images for my website and I don't want to have to pay Github every month for Git LFS storage and bandwidth costs.
I know the Cloudflare team hangs out here, so thanks, and great job! This was absolutely necessary for my line of work. Couple of quick questions/confirmations:
* R2 will support the same object sizes as S3? We have 500GB+ objects and could go to a 1TB per object.
* R2 will support HTTP Range GETs, right?
Egress bandwidth for objects on S3 is the biggest line item on the AWS bill for a company I work for, by an order of magnitude, and this will just wipe it off for the most part.
Yes to range requests. Current object limit is smaller than that, but we don't have a fundamental restriction there. Shoot me an email to gmckeon [at] cloudflare.
I personally think uploading massive files is not usually desirable. Better would be an easy way to chunk it and upload and have the server put the file back together, which would increase reliability.
Tus is over-complicated IMO. Why require the server to store state that's already available on the file system? Just support PATCH requests with an offset parameter. If an upload fails partway through, HEAD the file to see how much got copied to disk and resume from there.
Really excited for R2 Storage. I am wondering if R2 can solve a limitation I was running into with BackBlaze.
I had tried using BackBlaze 8 months ago as a much cheaper (especially with B2 and CF's Free Data Transfer partnership) replacement for Amazon S3 and was running into a limitation on B2.
I had a scenario where my users can upload images from the browser to BackBlaze. I wanted the ability to control the file name of the uploaded file. I don't want the user to be able to modify the network request to upload the file with a different file name. Nor do I want the users to be able to upload files with names which would overwrite existing files.
B2 didn't let me upload files with a specific filename in the pre-signed URL.
But this allowed my users to upload file with any name they want. It would also allow them to overwrite existing files (from other users).
My question is more from a security point of view so preventing one user from overwriting another user's content is crucial. For example, lets say you right click on an image from someone else on facebook and get the actual image's file name. Now you try to upload an image on facebook and you edit the network request in the browser's inspector tool to the image file name which you got for another user. Facebook obviously prevents this in their own way using pre-signed urls which include the filename in the signature. However on BackBlaze if I try this, the "pod" url which is received doesn't include any file name signature. The pod URL is just where the image gets stored on your end. A user can easily edit the network request and modify the "X-Bz-File-Name" Header to another user's filename. This would be a major security vulnerability if I went with BackBlaze. As a workaround, right now it seems like users would first have to upload files to my own server, then my server would have to upload them to BackBlaze to avoid this issue. This sounded like hassle.
Amazon S3 solves this problem using createPresignedPost which includes a signature of the filename in the URL. I contacted BackBlaze's support and got a response their S3 api doesn't support createPresignedPost:
Is there a way to prevent this on R2? Something where the link provided by b2_get_upload_url (whatever R2's equivalent will be) only works for specific a file name?
I have this exact same use case for an app I’m building and would love an answer as well. I built on S3 as a result. When a product says “full S3 api compatibility” this becomes my question.
Going further I don’t want to become someone else’s back door file repo for illegal shit. So presigned upload urls with an enforced file name and configurability over the size limit and expiration of the presignedpost (both in terms of time and number of files) is pretty important to me. S3 does a good job here.
Nilay from Backblaze here. Amongst other things, the solution engineering team reports up to me... and when I saw this comment yesterday, I had them dig into why the B2 S3 API doesn't did not work for you.
It turns out that Backblaze B2 absolutely does support uploading objects via S3 style presigned URLs. However, there are a couple of caveats:
1. B2 doesn't currently support PostObject operations. Therefore, you must use PutObject to upload to a pre-signed URL. Many of the AWS SDKs default to PostObject.
2. B2 only supports AWS v4 authentication and not v2, which AWS has deprecated. However, some AWS SDKs default to v2.
I am currently away for this week, so I can't try it this week. Can I reach out to you next week with my findings? Is there an email address I can reach you at?
Just wanted to mention that I had raised my issues on the backblaze subreddit back then:
I've always found pre-signed URLs fragile and a pain to work with, when we solved this problem we just put cloudflare workers in front of b2, so the worker can sign requests to b2 itself, so the interface to the user can be simpler. Could just be a straight POST with bytes, then the worker will turn that into a PutObject call to the s3 api, it works pretty damn well.
If you are reading files back out again too you can use cloudflare caching to speed things up, its a good combo.
Sounds great (nearly too-good-to-be-true great). Wonder how the SLA will look like. I have been using gcs, s3 and firestore - and their actual reliability varies significantly, while advertised slas are similar. For instance, with firestore one has to implement a pretty lenient expotential backoff in case of a timeout, and if the backoff results in the object being retrieved in, say, 2 minutes -- thats still ok as per gcs sla. It obviously makes it hard to use it for user-facing stuff, such as chatbots, where you can't afford to wait that long. In my anecdotal experience of using firestore for about 10 million operations per day, we will usually have a problem like that every few days, and that means user-noticeable failure. It would be great to read more on cloudflare's approach to reliability defined as "99%" percentile max latencuy. Can't wait to give it a try with our workloads.
> R2 is designed with redundancy across a large number of regions for reliability. We plan on starting from automatic global distribution and adding back region-specific controls for when data has to be stored locally, as described above.
Does that mean automatic caching across regions? Low-latency read access everywhere without an extra CDN in front of it?
We're still deciding on how we want to handle caching. We integrate with Workers, so manually caching is always possible. The catch is we're currently building for strong consistency - if we added a cache in front, we'd weaken that - so it will likely be a configuration option we add later.
Check tebi.io - it is a geo-distributed S3-compatible storage that does exactly that. You can configure global consistency level for each bucket individually.
Scaleway also provides a similar service. Had an issue recently (deleted a bucket, and couldn't create another bucket with the same name) and their support replied back in minutes.
Love the team at DigitalOcean but Spaces was NOT reliable the last time I played with it. It’s also lacking key features in lifecycle and notification areas. Maybe they’ve gotten things together in recent months/years and I haven’t gotten the update, but it was very “you get what you pay for” when I tried to stage a project using it in 2018/19.
Spaces is not good, in my experience. Their Frankfurt storage just stopped working for like a week. And last time I checked they still didn't support critical features like per-bucket access keys or index.html support.
There is one major reason S3 remains the king of storage for mobile media uploads: bucket notifications. Does R2 implement this feature? If so, I’m going to have to run some experiments with this...
To be honest, I think the biggest draw will be for companies (like where I work) that put large objects on S3 and distribute it to hundreds / thousands / millions of customers. The egress direct from S3 is on the order of 8 cents a GB, and with Cloudfront in front of it it’s a few cents lower, and you can negotiate pricing a little lower if you’re big enough. But not an order of magnitude.
We’d stick R2 in front of an S3 bucket and wipe off a the biggest portion of our bill.
S3 does absolutely have a ton of other stuff like events, lifecycle, Glacier, Lambda etc and is plugged into the AWS ecosystem, so I doubt we’re exiting it completely. But this is a solid option as a front for egress from S3.
We're fully integrated with Workers, so you can write a Worker that calls additional logic when a request is made to the bucket's url.
We have notifications, where a change to the bucket invokes a worker with a specific event, on our roadmap, but have a bunch of other work prioritized ahead of them.
Hi Greg, thanks for the feedback. It’d be great if you could get around to putting up an example of how to do bucket notifications + lifecycle using Workers as a temporary workaround until it’s part of the “core.” I don’t think I’m the only person with this use case, but maybe I’m more of an edge-case minority than I imagine... In any case, a code library / “recipe collection” (do we still call them cookbooks?) would be great when this launches.
I know this is irrational and not very helpful, but it actually makes me angry when I see a response like "but you can cobble together some janky custom code with Workers to do what you want".
It's like going to a restaurant, asking if they have shrimp scampi, and getting a reply that you can go to a supermarket and buy the ingredients and make the dish and bring it back to the restaurant to have with your meal.
Is this going to be content-neutral, like Cloudflare was when fronting ISIS websites?
Or is this going to be fine-until-bad-PR, like when Cloudflare decided to stop hosting The Daily Stormer?
There is a special kind of lock-in when it comes to object storage, as generally you use something like this when the data is too big to store another copy of locally
or at another provider. It's not like you can easily maintain provider independence, and if Cloudflare decides one day that some of your UGC in a bucket isn't something they want to host, what happens then?
Is the data lost forever because your account is nuked? Is there a warning or grace period?
I am hesitant to put any large amount of data into a service without a crystal clear statement on this, so that I can know up front whether or not a business needs to maintain a second, duplicate object store somewhere else for business continuity.
If Cloudflare in practice is going to nuke the account the moment your site ends up hosting something objectionable, this DR requirement (a second provider that also stores all objects) needs to be factored into a customer's costs. (It may be that the bandwidth savings still make it worth it to use Cloudflare even with double storage.)
> I am hesitant to put any large amount of data into a service without a crystal clear statement on this, so that I can know up front whether or not a business needs to maintain a second, duplicate object store somewhere else for business continuity.
It's a mistake to rely on a clear statement when you can't afford to lose your data. Stuff happens all the time... mistakes, malware, an expired credit card, etc. Independently of the provider you decide to use, I'm not sure if a backup is optional in your case.
219 comments
[ 0.22 ms ] story [ 306 ms ] thread"Our vision for R2 includes multi-region storage that automatically replicates objects to the locations they’re frequently requested from."
If R2 can be approximately as reliable on ingest as AWS/GCS/Azure is, but without the egress fees of the other major providers, then $0.015/GB-month seems like a pretty good deal.
Use of the Services for serving video or a disproportionate percentage of pictures, audio files, or other non-HTML content is prohibited, unless purchased separately as part of a Paid Service or expressly allowed under our Supplemental Terms for a specific Service. [1]
--
[1] 2.8 Limitation on Serving Non-HTML Content https://www.cloudflare.com/terms/
See: https://www.cloudflare.com/supplemental-terms/ § Cloudflare Pages and Cloudflare Workers®
The said limitation should apply however to their tranditional service with orange on whether it is B2 or not. I am not sure if being a Bandwidth Alliance partner makes a difference.
Backblaze, DO Spaces simply were not designed for this in the first place.
> Behind the scenes, R2 automatically and intelligently manages the tiering of data to drive both performance at peak load and low-cost for infrequently requested objects. We’ve gotten rid of complex, manual tiering policies in favor of what developers have always wanted out of object storage: limitless scale at the lowest possible cost.
We support per-object TTLs, so this should work!
Is it built on Ceph's S3 compatibility?
Your durability numbers imply erasure coding. Is that the case?
Well, you can't really build inexpensive, large scale, reliable storage without erasure coding. So that's probably a given.
On a serious note i’m wondering about the signed urls and ACL capabilities of the cloudflare offering cause this is something we use.
I’m also interested does R2 replace S3 and CloudFront at the same time? That’d be nice and one headache less.
Seems like they are going to automatically replicate data to other regions. Something like tebi.io is doing for a long time already, it is a geo-distributed S3-compatible storage that is replicating data across the world to reduce read/write latency and increase speed.
If it is done right, this might increase download speeds by a lot especially for big infrequently accessed files.
>> This cheaper price doesn’t come with reduced scalability. Behind the scenes, R2 automatically and intelligently manages the tiering of data to drive both performance at peak load and low-cost for infrequently requested objects. We’ve gotten rid of complex, manual tiering policies in favor of what developers have always wanted out of object storage: limitless scale at the lowest possible cost.
The amount of effort it takes to understand and account for S3 Intelligent Tiering is somewhat mind-blowing so to get rid of all of that (and the corresponding fees) would be really nice and TheWayThingsShouldBe™ for the customer -- on top of that most users just don't even know S3 Intelligent Tiering exists so it'll be great if Cloudflare just handles that automatically.
We at https://vantage.sh/ (disclosure, I'm the Co-Founder and CEO) recently launched a cross-provider cost recommendation for CloudFront Egress to Cloudflare which was really popular and I can imagine doing something similar for S3 -> R2 once it is live and we are able to vet it.
When looking at object storage, tail latency is probably the single most overlooked metric, and the most material differentiator between providers after bandwidth costs. Don't sweat the cent spent on storing an object, worry about the cost of the 6,000,0000 copies of it you'll ship after it's stored.
As for bandwidth, CloudFlare becomes uninteresting the moment your account starts to see any real consumption, even AWS are easier to negotiate with.
Workloads you couldn't host on a cable modem
Video files are large, although ~20 cents per video streamed for a small website is manageable (S3, Cloud Storage, Azure...), it's the potential for abuse that could drive my bill up that terrifies me, which is why I decided to stick to Hetzner VMs with their 20TB of free egress.
https://www.cloudflare.com/en-gb/products/cloudflare-stream/
Both of these services significantly reduce the file-size with a re-encode, even if they promote an "impercievable quality loss". They seem to be more suited to high traffic on-demand streaming for websites, promotional material, etc.
That doesn't mean they can't stream, there is an option to watch it in the browser, they just need a reliable connection (~30 Mbit/s).
FYI Mux isn’t an okayish platform for mid-level use-cases, it’s the gold standard for high-performance live streaming and VOD, created by the guys behind a bunch of other stuff pretty central to the streaming industry overall, and is used at the highest level by folks like CBS, Mandolin, etc. Of course you don’t have to use it, but it’s certainly no toy.
Just curious, does your audience expect to download it in full to watch later or on a USB stick as you describe, or is that a side effect due to the large file sizes?
EDIT: I am not a Mux employee, nor is any of this very important, I've just been working in this space for 10 years and very seldom run across something that needs these requirements and I'm curious :)
I encode H.264/AAC for compatibility, usually 2-pass at 8Mbit/s for FHD content or around 30Mbit/s for UHD, 256-320 kbps for AAC. This gives a size of 1-4 GB per video. My Dad worked in broadcast, gave me those figures 10 years ago, I generally stick by them for any work I'm proud of!
You are right, that bitrate absolutely isn't necessary :D , and there are better codecs too. I don't have more than 17 films at the moment, the whole backend costs me about 4 euros a month on Hetzner with an attached 32 GB of block storage, no bandwidth costs (300Mbit/s max), single-core running Node.js and Nginx.
I make films during travel and events, and share them with friends and family who were there and for who it has some intrinsic value. They're mostly personal, not just for consumption. Short compilations that I would like to keep for the future, like old photos. Hence why people watch (and hopefully rewatch!) them on a TV and don't mind the wait.
Buffer-less streaming is absolutely not a priority (although nowadays I think it's more expected, people have shorter attention spans, despite the hours of work that goes into a short travel film). It's a very niche case, but would have cost me at least $50 in bandwidth alone with the big three. It's not going to break the bank, but it's also unnecessary.
You don't usually notice the drop in quality on YouTube or Netflix, until you actually try working with high quality source footage (high-bitrate from a dedicated camera, not phone footage). Then it's very noticeable (purple/green flickering on even surfaces from chroma subsampling, I'm guessing), and makes you sad when you realise what everyone is missing!
If you're still curious, my website is still live. I beg you not to judge, I started making short films very young and never got very good at them either (I'm studying nanotech & engineering, no film career for me)!
https://mastermovies.uk/glacier
For example, encoder might choose 1Mbps bitrate for a scene which is mostly static and 20Mbps for a part of the video with a lot of movement. Your 8Mbps constant bitrate will be an overkill for the first scene and too low for the second. Let encoder decide the optimal bitrate.
What happens to your Object Storage buckets when Cloudflare has an outage? - https://filebase.com/blog/what-happens-when-my-cloud-goes-do...
that for those in the back is sarcasm....
Similarly, 3-2-1 is a backup strategy and the pricing appears to already include multiple copies using the same mechanism so the correct calculation would be the cost of R2 plus whatever _different_ mechanism you choose for disaster recovery purposes such as on-premise storage or a completely different provider.
1. For example, if you use Cloudflare as your CDN / DNS provider and they have a catastrophic failure, the fact that your storage is inaccessible is just another facet of the same issue.
How frequent is infrequent? In our case it's "never unless other backups fail" and for that S3 Glacier Deep Archive is still cheaper ($0.00099 per GB).
This is a little bit above your quoted paragraph
https://www.protocol.com/enterprise/cloudflare-r2-storage-aw...
So.... I still need to test, but if that is true can be a game changer. Of course... $0.00099 is almost free, but if everything is done automagically will be awesome.
* R2 will support the same object sizes as S3? We have 500GB+ objects and could go to a 1TB per object. * R2 will support HTTP Range GETs, right?
Egress bandwidth for objects on S3 is the biggest line item on the AWS bill for a company I work for, by an order of magnitude, and this will just wipe it off for the most part.
Yes to range requests. Current object limit is smaller than that, but we don't have a fundamental restriction there. Shoot me an email to gmckeon [at] cloudflare.
What I'd like to see is PATCH with multipart/byteranges support. =D
I had tried using BackBlaze 8 months ago as a much cheaper (especially with B2 and CF's Free Data Transfer partnership) replacement for Amazon S3 and was running into a limitation on B2.
I had a scenario where my users can upload images from the browser to BackBlaze. I wanted the ability to control the file name of the uploaded file. I don't want the user to be able to modify the network request to upload the file with a different file name. Nor do I want the users to be able to upload files with names which would overwrite existing files.
B2 didn't let me upload files with a specific filename in the pre-signed URL.
For example there's this API:
curl https://apiXXX.backblazeb2.com/b2api/v2/b2_get_upload_url -H 'Authorization: XXXXXXXXXXXXXXXXXXXXXXXXX=' -d '{"bucketId": "XXXXXXXXXXXXXXX"}'
which gave me an upload URL where I can upload the file to.
And then there's this:
https://www.backblaze.com/b2/docs/b2_upload_file.html
which lets users upload files to that URL.
But this allowed my users to upload file with any name they want. It would also allow them to overwrite existing files (from other users).
My question is more from a security point of view so preventing one user from overwriting another user's content is crucial. For example, lets say you right click on an image from someone else on facebook and get the actual image's file name. Now you try to upload an image on facebook and you edit the network request in the browser's inspector tool to the image file name which you got for another user. Facebook obviously prevents this in their own way using pre-signed urls which include the filename in the signature. However on BackBlaze if I try this, the "pod" url which is received doesn't include any file name signature. The pod URL is just where the image gets stored on your end. A user can easily edit the network request and modify the "X-Bz-File-Name" Header to another user's filename. This would be a major security vulnerability if I went with BackBlaze. As a workaround, right now it seems like users would first have to upload files to my own server, then my server would have to upload them to BackBlaze to avoid this issue. This sounded like hassle.
Amazon S3 solves this problem using createPresignedPost which includes a signature of the filename in the URL. I contacted BackBlaze's support and got a response their S3 api doesn't support createPresignedPost:
https://www.reddit.com/r/backblaze/comments/kzszym/is_backbl...
You can read B2's staff's response to my question:
https://www.reddit.com/r/backblaze/comments/l0c9s7/is_there_...
Is there a way to prevent this on R2? Something where the link provided by b2_get_upload_url (whatever R2's equivalent will be) only works for specific a file name?
Going further I don’t want to become someone else’s back door file repo for illegal shit. So presigned upload urls with an enforced file name and configurability over the size limit and expiration of the presignedpost (both in terms of time and number of files) is pretty important to me. S3 does a good job here.
It turns out that Backblaze B2 absolutely does support uploading objects via S3 style presigned URLs. However, there are a couple of caveats:
1. B2 doesn't currently support PostObject operations. Therefore, you must use PutObject to upload to a pre-signed URL. Many of the AWS SDKs default to PostObject.
2. B2 only supports AWS v4 authentication and not v2, which AWS has deprecated. However, some AWS SDKs default to v2.
I've put together a very simple python code sample that successfully works for me. https://gist.github.com/nilayp/2c2a04f033d8992ce4b8f591ab449...
Would this solve your issue?
I am currently away for this week, so I can't try it this week. Can I reach out to you next week with my findings? Is there an email address I can reach you at?
Just wanted to mention that I had raised my issues on the backblaze subreddit back then:
https://www.reddit.com/r/backblaze/comments/m9tioi/backblaze...
https://www.reddit.com/r/backblaze/comments/kzszym/is_backbl...
https://www.reddit.com/r/backblaze/comments/l0c9s7/is_there_...
Drop me an email using my GitHub username at backblaze.com
We will correct the record on Reddit after hearing that this was successful for you.
If you are reading files back out again too you can use cloudflare caching to speed things up, its a good combo.
Does that mean automatic caching across regions? Low-latency read access everywhere without an extra CDN in front of it?
We're still deciding on how we want to handle caching. We integrate with Workers, so manually caching is always possible. The catch is we're currently building for strong consistency - if we added a cache in front, we'd weaken that - so it will likely be a configuration option we add later.
Disclaimer: I haven't used it, but planning to, since I already use their VPS.
This announcement is noteworthy because there’s no egress fees. Spaces still charges 1c per gigabyte for egress. Not comparable in that aspect.
How will other providers respond to this now?
AWS, GC, and others do not really pay for egress charges themselves. Those super high egress charges are pretty ridiculous.
we are still living in 2007?
We’d stick R2 in front of an S3 bucket and wipe off a the biggest portion of our bill.
S3 does absolutely have a ton of other stuff like events, lifecycle, Glacier, Lambda etc and is plugged into the AWS ecosystem, so I doubt we’re exiting it completely. But this is a solid option as a front for egress from S3.
We're fully integrated with Workers, so you can write a Worker that calls additional logic when a request is made to the bucket's url.
We have notifications, where a change to the bucket invokes a worker with a specific event, on our roadmap, but have a bunch of other work prioritized ahead of them.
It's like going to a restaurant, asking if they have shrimp scampi, and getting a reply that you can go to a supermarket and buy the ingredients and make the dish and bring it back to the restaurant to have with your meal.
Just tell me you don't have shrimp scampi.
Is this going to be content-neutral, like Cloudflare was when fronting ISIS websites?
Or is this going to be fine-until-bad-PR, like when Cloudflare decided to stop hosting The Daily Stormer?
There is a special kind of lock-in when it comes to object storage, as generally you use something like this when the data is too big to store another copy of locally or at another provider. It's not like you can easily maintain provider independence, and if Cloudflare decides one day that some of your UGC in a bucket isn't something they want to host, what happens then?
Is the data lost forever because your account is nuked? Is there a warning or grace period?
I am hesitant to put any large amount of data into a service without a crystal clear statement on this, so that I can know up front whether or not a business needs to maintain a second, duplicate object store somewhere else for business continuity.
If Cloudflare in practice is going to nuke the account the moment your site ends up hosting something objectionable, this DR requirement (a second provider that also stores all objects) needs to be factored into a customer's costs. (It may be that the bandwidth savings still make it worth it to use Cloudflare even with double storage.)
You raise an interesting point regarding object storage being a relatively unique type of lock-in.
It's a mistake to rely on a clear statement when you can't afford to lose your data. Stuff happens all the time... mistakes, malware, an expired credit card, etc. Independently of the provider you decide to use, I'm not sure if a backup is optional in your case.