It may depend on the makeup of data or something. They "requested" one of my prior projects go on the enterprise plan after about 50TB, granted the overwhelming majority of transfer was for distributing binary executables so I was in pretty blatant violation of their policy. This was 2015ish, so the limit could also have gone up over time as bandwidth gets cheaper too.
And they just added TCP client sockets in Workers. We are just one step step away from being able to serve literally anything on their amazing platform (listener sockets).
Only client sockets are available. So what you can do is build a worker that receives HTTP requests and then uses TCP sockets to fetch data from wherever, returning it over HTTP somehow.
> If your monthly egress data transfer is less than or equal to your active storage volume, then your storage use case is a good fit for Wasabi’s free egress policy
> If your monthly egress data transfer is greater than your active storage volume, then your storage use case is not a good fit for Wasabi’s free egress policy.
Unless its 100TB/mo of pure HTML/CSS/JS (lol) cloudflare will demand you be on enterprise plan long before 100TB/mo. The fine print makes it near useless for any significant volume.
At some point you still need a seed for that 10TB of data with some level of reliability. WebTorrent only solves the monthly bandwidth iff you've got some high capacity seeds (your servers or long-term peers).
Depends on what exactly you want to do with it. Hetzner has very cheap Storage boxes (10TB for $20/month with unlimited traffic) but those are closer to FTP boxes with a 10 connection limit. They are also down semi-regularly for maintenance.
For rock-solid public hosting Cloudflare is probably a much better bet, but you're also paying 7 times the price. More than a dedicated server to host the files, but you get more on other metrics.
I don't know about any special offers, but looking at standard pricing on rsync.net it would cost me $15/month for 1TB, while on Hetzner the same would cost me €3.94/month.
> Hetzner has very cheap Storage boxes (10TB for $20/month with unlimited traffic)
* based on fair use
at 250 TB/mo:
> In order to continue hosting your servers with us, the traffic use will
need to be drastically reduced. Please check your servers and confirm
what is using so much traffic, making sure it is nothing abusive, and
then find ways of reducing it.
Backblaze B2 is only free with Bandwidth Alliance partners, otherwise $0.01/GB, individual transactions will also cost you, depending on how much you use.
That's if you use their CDN. Cloudflare R2 doesn't charge for egress bandwidth. If you have 100TB/mo to serve, try it and see what happens. I haven't heard of anyone being kicked off of R2 for using too much egress bandwidth yet.
At scale, you'll pay a couple thousand dollars for Class B operations on R2, and another bunch for storing the 10 TB in the first place, but that's relatively cheap compared to other offerings where you'd pay for metered egress bandwidth.
CF is not particularly fond of non-Enterprise customers serving more than a few TB/mo. Source: $corp serves 150 TB/mo via CF and pays somewhere north of 50k+/yr for it
The whole point of R2 is to remove predatory practices of egregious charging for egress, and if somehow they went back on this promise that would be a very bad PR.
If we are talking about serving files publicly I'd go with the €40 server for flexibility (the storage boxes are kind of limited), but still get a €20 Storage Box to have a backup of the data. Then add more servers as bandwidth and redundancy requires.
But if splitting your traffic across multiple servers is possible you can also get the €20 storage box and put a couple Hetzner Cloud servers with a caching reverse proxy in front (that's like 10 lines of Nginx config). The cheapest Hetzner Cloud option is the CAX11 with 4GB RAM, 40GB SSD and 20TB traffic for €3,79. Six of those plus the Storage Box gives you the traffic you need, lots of bandwidth for usage peaks, SSD cache for frequently requested files, and easily upgradable storage in the Storage Box, all for €42. Also scales well at $3,79 for every additional 20TB traffic, or $1/TB if you forget and pay fees for the excess traffic instead.
You will be babysitting this more than the $150/month cloudflare solution, but even if you factor in the cost of your time you should come out ahead.
> even if you factor in the cost of your time you should come out ahead
There is always the hidden cost of not spending time on activities that are core to your business (if this is indeed for a business) that would make multiples of the money CF costs you.
That, and also NixOS - I’m discovering it for myself now, and it’s been a revelation! Configuring absolutely everything declaratively from scratch, even the disk partitions - a dream for reproducibility. It even has configurable “micro-VMs”, which would not be as easy to do via Proxmox (not counting LXC), since they would have to be built manually. Though Proxmox does have some nice benefits over it as well, especially considering their ecosystem with PBS, mail server etc
Thanks for the intro to NixOS. I was trying to remember one I had seen & forgotten and I think this may have been it.
I have been playing more and more with UTM in the Mac world and it's encouraging how mature it seems already and hopefully can be picked up into NixOS, Sandstorm, etc.
I like proxmox more personally, however have changed my stance recently where nix and sandstorm could just be run in a proxmox vm and then provide more of an IaaS role. The newer versions of ProxMox are even easier and they were pretty OK the past 5-7 years.
Exactly, and also you get to actually understand how it all works together, unlike a bunch of proprietary APIs that only tie you to their particular platform.
(for those not on the same page, I’m talking from a position of substantial experience with all 3 major clouds)
Plus, these days the maintenance burden of the OS layer is really heavily overstated. With certain self-updating open-source container OSes one doesn’t even really have to think about patches and all that ancient crap.
The real appeal of the big players in my mind is only in one use case - scale. If you need 10k servers for heavy “big” data processing like in genomics or ‘AI’ (whatever that means), only then they start to be indisposable. Otherwise, the considerable burden of training all personnel on proprietary APIs is just not worth it - it literally costs less to buy and configure your own system (or a traditional VPS or dedicated server). Cloud architects ain’t cheap!
I think it’s more about peace of mind, unlimited really means I won’t wake up tomorrow with a $10k bill, as it happened many times (not to me) on AWS and the like. That is the disgusting practice the big cloud providers like to impose, for no apparent reason but to keep you in their roach motel and pay up. Disgusting!
I would also like to ask everyone about suggestions for deep storage of personal data, media etc. 10TB with no need for access unless in case of emergency data loss. I'm currently using S3 intelligent tiering.
I think they'll charge me only when my current monthly statement is enough to charge. Pretty sure I've never been charged so far with my monthly statement being like 0.02€.
Some tens of gigabytes at this point? It's definitely not a lot. Mostly just some stuff that doesn't make sense to keep locally but I still want to have a copy in case a disaster strikes.
S3 is tough to beat on storage price. Another plus is that the business model is transparent, i.e., you don't need to worry about the pricing being a teaser rate or something.
Of course the downside is that, if you need to download that 10TB, you'll be out $900! If you're worried about recovering specific files only this isn't as big an issue.
Wasabi is the best option for you. 10TB would be around 60$/month and they offer free egress as much as your storage. So you can download upto 10TB per month.
I like to use rsync.net for backups. You can use something like borg, rsync, or just sftp/sshfs mount. Its not as cheap as something like S3 deep (in terms of storage) but it is pretty convient. The owner is a absolute machine and frequently visits HN too.
Glacier Deep Archive is exactly what you want for this, that would be something like $11/month ongoing, then about $90/TB in the event of retrieval download. Works well except for tiny (<150KB) files.
Note that there is Glacier and Glacier Deep Archive. The latter is cheaper but longer minimum storage periods. You can use it as a life cycle rule.
Surprised no one has said Cloudflare Pages. Might not work though depending on your requirements since there’s a max of 20,000 files of no more than 25 mb per project. But if you can fit under that, it’s basically free. If your requirements let you break it up by domain, you can split your data across multiple projects too. Latency is amazing too since all the data is on their CDN.
Spend some time on https://www.webhostingtalk.com/ and you will find a lot of info. For example https://www.fdcservers.net/ can give you 10TB storage and 100GB bw for around $300....but keep in mind the lower the price you pay, the lower the quality...just like any other products.
I once had a Hetzner dedicated server that held about 1 TB of content and did some terabytes of traffic per month (record being 1 TB/24 hours). Hetzner charged me 25€/month for that server and S3 would've been like $90/day at peak traffic.
The answer to this question depends entirely on the details of the use case. For example, if we're talking about an HTTP server where a small number of files are more popular and are accessed significantly more frequently than most others, you can get a bunch of cheap VPS with low storage/specs but a lot of cheap bandwidth to use as cache servers to significantly reduce the bandwidth usage on your backend.
The 100TB was just an example. They don't want you using more bandwidth than your storage. If you're storing 500GB, then your bandwidth usage should be less than 500GB.
Wasabi isn't meant for scenarios where you're going to be transferring more than you're storing.
That reminds me of the entertaining "I just want to serve 5 terabytes. Why is this so difficult?" video that someone made inside Google. It satirizes the difficulty of getting things done at production scale.
The interesting thing is there also paradoxes of large scale: things that get more difficult with increasing size.
Medium- and smaller-scale can often be more flexible because they don't have to incur the pain of nonuniformity as scale increases. While they may not be able to afford optimizations or discounts with larger, standardized purchases, they can provide more personalized services large scale cannot hope to provide.
On a related note, providers that have independent instances for each customer (so no multi-tenancy) typically get about 3 more nines than, say, AWS. On prem enterprise is a typical example of this, and it is still used in safety critical industries for this reason.
Eventually, all outages are black swan events. If you have 1000 independent instances (i.e., 1000 customers), when the unexpected thing hits, you’re still 99.9% available during the time when the impacted instance is down.
Also, you can probably permanently prevent the black swan from hitting again before it hits again.
Nothing in that video is about scale. Or the difficulty of serving 5TB. It's about the difficulty of implementing n+1 redundancy with graceful failover inside cloud providers.
User: "I want to serve 5TB."
Guru: "Throw it in a GKE PV and put nginx in front of it."
Congratulations, you are already serving 5TB at production scale.
It's impossible to answer this question without more information. What is the use profile of your system? How many clients, how often, what's the burst rate, what kind of reliability do you need? These all change the answer.
"Impossible", yet many others have succeeded commendably... explore what they can do but you cannot. Or else offer examples wherein your constraints exist and drive another solution. "No solution without more info" is a cop-out.
I'm sorry, let me clarify since you seem to be very pedantic. It's impossible to answer well without a bunch more information. Yes, there are other answers in this thread, but I would argue they aren't particularly helpful to either OP or any other reader.
It's kind of like someone going to a group of doctors and saying "I'm in pain", and then the doctors start throwing out reasons the person may be in pain and solutions to that pain.
Sure, there may be some interesting ideas there, but it doesn't really do OP any good without describing where the pain is, when it started, if they have any other known conditions, etc. etc.
I know you think you were helping with this comment, but you really weren't.
The comment that is unhelpful is the one that has to be voiced but refuses to participate. You're just creating a clamor where a conversation used to be by adding your noise. If you aren't going to participate in the answer beyond saying "I'm not going to answer." Then just don't.
If price is a consideration, you might consider two 10 TB hard drives on machines on two home gbps Internet connections. It's highly unlikely that both would go down at the same time, unless they were in the same area, on the same ISP.
Just use two A records for the one DNS name, and let the clients choose.
The other way is to have two names, like dl1 and dl2, and have your download web page offer alternating links, depending on how the downloads are handled.
You very rarely can do multi-ISP bonding, often not even with multiple lines from the same ISP, unfortunately.
I'd suggest looking into "seedboxes" which are intended for torrenting.
I suspect the storage will be a bigger concern.
Seedhost.eu has dedicated boxes with 8TB storage and 100TB bandwidth for €30/month. Perhaps you could have that and a lower spec one to make up the space.
Prices are negotiable so you can always see if they can meet your needs for cheaper than two separate boxes.
Yep, resellers of dedicated machines rent servers in bulk so you can often get boxes for way cheaper than you would directly from the host. Take a look at https://hostingby.design as an example.
I've been using a HostingBy.Design seedbox (formerly Seedbox.io) for years to distribute content to my patrons for 3 years. They have excellent uptime and their customer service is knowledgeable.
> I'd suggest looking into "seedboxes" which are intended for torrenting.
Though be aware that many (most?) seedbox arrangements have no redundancy, in fact some are running off RAID0 arrays or similar. Host has a problem like a dead drive: bang goes your data. Some are very open about this, afterall for the main use case cheap space is worth the risk, some far less so…
Of course if the data is well backed up elsewhere or otherwise easy to reproduce or reobtain this may not be a massive issue and you've just got restore time to worry about (unless one of your backups can be quickly made primary so restore time is as little as a bit of DNS & other configuration work).
Sounds like you could find someone with a 1Gbps symmetric fiber net connection, and pay them for it and colo. I have 1Gbps and push that bandwidth every month. You know, for yar har har.
And that's only 309Mbits/s (or 39MB/s).
And a used refurbished server you can easily get loads or ram, cores out the wazoo, and dozens of TB's for under $1000. You'll need a rack, router, switch, and batt backup. Shouldn't cost much more than $2000 for this.
you can definitely do this at home on the cheap. As long as you have a decent internet connection, that is ;)
10TB+ harddisks are not expensive, you can put them in an old enclosure together with a small industrial or NUC PC in your basement
I current have 45 WUH721414ALE6L4 drives in a Supermicro JBOD SC847E26 (SAS2 is way cheaper than SAS3) connected to an LSI 9206-16e controller (HCL reasons) via hybrid Mini SAS2 to Mini SAS3 cables. The SAS expanders in the JBOD are also LSI and qualified for the card. The hard drives are also qualified for the SAS expanders.
I tried this using Pine ROCKPro64 to possibly install Ceph across 2-5 RAID1 NAS enclosures. The problem is I can't get any of their dusty Linux forks to recognize the storage controller, so they're $200 paperweights.
I wrote a SATA HDD "top" utility that brings in data from SMART, mdadm, lvm, xfs, and the Linux SCSI layer. I set monitoring to look for elevated temperature, seek errors, scan errors, reallocation counts, offline reallocation, and
probational count.
They don't really maintain the regular Sync client anymore, only the expensive enterprise Connect option. My wife and I used Resilio Sync for years, but had to migrate away, since it had bugs and issues with newer OS versions, but they didn't care to fix them. Let alone develop new features.
196 comments
[ 3.4 ms ] story [ 326 ms ] thread> If your monthly egress data transfer is greater than your active storage volume, then your storage use case is not a good fit for Wasabi’s free egress policy.
https://wasabi.com/paygo-pricing-faq/
[0] https://github.com/webtorrent/webtorrent
One multi-vendor, zero-knowledge, HA solution is https://tahoe-lafs.org/trac/tahoe-lafs
For rock-solid public hosting Cloudflare is probably a much better bet, but you're also paying 7 times the price. More than a dedicated server to host the files, but you get more on other metrics.
I would love to use them as they hang out here, but their prices are way too expensive for my needs.
* based on fair use
at 250 TB/mo:
> In order to continue hosting your servers with us, the traffic use will need to be drastically reduced. Please check your servers and confirm what is using so much traffic, making sure it is nothing abusive, and then find ways of reducing it.
Unlimited rarely is.
Looks like backblaze (other post below) has a free bandwidth and cheap storage solution
Bandwidth Alliance introduced me to bunny.net, interesting find.
At scale, you'll pay a couple thousand dollars for Class B operations on R2, and another bunch for storing the 10 TB in the first place, but that's relatively cheap compared to other offerings where you'd pay for metered egress bandwidth.
https://developers.cloudflare.com/r2/pricing/ https://r2-calculator.cloudflare.com/
The whole point of R2 is to remove predatory practices of egregious charging for egress, and if somehow they went back on this promise that would be a very bad PR.
support seems non existent cos no one answers emails or web chat…
Thank you!
Throw that in RAID10 and you'll have 12TB usable space with > 300TB bandwidth.
But if splitting your traffic across multiple servers is possible you can also get the €20 storage box and put a couple Hetzner Cloud servers with a caching reverse proxy in front (that's like 10 lines of Nginx config). The cheapest Hetzner Cloud option is the CAX11 with 4GB RAM, 40GB SSD and 20TB traffic for €3,79. Six of those plus the Storage Box gives you the traffic you need, lots of bandwidth for usage peaks, SSD cache for frequently requested files, and easily upgradable storage in the Storage Box, all for €42. Also scales well at $3,79 for every additional 20TB traffic, or $1/TB if you forget and pay fees for the excess traffic instead.
You will be babysitting this more than the $150/month cloudflare solution, but even if you factor in the cost of your time you should come out ahead.
There is always the hidden cost of not spending time on activities that are core to your business (if this is indeed for a business) that would make multiples of the money CF costs you.
I have been playing more and more with UTM in the Mac world and it's encouraging how mature it seems already and hopefully can be picked up into NixOS, Sandstorm, etc.
I like proxmox more personally, however have changed my stance recently where nix and sandstorm could just be run in a proxmox vm and then provide more of an IaaS role. The newer versions of ProxMox are even easier and they were pretty OK the past 5-7 years.
(for those not on the same page, I’m talking from a position of substantial experience with all 3 major clouds)
Plus, these days the maintenance burden of the OS layer is really heavily overstated. With certain self-updating open-source container OSes one doesn’t even really have to think about patches and all that ancient crap.
The real appeal of the big players in my mind is only in one use case - scale. If you need 10k servers for heavy “big” data processing like in genomics or ‘AI’ (whatever that means), only then they start to be indisposable. Otherwise, the considerable burden of training all personnel on proprietary APIs is just not worth it - it literally costs less to buy and configure your own system (or a traditional VPS or dedicated server). Cloud architects ain’t cheap!
There’s always a limit, that might be measured in TB, PB or EB, and may be what you determine practical or not, but it’s there
Of course the downside is that, if you need to download that 10TB, you'll be out $900! If you're worried about recovering specific files only this isn't as big an issue.
Also read their free egress policy first: https://wasabi.com/paygo-pricing-faq/#free-egress-policy
Note that there is Glacier and Glacier Deep Archive. The latter is cheaper but longer minimum storage periods. You can use it as a life cycle rule.
See: https://wasabi.com/cloud-storage-pricing/#cost-estimates
They could really do with making the bandwidth option on this calculator better.
Wasabi isn't meant for scenarios where you're going to be transferring more than you're storing.
Which is basically a roundabout way of saying, they're offering storage for backups, not for content distribution.
https://wasabi.com/paygo-pricing-faq/#minimum-storage-durati...
https://www.youtube.com/watch?v=3t6L-FlfeaI
Medium- and smaller-scale can often be more flexible because they don't have to incur the pain of nonuniformity as scale increases. While they may not be able to afford optimizations or discounts with larger, standardized purchases, they can provide more personalized services large scale cannot hope to provide.
Eventually, all outages are black swan events. If you have 1000 independent instances (i.e., 1000 customers), when the unexpected thing hits, you’re still 99.9% available during the time when the impacted instance is down.
Also, you can probably permanently prevent the black swan from hitting again before it hits again.
User: "I want to serve 5TB."
Guru: "Throw it in a GKE PV and put nginx in front of it."
Congratulations, you are already serving 5TB at production scale.
It's kind of like someone going to a group of doctors and saying "I'm in pain", and then the doctors start throwing out reasons the person may be in pain and solutions to that pain.
Sure, there may be some interesting ideas there, but it doesn't really do OP any good without describing where the pain is, when it started, if they have any other known conditions, etc. etc.
I know you think you were helping with this comment, but you really weren't.
That is yourdomain.com -> IP_ISP1, IP_ISP2
Going the other way from yourserver -> outside would indicate some sort of bonding setup.
It is not trivial for a home lab.
I use 3 ISPs at home and just keep each network separate (different hardware on each) even though in theory the redundancy would be nice.
The other way is to have two names, like dl1 and dl2, and have your download web page offer alternating links, depending on how the downloads are handled.
You very rarely can do multi-ISP bonding, often not even with multiple lines from the same ISP, unfortunately.
I suspect the storage will be a bigger concern.
Seedhost.eu has dedicated boxes with 8TB storage and 100TB bandwidth for €30/month. Perhaps you could have that and a lower spec one to make up the space.
Prices are negotiable so you can always see if they can meet your needs for cheaper than two separate boxes.
Though be aware that many (most?) seedbox arrangements have no redundancy, in fact some are running off RAID0 arrays or similar. Host has a problem like a dead drive: bang goes your data. Some are very open about this, afterall for the main use case cheap space is worth the risk, some far less so…
Of course if the data is well backed up elsewhere or otherwise easy to reproduce or reobtain this may not be a massive issue and you've just got restore time to worry about (unless one of your backups can be quickly made primary so restore time is as little as a bit of DNS & other configuration work).
And that's only 309Mbits/s (or 39MB/s).
And a used refurbished server you can easily get loads or ram, cores out the wazoo, and dozens of TB's for under $1000. You'll need a rack, router, switch, and batt backup. Shouldn't cost much more than $2000 for this.
I tried this using Pine ROCKPro64 to possibly install Ceph across 2-5 RAID1 NAS enclosures. The problem is I can't get any of their dusty Linux forks to recognize the storage controller, so they're $200 paperweights.
I wrote a SATA HDD "top" utility that brings in data from SMART, mdadm, lvm, xfs, and the Linux SCSI layer. I set monitoring to look for elevated temperature, seek errors, scan errors, reallocation counts, offline reallocation, and probational count.
It's like Dropbox except peer to peer. So it's free, limited only by your client side storage.
The catch is it's only peer to peer (unless they added a managed option), so at least one other peer must be online for sync to take place.