One thing I noticed is that some limits seem rather low, I'm not sure if this is for the beta only or by design. Particularly the 1 Gbit/s limit per bucket and the 10 buckets for all projects.
Is it typical to have such per-bucket limits? This means that you might have to take them into account when deciding on how to organize the data to avoid bottlenecks.
Having implemented object storage at a service provider - it's extremely rare for any customer to have 1gbit/sec sustained throughput on an account, much less a bucket.
Those that do likely will already have an actual sales rep and the means to get these restrictions lifted, with a direct line to engineering.
Hetzner certainly markets to a different crowd, which I suspect is why these limits are being put in place ahead of time. It's very difficult to design around noisy neighbor, and will take some battle tested time in production to get right for the "average Joe" style customer who won't engage engineering resources to work with a provider to avoid hot spots and impacting other customers.
I'm more concerned about bursts, not sustained throughput. When handling large files a single client can load at 1 Gbit/s if the other side supports that.
My suspicion was also that they're intentionally targeting a different crowd than S3 and focusing on their niche here.
Definitely agreed. I would be pretty surprised if this was burst capacity though. I have no insight as to how Hetzner implemented their object storage, but in general terms it's pretty difficult to ratelimit burst without adverse effects. Sustained is a lot easier to implement since you are typically having to have multiple load balancers coordinate to achieve such results. Attempting to do this at sub-minute time intervals is pretty tricky at scale.
Presumably it's built on top of their existing VPS and dedicated server boxes, which AFAIK only have 1gbit uplinks, so they'd only be able to exceed that speed when the bucket is split between multiple boxes.
Not quite correct. Their dedicated servers base configuration have 1gig. But when ordering you can add 10gig as an easy option (or add later).
The 1 gig is too cheap to meter with free traffic. If you choose 10gig you pay for the traffic. Those who need it might use it :-)
Their VPS hosts have a 10GBit uplink, but they offer no bandwidth guarantees per VPS and state you can expect 300-500MBit/s[1]. Their dedicated servers have a 1GBit uplink with guaranteed 1Gbit/s bandwidth.
The Hetzner Cloud ARM instances I believe are on 2.5Gb or 1Gb links. I've never been able to get > 1Gbit on the ARM instances, but easy getting it on x86_64.
At the Hetzner Summit two a few weeks ago they presented the servers used for this.
I am not sure about the exact specifications anymore, but they are building dedicated servers for this with custom chassis where each server has a ton of drives and I think they each had 40 GBits networking. These are special servers that are not available to customers directly.
Just got a reply from them regarding the bucket amount and permission amount limits and both are in fact beta limits.
No idea about the bandwidth though.
I was thinking of switching to BlackBlaze as I'm currently using Digital Ocean Spaces -- however I might look at Hetzner again -- not revolutionary but quite competitive.
I know, but I prefer fair pricing over free in situations like this. There are plenty of stories going around of CF forcing users to upgrade to an enterprise plan due to their usage. When there is a price tag, at least I know that won't happen to me (not that my usage would be on CFs radar anyway, it's the principle of it).
I am always scared to put too much trust in new Object Storage services. I love Hetzner and similar but until this new service has been around for a while, I'd only use them for stuff I can afford to lose. From the outside as a consumer of these services you do not know how robust they actually are on the inside.
I remember the data loss from OVH where they put backups in the same building as the primaries and people only found out about this once a fire took out the whole building:
You know that if US East suffers a FULL data loss that the recovery would take weeks with the question if it would even be possible. That's what happened to ovh... it wasn't just one building.
> It's not like there is a mirror datacenter just two blocks away
Isn't that exactly what Availability Zones are for? They're physically separate[0] datacenters and each one contains a copy of each S3 object (unless using the explicit single-zone options)
It's also straightforward (although not necessarily that cheap) to replicate S3 objects to another region
If us-east-1 ever suffered a “FULL” data loss, it would be a company-ending event for so many companies that it would practically end society as we know it.
OVH’s failure was a single building. That’s the problem with a lot of server hosters - even Google has their availability zones all co-located in the same building, so a physical event like a fire could take down an entire region. AWS has AZs in physically separate locations, each with 1+ separate DCs.
I take the typical formulation (e.g., [1]), and translate it into:
- Keep 3 copies of your data: production + 2.
- Keep 2 snapshots, stored separately.
- Keep 1 snapshot on-hand (literally in your possession), or with a different provider.
It's great to see more options for "different provider". If I were an early adopter, I would either target this as my on-Hetzner snapshot of my Hetzner-hosted data, and replicate it elsewhere; or I would consider trialing it as a backup to non-Hetzner data. To your point, though, I'd probably wait on the latter until it has gone through some growing pains.
Yeah, don't use Sippy. It just stopped working after 2 weeks on multiple buckets and support came back with "have you tried turning it off and on again". It's not production ready.
Also R2 is extremely basic compared to S3. For example only trivial permissions are supported and auth tokens are bound to an actual account.
Couldn't agree with you more. Last spring talked to a senior Hetzner engineer who wasn't too sure about the Object Storage design/implementation and was happy he wasn't involved.
As everyone knows, this is in sharp contrast with serious engineering operations like Amazon or Google, where there are no disagreements on technical designs. Ever.
I should mention Tigris[0] here. They're also a new Object Storage service, but they have this two-way replication facility with another S3-compatible service. The primary purpose they built it for is to mirror files from your existing S3 to Tigris as files are requested.
However they also have an option to copy files that are added to Tigris, to S3 automatically [1] (`--shadow-write-through`). I asked their founder if it's okay to use it as an extra redundancy continuously instead of a one-time migration, and they said they have no issues with it.
I toured a data center in Tornado alley back when leasing cages was pretty common. I asked them about disaster planning regarding getting completely wiped off the map and they sorta scoffed at me. Literally two weeks later a tornado missed them by about a 1/4 mile. Would have loved to be a fly on the wall after that.
another story, a small company I worked for contracted with a small data center. They did things right and ran a tight ship. However something happened like a lightning strike or so other electrical fault and a critical component welded itself into a position they could t get out of. I wish I could remember the details better but they were down multiple days.
Years ago there was a fire in OVH who I always trusted as reliable and stable, never needed to worry.. Lost a lot that day. Probably I could have had offsite backup managed myself though. I see no difference here, you can manage your own disaster recovery.
From what I see, this actually might be a great backup/recovery solution for S3 in my case at least.
Maybe if OVH had installed an automatic fire extinguishing system or an electric mains cut-off switch, the firefighters wouldn't have struggled with meter-long electrical arcs and could put out the fire quicker.
Yes, that hosting center was unsafe as hell. All of OVH's early year featured a lot of duct-tape and daring corner-cutting - that is how they were incredibly cheap. But nowadays they are just as mainstream industrial as their competitors and the 2021 fire marked the end of an era... It is sad that it happened as they were already turning the company around to being an established player.
> But nowadays they are just as mainstream industrial as their competitors
Shame their control panel is massively behind their competitors, it's the worst cloud control panel I've had the experience of using, even worse than Azure and Google.
It's slow slow slow slow slow and horribly buggy, did I mention it's slow?
People seem to have this misconception that "cloud computing" is some kind of magic bullet that guarantees 100% durability. News flash, your data is still being stored on physical hard drives, in some data center. If the building burns down in a fire, of course your data will be gone, the hard drives are hosed, it's not magic. You really have no one to be mad at other than yourself...
Most of the "big" cloud providers aren't as negligent as OVH though. That entire datacenter was just one blunder after the other (wood structures, barely functional fire suppression, no proper power shut down, insane replication strategy that caused most of the replicas to burn in the very same fire in the same building...)
> S3 Standard, [...] redundantly store objects on multiple devices across a minimum of three Availability Zones in an AWS Region. An Availability Zone is one or more discrete data centers with redundant power, networking, and connectivity in an AWS Region. Availability Zones are physically separated by a meaningful distance, many kilometers, from any other Availability Zone, although all are within 100 km (60 miles) of each other.
I'd probably look at using S3 as a backup and only have the working copy with Hetzner. Not sure how much more expensive that would be, the pricing for the non-standard S3 and glacier does seem to require a bit more work to understand properly.
Terrific to see native object storage in Hetzner. I think object storage is become a staple for most providers now.
I can see that Hetzner is starting with WORM capabilities [0]. I wonder if this nascent product is successful they'll consider some of the features that other providers offer such as mutating objects, storage policies, tiering, and conditional writes. I appreciate you've got to start somewhere and this looks terrific for an opening gambit. Kudos Hetzner.
[0] > Object Storage is mainly used to store and share data as it is not possible to edit any data that you uploaded to a Bucket (objects are immutable). So the main purpose of Object Storage is "WORM", which is short for, "Write once, read many [times]".
Unfortunately their signature dirt cheap bandwidth isn't so cheap in Singapore, you only get 1TB inclusive instead of 20TB and the overage rates are 7.4x higher than their EU and NA datacenters.
That's a problem but still cheaper than UpCloud. I do find Cloudflare pricing is more attractive for read heavy website and startup plan is useful for us.
If you're trying to sensibly engage people in discussion about this then you probably want to provide some context. I personally have no idea what you're referring to.
The context is that Hetzner is well known to delete accounts of users that they suspect of running processes or storing files that have the word "bitcoin" or any other word that Hetzner does not approve of. Hetzner does this without warning too. It doesn't matter if there is no network service, as local processing is sufficient for this action.
I can find references to less egregious incidents - i.e. they ban running actual crypto software. Do you have any links I could read to find out more about the more extreme end of your claim? i.e. " storing files that have the word "bitcoin" or any other word that Hetzner does not approve of"?
If this is true then it's a reason for anyone to avoid Hetzner as accidentally triggering it would be possible on almost all websites. If you're exaggerating for dramatic effect then that's fine I guess - I'd just like to find out the real state of affairs.
I am not exaggerating it. I stand by what I said. My comment has nothing to do with running actual crypto software, implying that you don't have to be running it to get your services wiped. You also don't have to be using the internet. The ultimate discretion is up to the mood of Hetzner staff and up to when they get around to seeing what you do. If the word "bitcoin" occurs more than randomly in your usage, expect it to be ended without warning. I don't believe it to be an automated process.
they also block external servers on their network that are somehow related to crypto
so if you have some kind of service/api/trading bot/scraping/monitoring hitting a somewhat related to crypto node you may find that you will have no routing from hetzner core router to said server
and support is as friendly as other comments describe :)
> and support is as friendly as other comments describe :)
In my experience, the support is not friendly, but it is no nonsense. Every time I've reached out, they responded quickly, tersely, and took appropriate actions. While I don't personally mind formalities, there's something to be said for their efficacy.
I know what you meant, just saying that somebody reading "brand of cheap but decent" may read as "brand of cheap stuff" and for clarification that's not what they represent.
seriously, with an opportunity like that it ABSOLUTELY does not matter if your acronym matches. They must now suffer publicly, forever, for we know where their unpoetic hearts lie.
i was interested in hetzner and their services, until i tried to sign up and, after they verified my payment card and all went well, i instantly got account suspended email and that was it. and i am glad it happened. i cannot imagine having a tiny problem later on, costing me money, with a company like that.
btw backblaze is the best object storage offer on the planet at this time(i have research all options). second would be wasabi.
Backblaze gives you 3x free egress of your stored data per month. Wasabi gives you only up to 1x. So it's like you have 1 TB of data and you can download 1 TB each month via Wasabi for free, but 3TB with backblaze. Since traffic is the most costly expense when it comes to object storage, this is priceless.
This is highly dependent on the use case. Backblaze gives you 3x free egress, but after you hit that you pay for any additional egress. Wasabi has terms of use not to exceed 1x your storage, but is not in their model to pay egress. As a long-term storage user, you can restore a full system-wide backup without any concern of charges at Wasabi(typically, you're not restoring everything, just recent backups), as long as you are not consistently doing it. Backblaze will get you more egress for sharing data over the public internet, but if you need more, you will get charged.
> typically, you're not restoring everything, just recent backups
Not knowing the pricing difference between the two and assuming they are similar, I would favor Backblaze as it would allow me to exceed the limit if I needed. Based on how you framed it, I would expect that with Wasabi you might hit a hard limit.
Wasabi doesn't have an egress hard limit and doesn't charge for egress. You get consistent pricing, and if you need to recover everything, it's not an issue at all, and you won't pay for it.
"The reasonable use egress policy indicates that if your monthly downloads (egress) are greater than your active storage volume, then your storage use case is not a good fit for Wasabi’s free egress policy, and we reserve the right to limit or suspend your service."
If you're using Wasabi for normal backup data storage, you shouldn't worry about egress. It is meant to prevent malicious users from uploading data and using up all the egress bandwidth, for example, a 500GB user egressing 5TB with public access or using Wasabi as a dump point to upload in one location and download in another region 1:1 ratio. As your storage goes up, your available consistent monthly egress goes up. It only becomes an issue if you abuse the account by uploading/downloading in a 1:1 ratio on a consistent basis.
What happens is you get an email from support asking if something changed in your use case. If so, they will help troubleshoot it(Think of a CDN scenario, where the CDN gets misconfigured). You also have an egress monitor for suspicious activity in case you aren't normally downloading all your data, and then you see a rise in egress https://docs.wasabi.com/docs/en/whats-new?highlight=egress#e....
problem with wasabi is that their ToS are not concrete with their conditions. backblaze is specific. black on white. no buts of ifs. you know exactly what you pay for. with wasabi, you don't. if you run a business, wasabi is not the way to go. if you run personal things, it is an ok option. but in the end, 1x vs 3x is such a major difference, that there is just no point in discussing it.
I don't understand the pricing. There's a base fee of 0.0081 €/h, which I guess then is translated into 5.00 €/month? Or is that 0.0081 €/h + 5.00 €/month? And that gives me a 1 TB-hour of free usage, and if I use the object storage for the whole month, it's 720 TB-hours of free usage, except if the month has more or less than 30 days (so it's actually between 672 TB-hour and 744 TB-hour of free quota of storage)? And those TB-hours expire at the end of the month, so you might as well store files if you're under a 1 TB?
Agree that the pricing model is highly unclear (which is usual for cloud services).
It does not tell me if I should count hours in hour-of-the-day, or lapsed time. Also the example tries to demonstrate a case of "you don´t have to pay extra", but then falls silent. Nice, not the info I am looking for.
What about envisioning a customer who asks «what am I going to pay? Specify it right now, right here».
Yeah we are big fans of Hetzner and this pricing model makes no sense. I'm still trying to wrap my head around it and the strange limits associated with it.
It's not explained very clearly and made more complicated by being charged hourly instead of monthly, but essentially there's a minimum charge of €5 per month, which includes 1TB of storage and about 1TB of bandwidth.
That's not how I read it or what their example says. As I understand it, if you create a bucket and then delete it again within the hour, you only pay for one hour. I think the 5€/month is if you have active buckets the whole month, since it's less than if you actually had to pay for all the hours.
That's correct, that's why the hourly billing makes it more complicated. If you don't use cloud storage the entire month you only pay for the time you do use it, and you only get included quota proportional to the amount you pay for. The sum of hourly costs is capped at €5 per month, which is also how it works for their VPSes.
I am trying to register on Hetzner and not able to. I get the account verification email but after verification I am not able to login to the account and also do not receive any Hetzner email with login or account information. Strange. Yeah, I am from from India, so maybe that's why. Anyone else? Who is not able to register?
Might be a stupid question, but why did Hetzner take off? I feel like I’d barely heard of them before last year, but now they’re the go-to hosting suggestion from many. Are they just a solid host, or do they offer something extra? Their website makes them looks almost generic, so I assume it’s not marketing.
I got a pretty nice dedicated server with 2x3 TB HDD + 20 TB/month of traffic for just 25€/month. Was great for hosting >1 TB of video content publically.
It's very cheap. They've been around for years. I've been using them for my personal server for the past 2-3 years without problems. The only problems I have seen from them are the control plane API going down (so, like, creating new servers isn't available).
They have been around for a long time. I heard people praising them already in startup events 10+ year ago (I'm not that active in the scene myself any more). They are known for great price/quality ratio for the dedicated servers. There are some gotchas, as you might get random crappy hardware or whatever. The firms I have been working for and which decided to go build their infra mostly with hetzner, I would estimate that they have saved millions in comparison to common cloud services. The staff often complains because managing your own dedicated servers is a bit more work than going with something like AWS, but if you put in the numbers, the amount of money saved in the end can be insane.
I've associated them for a while with uber cheap hosting, while also being allegedlly* reasonable (Most of the time) and reliable
* Couldn't tell you, as they flagged my account and asked me for an ID Document I couldn't provide, Maybe I could have sorted it with Customer Service, but instead I just picked a different german hosting
Their accounts department is egregious. Amazing offerings but it's so frustrating.
I had some servers that I had autopay setup for I thought. But the bill went over as unpaid and they shut off the servers. Then I noticed & tried to login to their web account system but my account was locked. Support wanted me to send a wire transfer. I begged and begged, told them how it would cost as much as by debt, and they opened my account for ~36 hours, but after some delay, and I was away vacationing. I sent them the wire transfer almost two months ago now & still nothing from their accounts/billing people, even though the transfer had the account & payment #'s. It's been so frustrating, & so unnecessarily over the top bad & they seem to love making it worse every step. What the hell Hetzner?
They became a serious VPS contender amongst DigitalOcean, Linode, and other similar providers. DigitalOcean raised prices while Hetzner gave more powerful server capacity (read double) at lower price points. Hetzner has been increasingly popular in self-hosting communities.
I started using their services since last year for some of the operations, including business, and it's been rock solid. I plan on increasing usage, but will hold on this new object storage offering.
I don't know about the last year, maybe they ramped up their marketing. But they've been pretty popular in my circle for the past 6-7 years, mostly because price/performance ratio is really favorable. I use it for the past ~3y.
They are cheap. People have more of a focus on cost recently which pushes back against AWS. As far as vibes go, more people seem to be getting back into managing their own machines and self-hosting. I think Linode used to fill that void but for various reasons they relinquished that opportunity.
They've existed since 1997 and have always been a solid host with good pricing on both bare metal and later VPS services. They seem to have managed to survive the Cloud onslaught as well by continuing to provide reliable service for good prices.
Perhaps you heard less about them because they're based in Europe and thus less hype sorrounds them?
I was with linode for a loooong time for my personal VM-in-cloud (like 15 years). But I got an itch to check around to see if there was a cheaper option earlier this year and found that Hetzner would give me literally twice the machine for about two thirds the cost. ($6/mo vs €4~/mo, 1gb nanode vs. CX22 so 1vcpu/1gb to 2vcpu/4gb and 25gb disk to 40gb)
So far I'm happy with the service. Linode wasn't bad per se, Hetzner just offered more for less. Their admin console is a bit more spartan in comparison to Linode's, but that's not a place I spend much time anyway. If Russia glasses Finland and Germany I'd have to find another provider but I'm pretty sure I'd have bigger problems at that point than where to park my wordpress blahg and irc session, lol.
I think if they glass Finland/Germany, then they know the US will next attempt to glass Russia, so they will launch it all at once, which of course results in all of Russia, nearly all of Europe, and the USA all being glassed and China picking up the pieces. So it’s probably better to learn Mandarin and find an AWS like data center there.
Hetzner is well known and beloved in Europe for a decade or two, so it didn't really start out of nowhere. Maybe you (and most of HN) became aware of them, when they opened their American datacenters in 2021 & 2022, and became a viable option for people outside of Europe.
I've been with them ~15 years, and they have been solid. Back then I had a dedicated server I was developing an online multiplayer java game on, great fun! until the whole java webstart certificate fiasco. A few years back I was porting that old game over to wasm, and they added the mime type support within a few hours of asking. They are some very knowledgeable engineers, and maybe that is reflected in the simple website design. But I like that simple productive interface, and dread the day the MBAs or marketing teams get their hands on it.
It seems the Hetzner's docs say they use MinIO to expose their S3 managed service. I guess everything will be the same as MinIO upstream features and contigurations
I can only find mentions of the minio client as a way of accessing their S3 service. Their introductory page[0] says "Any data you save in your Bucket is saved in a Ceph cluster.", which suggests they use Ceph's radosgw that offers the S3-compatible API[1].
> Our S3-compatible Object Storage provides you with storage capacity for saving data in "Buckets". Any data you save in your Bucket is saved in a Ceph cluster.
Admittedly I got my money's worth, for about 3 EUR for the very basic plan of 250 GB storage, I get upload speeds of about 6-9 MB/s and download speeds of about 25 MB/s, which is okay for what I'm trying to do. That said, it doesn't seem like there's a way to create additional users with access to only specific buckets not all of them for the same service and the overall offering does feel a bit jank (e.g. when you don't have an active service, you can't log into Contabo at all and while their Object Storage site is nice, the regular VPS one feels functional but dated).
What I'm saying is that when it comes to budget options, Hetzner will probably do great, since they have a good track record and it's not like there are that many other alternatives out there!
Of course, I could have also gone with just self-hosting with MinIO, Garage or SeaweedFS, as long as the VPS that I'd get would also have enough storage. It's nice that there are self-hosted options, too, I almost dread when I see bespoke object storage solutions at work, either developed before S3 was a thing or because the devs just didn't know or had a case of NIH, so I have to look at how a bunch of blobs are passed through servlets and serialized/deserialized, as well as deal with custom metadata and permission mechanisms.
192 comments
[ 3.9 ms ] story [ 221 ms ] threadIt's nice to see more competition in this space.
I'm a big Hetzner fanboy, quite sad that pricing is't that competetive...
Is it typical to have such per-bucket limits? This means that you might have to take them into account when deciding on how to organize the data to avoid bottlenecks.
Those that do likely will already have an actual sales rep and the means to get these restrictions lifted, with a direct line to engineering.
Hetzner certainly markets to a different crowd, which I suspect is why these limits are being put in place ahead of time. It's very difficult to design around noisy neighbor, and will take some battle tested time in production to get right for the "average Joe" style customer who won't engage engineering resources to work with a provider to avoid hot spots and impacting other customers.
My suspicion was also that they're intentionally targeting a different crowd than S3 and focusing on their niche here.
[1]: https://docs.hetzner.com/cloud/technical-details/faq#what-ki...
I am not sure about the exact specifications anymore, but they are building dedicated servers for this with custom chassis where each server has a ton of drives and I think they each had 40 GBits networking. These are special servers that are not available to customers directly.
I run minio on hetzner and wouldn't send stuff across some other network to backblaze.
We use it for multiple use cases and all of them require lots of retries and error handling
essentially they're build for backups use case, with lots of spinning rust, any kind of "working data" easily underperforms
backups work fine because not time sensitive and retries handle the problems
How much debt do they have on their balance sheet?
I remember the data loss from OVH where they put backups in the same building as the primaries and people only found out about this once a fire took out the whole building:
https://blocksandfiles.com/2023/03/23/ovh-cloud-must-pay-dam...
Isn't that exactly what Availability Zones are for? They're physically separate[0] datacenters and each one contains a copy of each S3 object (unless using the explicit single-zone options)
It's also straightforward (although not necessarily that cheap) to replicate S3 objects to another region
[0] https://aws.amazon.com/ec2/faqs/?nc1=h_ls#Platform
OVH’s failure was a single building. That’s the problem with a lot of server hosters - even Google has their availability zones all co-located in the same building, so a physical event like a fire could take down an entire region. AWS has AZs in physically separate locations, each with 1+ separate DCs.
I take the typical formulation (e.g., [1]), and translate it into:
- Keep 3 copies of your data: production + 2.
- Keep 2 snapshots, stored separately.
- Keep 1 snapshot on-hand (literally in your possession), or with a different provider.
It's great to see more options for "different provider". If I were an early adopter, I would either target this as my on-Hetzner snapshot of my Hetzner-hosted data, and replicate it elsewhere; or I would consider trialing it as a backup to non-Hetzner data. To your point, though, I'd probably wait on the latter until it has gone through some growing pains.
[1] - https://www.backblaze.com/blog/the-3-2-1-backup-strategy/
S3 egress, on the other hand, is so expensive you can often justify putting a writethrough cache for your entire working set at Hetzner...
edit: apparently not writethrough: https://developers.cloudflare.com/r2/data-migration/sippy/
Also R2 is extremely basic compared to S3. For example only trivial permissions are supported and auth tokens are bound to an actual account.
However they also have an option to copy files that are added to Tigris, to S3 automatically [1] (`--shadow-write-through`). I asked their founder if it's okay to use it as an extra redundancy continuously instead of a one-time migration, and they said they have no issues with it.
[0] https://www.tigrisdata.com
[1] https://www.tigrisdata.com/docs/migration/#starting-the-migr...
This is a massive claim that unless they're storing everything in memory I don't know how you can get close to Redis speed.
Even the flash-based Redis alternatives can't get close (but are good enough)
another story, a small company I worked for contracted with a small data center. They did things right and ran a tight ship. However something happened like a lightning strike or so other electrical fault and a critical component welded itself into a position they could t get out of. I wish I could remember the details better but they were down multiple days.
https://www.switch.com/switch-shield/
The Switch one I'm in (Grand Rapids Pyramid), the servers are about 3/4's underground.
From what I see, this actually might be a great backup/recovery solution for S3 in my case at least.
A great writeup of the incident and its context, three years afterwards: https://www.datacenterdynamics.com/en/analysis/ovhcloud-fire...
Shame their control panel is massively behind their competitors, it's the worst cloud control panel I've had the experience of using, even worse than Azure and Google.
It's slow slow slow slow slow and horribly buggy, did I mention it's slow?
From https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDu... :
> S3 Standard, [...] redundantly store objects on multiple devices across a minimum of three Availability Zones in an AWS Region. An Availability Zone is one or more discrete data centers with redundant power, networking, and connectivity in an AWS Region. Availability Zones are physically separated by a meaningful distance, many kilometers, from any other Availability Zone, although all are within 100 km (60 miles) of each other.
rclone to somewhere else at least.
And 3-2-1 rule for backups if you're serious.
The trick is not to put all eggs in the same basket and it's easier to do that when you're paying much less than you'd pay on providers like AWS.
I can see that Hetzner is starting with WORM capabilities [0]. I wonder if this nascent product is successful they'll consider some of the features that other providers offer such as mutating objects, storage policies, tiering, and conditional writes. I appreciate you've got to start somewhere and this looks terrific for an opening gambit. Kudos Hetzner.
[0] > Object Storage is mainly used to store and share data as it is not possible to edit any data that you uploaded to a Bucket (objects are immutable). So the main purpose of Object Storage is "WORM", which is short for, "Write once, read many [times]".
My only hope is that they won't make it more complex than it should be in an effort to match AWS.
You can see it in the minio repo and issues .
Well - not that well known.
I can find references to less egregious incidents - i.e. they ban running actual crypto software. Do you have any links I could read to find out more about the more extreme end of your claim? i.e. " storing files that have the word "bitcoin" or any other word that Hetzner does not approve of"?
If this is true then it's a reason for anyone to avoid Hetzner as accidentally triggering it would be possible on almost all websites. If you're exaggerating for dramatic effect then that's fine I guess - I'd just like to find out the real state of affairs.
so if you have some kind of service/api/trading bot/scraping/monitoring hitting a somewhat related to crypto node you may find that you will have no routing from hetzner core router to said server
and support is as friendly as other comments describe :)
In my experience, the support is not friendly, but it is no nonsense. Every time I've reached out, they responded quickly, tersely, and took appropriate actions. While I don't personally mind formalities, there's something to be said for their efficacy.
Keen to see what is available to protect public buckets though to prevent huge bills from malicious actors
They know what they’re doing and they’re doing it well, cheap is more side effect than primary goal.
Wonderful tool. Useful for figuring out where some of these cheaper S3 alternatives make their cuts.
btw backblaze is the best object storage offer on the planet at this time(i have research all options). second would be wasabi.
Not knowing the pricing difference between the two and assuming they are similar, I would favor Backblaze as it would allow me to exceed the limit if I needed. Based on how you framed it, I would expect that with Wasabi you might hit a hard limit.
Wasabi doesn't have an egress hard limit and doesn't charge for egress. You get consistent pricing, and if you need to recover everything, it's not an issue at all, and you won't pay for it.
"The reasonable use egress policy indicates that if your monthly downloads (egress) are greater than your active storage volume, then your storage use case is not a good fit for Wasabi’s free egress policy, and we reserve the right to limit or suspend your service."
If you're using Wasabi for normal backup data storage, you shouldn't worry about egress. It is meant to prevent malicious users from uploading data and using up all the egress bandwidth, for example, a 500GB user egressing 5TB with public access or using Wasabi as a dump point to upload in one location and download in another region 1:1 ratio. As your storage goes up, your available consistent monthly egress goes up. It only becomes an issue if you abuse the account by uploading/downloading in a 1:1 ratio on a consistent basis.
What happens is you get an email from support asking if something changed in your use case. If so, they will help troubleshoot it(Think of a CDN scenario, where the CDN gets misconfigured). You also have an egress monitor for suspicious activity in case you aren't normally downloading all your data, and then you see a rise in egress https://docs.wasabi.com/docs/en/whats-new?highlight=egress#e....
In 2023 Veeam backup offloading to Wasabi S3, Wasabi "messed up their catalog" and lost data: https://forums.veeam.com/object-storage-as-backup-target-f52...
2023, files missing from bucket, Wasabi support not replying for days, then said they "had system maintenance": https://old.reddit.com/r/msp/comments/13dqhgr/wasabi_storage...
In 2021 Wasabi migrating databases, lost customer data: https://forums.veeam.com/object-storage-as-backup-target-f52...
I've been hit by something like that and had to re-push data to them.
Amazon S3 pricing looking more and more sane.
It does not tell me if I should count hours in hour-of-the-day, or lapsed time. Also the example tries to demonstrate a case of "you don´t have to pay extra", but then falls silent. Nice, not the info I am looking for.
What about envisioning a customer who asks «what am I going to pay? Specify it right now, right here».
30 * 24 * 0.0081 = 5.83
28 * 24 * 0.0081 = 5.44
https://getdeploying.com/reference/object-storage
Up to 10 TB per object
Up to 1 GBit/s bandwidth per Bucket
Up to 1024 operations/s per Bucket
Up to 100 TB per Bucket
Up to 100,000,000 objects per Bucket
Up to 100 S3 credentials across all projects
Up to 10 Buckets across all projects
* Couldn't tell you, as they flagged my account and asked me for an ID Document I couldn't provide, Maybe I could have sorted it with Customer Service, but instead I just picked a different german hosting
I had some servers that I had autopay setup for I thought. But the bill went over as unpaid and they shut off the servers. Then I noticed & tried to login to their web account system but my account was locked. Support wanted me to send a wire transfer. I begged and begged, told them how it would cost as much as by debt, and they opened my account for ~36 hours, but after some delay, and I was away vacationing. I sent them the wire transfer almost two months ago now & still nothing from their accounts/billing people, even though the transfer had the account & payment #'s. It's been so frustrating, & so unnecessarily over the top bad & they seem to love making it worse every step. What the hell Hetzner?
I started using their services since last year for some of the operations, including business, and it's been rock solid. I plan on increasing usage, but will hold on this new object storage offering.
Perhaps you heard less about them because they're based in Europe and thus less hype sorrounds them?
So far I'm happy with the service. Linode wasn't bad per se, Hetzner just offered more for less. Their admin console is a bit more spartan in comparison to Linode's, but that's not a place I spend much time anyway. If Russia glasses Finland and Germany I'd have to find another provider but I'm pretty sure I'd have bigger problems at that point than where to park my wordpress blahg and irc session, lol.
It seems the Hetzner's docs say they use MinIO to expose their S3 managed service. I guess everything will be the same as MinIO upstream features and contigurations
[0] https://docs.hetzner.com/storage/object-storage/overview#obj... [1] https://docs.ceph.com/en/latest/radosgw/s3/
> Our S3-compatible Object Storage provides you with storage capacity for saving data in "Buckets". Any data you save in your Bucket is saved in a Ceph cluster.
from https://docs.hetzner.com/storage/object-storage/overview#obj...
I'm speculating but those are at least the two I can think of that aren't explicitly linked to speed equivalency of a basic filesystem.
Admittedly I got my money's worth, for about 3 EUR for the very basic plan of 250 GB storage, I get upload speeds of about 6-9 MB/s and download speeds of about 25 MB/s, which is okay for what I'm trying to do. That said, it doesn't seem like there's a way to create additional users with access to only specific buckets not all of them for the same service and the overall offering does feel a bit jank (e.g. when you don't have an active service, you can't log into Contabo at all and while their Object Storage site is nice, the regular VPS one feels functional but dated).
What I'm saying is that when it comes to budget options, Hetzner will probably do great, since they have a good track record and it's not like there are that many other alternatives out there!
Of course, I could have also gone with just self-hosting with MinIO, Garage or SeaweedFS, as long as the VPS that I'd get would also have enough storage. It's nice that there are self-hosted options, too, I almost dread when I see bespoke object storage solutions at work, either developed before S3 was a thing or because the devs just didn't know or had a case of NIH, so I have to look at how a bunch of blobs are passed through servlets and serialized/deserialized, as well as deal with custom metadata and permission mechanisms.