That's great to hear, happy Hetzner customer for many years. Now if they start offering dedicated servers in US locations with equally competitive pricing as for their Germany and Finland DCs then a lot of other players are going to feel the heat.
Thanks for the feedback about our dedicated root servers. It's great to know that we have fans there that would like to also see these products in the States. Installing the infrastructure for our own dedicated root servers in the USA means a much bigger investment in terms of making sure the DC/DC parks are up to our very high standards. For now at least, we are concentrating on gradually adding a few more cloud locations, and then we may test the waters when it comes to other products. But of course, if and when we do that, we'll let you know once it goes live! We hope you have fun today generating a few servers in our new location in Hillsboro, Oregon. :) :) --Katie
Your dedicated servers are by far the most interesting product to me. The US server market needs more price competition, and Hetzner's dedicated servers would bring a lot of competition here if they were priced similarly to how they are priced in Europe.
I'm also waiting eagerly for Hetzner dedicated servers in the US ever since before your expansion plans initio the US were announced. There's just nothing like your EU dedicated offering in the states.
This is for sure. I currently rent dedicated from for my client OVH because of strict data location requirement. Exactly the same dedicated on Hetzner is 2.5 or so times cheaper. Either beats the crap out of AWS price wise. Unfortunately we are in Canada and what are the chances of Hetzner to come here ;(
We are, indeed, very careful about new customers and limits, but we do this not to be mean, but to prevent abuse. Preventing abuse also means improving the overall performance for all of our customers. So by being careful about limits, we are also trying to make your experience -- in the end -- a much more positive one. --Katie, Hetzner
I mean, is there a feature in the UI to charge the account by like 20$ for old customers? Last time I checked the FAQ, you can only do so by direct wire transfer of your bank not Hetzner UI.
Well, why not? With SEPA ICT wire transfers are instant and cheap, why build a custom web UI with fraud handling and all that if you can just use SEPA transfers?
I've seen a lot of customers frustrated with the limits and how reluctant you are to raise them, I assume because people weren't paying their invoices.
Perhaps it could be an option to let people back up their words with money up front? I.e. put $10k in the account, proves you can probably afford more than 10 servers.
Given that they mention abuse elsewhere in this thread, I assume the problem is related to stolen cards and thus can't really be fixed by accepting money upfront.
It's slightly more work, in my experience, to set up and administrate a Hetzner VPS, than to do the same with DO or AWS. Slightly. Trivially slightly. I use Hetzner and I have been nothing but happy with their product and service.
Yeah. For me, the only thing I miss when using Hetzner is the lack of a good startup script. Hetzner has something like an Ansible YML file you can use, but then if you want to reimage your machine, it doesn't give you any choice (that I've found) except to reuse the original script.
What I'd like to have is something like Linode's start scripts feature which I've always found to be very nicely implemented.
Why have startup scripts when you can run your very own images? `Installimage` (https://docs.hetzner.com/robot/dedicated-server/operating-sy...) allows you to install and run your own custom images, so run any bootstrapping once, create image from the results and then install that on the machines.
If you really really need startup scripts, it'll be trivial to add it to your own image. Packer (and similar tooling) makes it trivial to create your own images as well.
Hetzner Cloud also has an extensive ISO library, and if there is something that is missing from that library that you would like to have -- including a specific custom image -- customers can write a support ticket on Cloud Console and our team can upload it for you. --Katie
It's just a minor gripe, but... a simple bash script (like Linode's start scripts) is a lot easier than what Hetzner offers-- a custom ISO? Nah. I'll just scp my script when the server is ready and run it myself.
Their cloud offering isn’t as polished yet. No managed databases for example.
Other than that they’ve got a long track record and a good reputation for providing competitively priced servers.
In my experience there's no real "catch". Just trade-offs.
Hetzner offers "managed bare metal", for extremely competitive prices. But, in my experience, exactly what I'd expects: "pay peanuts, get peanuts". I love, and chose them for some proof of concepts, early phase and hobby projects. But would not choose them for anything that requires serious stability and availability.
Not because they have flakey or even unpredictable service, but because the trade-off is that "bare metal" requires more work done by me, more responsibility for me, less options for quick failover and so on.
Just like for some situations, a server in your attic is the perfect fit, yet for others a managed cloud infra is the perfect fit, hetzner has some sweet spots.
Dedicated servers there were an option way before DigitalOcean, if I remember correctly (I recall at one point having some Linode and some Hetzner servers, DigitalOcean was not in the picture).
Nobody is getting "trickle down"-level rich from it. It's an organically grown company that bootstrapped without any big high risk/high reward investment that expects massive returns in absence of failure. Basically a mom&pop from what in the US would be considered flyover that just happens to have found its way to large scale competetiveness. Through a unique combination of frugality and decisive spending I think.
Early example: a weirdly memorable ad captaign around 2000 that for many years occupied the single most expensive computer related print ad slot in Germany (decisive spending) with a series of ads that seemed not quite "high production values", but also not deliberately grungy, a weird "definitely trying to be high gloss perfection, but somehow not quite there" (like in-house best effort or some local design house, certainly not the big-name agency you'd expect for ads on that slot). More recent example: their hardware seems to be a continuation from early-Google style "desktops on shelves" that's now a custom rack design (still noticeably lower density than typical 16") that's all about finding good price/reliability spots in cheap CotS parts, e.g. according to certain "begins the scene" blogger visits they sort for publicity, price-optimized custom versions of desktop mainboards (same PCB but not placing any parts they don't need). Chances are company with big investor backing would either go all standard rack parts (from a supplier like Dell or something like that) or go all in designing their own.
Very happy about this as a long-time Hetzner dedicated and cloud customer. The only "issue" I've had with cloud is the lack of video memory due to the virtualized environment they run in, but that's a special case because I'm running a custom Windows server install due to Adobe products, which is not officially supported. I await the day they add object storage to the mix, then my (hosting) life will be complete.
Object storage is, indeed, very high up on the list of hoped-for items on our customer wish list for cloud products. I can add a +1 to that for you and send it onto the dev team. As always, we don't announce roadmaps of upcoming features and when they will be ready. We prefer to announce things when they go live -- just like today! --Katie
You've been saying this for several years, and yet there is still no object storage. It's getting to the point where it seems more realistic to believe that it will never happen.
Very happy with both the cloud and dedicated servers, but it's weird how bad the communication is on the object storage thing. If it's not going to happen, why not say so?
I wouldn't hold your breath for this. I think Hetzner wants to stick to compute offerings, they don't have the manpower to run managed services like object storage or databases.
I'm personally fine with this, if it allows them to continue offering such solid value.
Wasabi is a great, inexpensive object store for anyone who's looking for that. I've had a really good experience with them (we have a little over 80TB of data there), and have found their support to be top-notch especially when you consider the price.
+1 to Wasabi. They are the only reason I was able to launch my product which wouldn't have been feasible on S3 or other providers just based on the storage and the API cost.
I mentioned it yesterday, but I will mention it again... take a look at Storj. [0].
Storj pricing is basically unbeatable. $4/TB/month, $7/TB/month for egress bandwidth which (I heard, yet to try) can be saved further if you put something like cloudflare in front of your bucket.
Speaking as someone who set up their own minio cluster (on Hetzner) as a way to have object storage at the lowest cost possible, if Storj was an option 3 years ago, I would have saved me quite a bit of money and time.
Wow! Twenty years in this industry is a lifetime! Thanks so much for supporting us all these years! I will pass on a +1 for you to the dev team about the managed databases. --Katie
I currently have hybrid DigitalOcean / Hetzner setup to take advantge of load balancer and managed database at DO. If Hetzner provided some of those, I'd gladly switch.
I am not managing replicated postgresql myself ever again.
We considered DO but found that it was cheeper to hire a sysops firm (Linpro) to manage the DBs on Hetzner. This has the added bonus of humans that you can call if needed...
I absolutely love using Hetzner. Top of my wishlist is Managed Databases and Managed Kubernetes. This is the main of the reason why I'm using Digitalocean for production and Hetzner for more stateless use cases.
Hetzner's performance is arguably better (Epyc 2nd gen on Hetzner vs Epyc 1st gen on Scaleway), but their support for IPv6 is certainly much better: Scaleway doesn't offer PTR record and changes the IPv6 assigned to you when they relocate your VPS in the datacenter.
Nice to mee another 20 year customer. When I first discovered Hetzner, I couldn't believe their dedicated physical servers were cheaper than the colocation costs I was paying in a Belgian datacentre (without the hardware!). Over the years, I've had some hardware failures, which have always been resolved very quickly. Every time I interact with their support crew, I'm pleasantly surprised by how knowledgeable they are.
+1 for managed Cloud DBs. I'm surprised they don't exist already, given that they have managed DBs on their cheap web hosting platform. Shouldn't be a big step to make that available in Cloud.
Another request I have is virtual routers, so I only need 1 dedicated IP address and can NAT (or whatever) everything else. I get that I can do this with a small Cloud Instance and a private network, but those things are a pain to manage and I'm sure Hetzner could do a better job than myself :-)
Since everyone is all of a sudden voting for managed databases, maybe there is a reason why Hetzner is so cheap and the reason is that they focus on the core and not on satisfying all kinds of feature requests?
It's not "all kinds of feature requests", if you think about it.
IT is basically 2 things, at its core:
* compute
* storage
Hetzner offers cheap compute, which is great because modern applications can have stateless web/app servers.
Now, the missing part is storage, and that's much harder and riskier since it's state, inherently. A server dies, you lose stuff if you misconfigure it.
So people want a fully managed IT solution.
It's simple and it's obvious conceptually, just hard.
And I guess Hetzner won't do it because it's hard thus expensive to do.
There already are full managed solutions. They’re 5x more expensive. It sounds to me like people expect the whole managed experience but 5x cheaper, which is never going to happen.
I'm working on this -- it will be a while till I get on Postgres (I really hope that's the managed DB you want), but I'd love to get you on the postgres beta list.
Postgres (and later some exciting flavors like Neon) is actually the only database I plan to support, at least for right now -- MySQL and MongoDB are not even on the roadmap really.
I might make an exception for RethinkDB since I loved it so dearly.
Don't know who is behind it -- a company named MostlyTyped (not sure if this has old RDB engineers in it), but warms my heart that the repo is still seeing commits and usage.
I want hetzner to concentrate on the core capabilities: servers, storage.
I want Hetzner to provide a PLATFORM to many different database-as-a-service on top of Hetzner. I want choice, and I don't want Hetzner doing what AWS does: stifling competition by releaseing their own flavor-of-open-source-database that puts the core projects sponsoring the open source code out of business.
I hope they created an offspring company with no access to the Hetzner main infrastructure. It would be very inconvenient to lose the "hosted in Germany" - a.k.a. "not accessible data for the US" - aspect in terms of GDPR.
To the best of my knowledge, this is already too late. I remember reading through their FAQ and seeing that hence they now have a US-based owner, they are then within reach of US 3-letter agencies.
That seems to be correct. My understanding (IANAL) of Schrems II is that the problem exists when a EU datacenter is under the direct or indirect control of a US company. Indirect in this case meaning operated by a EU company that is the subsidiary of a US company, as is the case with AWS, Google and Microsoft.
Since the EU datacenters seem to be operated by EU companies and the US company is merely a sibling subsidiary of Ensoxx, which itself is also an EU company, this should provide sufficient isolation to prevent interference from US agencies short of direct sabotage or espionage (since the EU staff is not in the chain of command of the US company).
So for a definitive answer you probably want your lawyers to talk to Hetzner's lawyers but at face value this is at least miles ahead of any US-based cloud provider, which in all honesty is still the default solution for most EU-based companies despite this ruling.
In summary, you as a customer do have influence - to a certain extent - on shaping who has access to the data on your servers. EU and US authorities do have to follow the laws and legal procedures in requesting data. However, this may give you a false sense of security since some authorities have been known to stretch or violate agreements. If you require a web hosting company that has absolutely no connections to the USA, then unfortunately, we may no longer be the best choice for you. Since Hetzner US LLC is part of the Hetzner Group, there certainly is a connection. We hope that we have explained things clearly from our point of view using the two above case studies.
Ok, but: "US authorities do not have direct access to your server or its content in the EU. US authorities have to comply with the regulations of the EU legislation.".
So, because Hetzner is not owned by a US company, stuff like the CLOUD act doesn't apply to them. So, if you have a contract with the German entity of Hetzner and use a German server, you should be fine in terms of GDPR.
I think it depends on how you read the Schrems II ruling and how you read Hetzners words.
Any of the big cloud providers can claim that they comply with EU legislation, but they also have to comply with US-legislation and if 3-letter agency wants to have some data from one of their subsidiaries in EU, then they can/will decide which contract to breach.
I read Hetzners statements as being that they can no longer guarantee that they will not be forced to do the same - but that can be my reading of their statement that is wrong.
If I already had them as hosting-partner for a solution that fell under Schrems II, I would have them confirm this, to be sure.
But what does "direct" mean here? Indirect could still be ordering them to give US authorities data and to keep silent about being ordered. Maybe (hopefully) that would be against EU regulations?
Lots of EU countries have their intelligence agencies doing close cooperation with five eyes (NSA and equivalent agencies of the smaller countries) and willing to turn a blind eye or actively collude in compromising security of IT infra in the EU. Or going further, a oft reported pattern is that when they want to spy on their own citizens but are forbidden by law, they ask the foreign allies to do the dirty work of spying on their soil and pass back the intelligence.
OK, be that as it may, in IT stuff, the question often becomes "Who is responsible?". If a state or its institutions violate the law, at least no one can blame you for GDPR violations, which you did not commit.
The GDPR largely came about as a response to the Snowden revelations of pervasive surveillance of netizens globally, and it says you need to protect PI from non-EU state actors. So you're possibly right as far as EU state adversaries go but you for defending against foreign state actors it's different.
I read it differently, especially in light of Schrems II. EU-datacenters from any of the big US-based providers does not automatically make you comply either.
Imho, as soon as you do business with the US or trade in US Dollars, you need to play nice with the relevant authorities.
If I understood it correctly, Hetzner is now "infected" in the same way as the three US cloud providers are. The Schrems II verdict and Cloud ACT basically concludes that no European company can exist in the US and vice versa without having to deal with the same pesky legislation.
An alternative could of course be that Hetzner created a new US based company where the EU parent Hetzner company only holds a minority ownership in the new US-based company. The EU based parent company in turn then "sells" its technology to the new US company. This way, the arrangement becomes more reminiscent of how IBM has sold its mainframe to European companies...
Why would it matter at all if it's a minority or majority stake in the ownership of the US subsidiary?
As far as I understood it the combination of GDPR and CLOUD act only disallows the combination of US mother-company with EU subsidiary, but the inverse should be fine, since the US has no legal influence over the parent company?
The US-based cloud providers also have European subsidiaries. But that doesn't help because they are bound by US law. That is the root of the problem.
What makes you think that a European company operating within US jurisdiction would not be subject to the same laws?
If the European company receives a request from the US authorities for information, they need to follow the same legislation as the US companies do. Just because it's a subsidiary won't help. The authority will say "we want to know everything you know about the following person, please give us the information, otherwise...". The authority will not distinguish whether it is a subsidiary or the parent company.
Of course have the choice to just ignore the request from US authorities, but then you have to be aware of the consequences, i.e. quickly give up and shut down the subsidiary and stop trading with US dollars.
This is the root of the problem. CLOUD act has been ruled illegal in the EU just as you said, but it is also illegal not to comply with CLOUD act in the US. And companies operating on both continents in practice need to comply with both laws, regardless of whether it is a parent company or a subsidiary.
Are there any cases of the US nationalizing/seizing companies outside of sanction/war-related acts?
Which would probably the only consequence the government can directly levy against the European company (the indirect ones they can also apply to pure European companies, so they don't really matter for this discussion).
But it really depends on how infectious just owning a company is, which I have no idea. But my gut-feeling is that it shouldn't be too infectious, since otherwise just buying a single share of a company operating in another country would put you into legal peril (who controls the subsidiary here is not really relevant, since the Cloud act wants to swim in the opposite direction in your scenario, therefore it shouldn't matter if it's 0.01% or 100%).
Hetzner Europe is owned by Hetzner Group, a German company. Hetzner US is also owned by that German company. Hetzner Europe isn't owned by a US company, it's just a sibling to one.
Hetzner Online GmbH and Hetzner Cloud GmbH are fully owned (resp. majority-owned) by ENSoXX Holding AG of which Martin Hetzner is the CEO and which seems to be the parent company[0]. I cannot find any indication in the publicly available documents that any US company (or any other company for that matter) holds more than a 25% share in ENSoXX Holding AG. (25% is the reporting threshold.)
In the 2022 annual financial statement they do mention expanding to the US, though they don't go into the legal details. As the link posted in the cousin comment mentions, though:
> Hetzner US LLC, as a subsidiary of Hetzner Online GmbH, provides data center services within the USA for the parent company, Hetzner Online.[2]
Hetzner is all about doing things efficiently, in unconventional ways.
There are some Youtube videos of people getting tours of their data centers. It's a very custom setup, keeping both component costs and energy costs down. Their earliest servers were basically tower PCs on shelves, their more recent generations are more akin to custom rack designs with inhouse-assembled servers, with a datacenter design that exploits natural convection to do a lot of the cooling.
Finding good people is difficult, especially back in 2021 when everyone was scaling IT like crazy. A good bonus is a good investment in employee retention. On top of that having employees participate in business success aligns incentives. It's just good business sense.
Non of them deliver equivalence of big cloud providers.
Also, track record of at least some of them is not the best, if we are to believe, for example, comments on HN. I've made initial research about most of them. Just to give some examples about Scaleaway: prices moving up by 75% without notice (https://news.ycombinator.com/item?id=25999148), zero notice of the removal of the ability to start ARM64 instances (https://news.ycombinator.com/item?id=22923413).
As a counterpoint, i quite like Scaleway. I've been using them for a few years for a personal Kubernetes cluster, with some random experiments with their serverless and managed database offerings, and i didn't have any issues (for my minimal workloads). They've changed pricing quite a few of times which is far from ideal, but I accept that situations evolve (e.g. the latest one is due to the massive increase in electricity costs in France). Even with all the changes they're still cheaper than many of the alternatives, so it's still "a win".
This is becoming ridiculous. Sure, I may deploy to EU based provider, but then they will get bought or, like Hetzner, expand to USA, and now my business is at a legal risk all of a sudden.
I am running a cluster of machines at Hetzner for years. Generally i am satisfied, but i like to share some bad experiences which took a month to fix.
At some point some of my servers were facing random cpu stalls mostly at occuring midnight. I spend hours and nights to find out the cause. But i wasnt able to find it. Tracing the issue resulting in different causes everytime.
After I contacted the Hetzner support team they moved some of the servers to a different host system. Apparently there was a resource issue in their virtualization layer. It fixed all the issues for a week, but then it started again. I contacted support again and received an arrogant email that the issue is related to my software and that they couldnt help me further. I was perplexed. I solved the issue by creating servers in a different zone. Exact copies of the so called faulty ones.
The whole stack is now running without issues for about 2 months. But still i am a bit worried.
Perhaps you could give me a ticket number (or the latest ticket number) about this issue, and I can ask a team member to review it for you. Or, you can send the ticket number directly to marketing@hetzner.com with a link to this page, and I can do the same thing. --Katie
High visibility customer support via hacker news is not a feature, it‘s a failure.
And those failed communications between provider and customer are why everyone is frustrated with the big corpos (google, cloudflare, stripe, etc.) which Hetzner is apparently bound to become.
Support is a very hard issue to solve when you have volume. It doesn't help that 99.9% companies regard support as a cost center so salaries are low and there's a lot of churn and disregard for supportpeople tools.
At least you have someone speaking for Hetzner officially here. Contrast that with Google, where Matt Cutts was kind of used as unofficial backchannel for customer complaints, up until he left Google, and now all we have are prayers that some unknown reader will be moved by our plights.
This is not to excuse bad support experiences. I just think we should appreciate it when there is still a channel to a real human being willing to be helpful and engage constructively, because it's getting increasingly rare these days.
It's not the first time I read a feedback about unprofessional customer support from Hetzner. Here or on Reddit. I admit that nearly every time someone from Hetzner shows up, says it might me a language/cultural issue or something of this kind and offer direct help to look into it.
Hetzner is on my radar for a long time as an alternative to simpler deployments on AWS. It's great they are monitoring internet forums and try to be helpful, but also very frustrating to read such problems apparently still occur.
Being the lucky winner of ones complaint being found by personal, C-suite, marketing (Hetzner) or top-engineer (Stripe) accounts on online forums and getting one time only high priority support is not a fix for the systemic failures that led to the complaint.
We recently had a mission- and time-critical issue with play store and our first reaction was to contact our gcp account manager and old google friends because we knew play store communication sucks. It should not be this way.
Something similar happened to me. A server I have was very frequently randomly restarting, and I tried everything to solve it, i.e factory reset, reinstalling my services. I contacted customer support and they offered me to take it offline and perform some tests on it that could take more than 10 hours. After I insisted a lot, they offered me a one time server replacement, which magically fixed the restarting issue.
Thanks for writing about how our team helped you resolve the issue. Have you been happy with the rest of your experiences using the cloud products? --Katie
You thinking this was a positive experience only makes me more concerned about your customer support.
Let's hope I don't need another "one-time" server replacement because of your faulty servers, because I already used mine. 10 hours of downtime with angry customers on my tail sounds fun.
> I contacted support again and received an arrogant email that the issue is related to my software and that they couldnt help me further.
Doesn't surprise me, German companies always know better and will try to prove you wrong, whether you're their customer or supplier. Such is the business culture, I think.
As someone who has customer service experience in both the USA and Germany, I can tell you that there are, indeed, cultural differences. There are certain situations in the USA where Germans and other Europeans don't always have the best experience because of a cultural misunderstanding.
I am also the in-house English teacher here at Hetzner. (Most companies have no in-house teacher.) Something that I personally work on with my students in my conversations class and customer service class is intonation, which can cause spoken language to come off as sounding "arrogant" by accident. In addition, written responses may accidentally come of as sounding too direct or "arrogant" for the same reason. We work on these situations in my classes.
So if you ever have a ticket that you think might make for good learning material for one of my classes, or that you would like to see escalated because of a serious language/cultural misunderstanding, please write to marketing@hetzner.com and mention my name (and include the relevant ticket number). --Katie
While it might be a cultural issue, I have to admit that this isn't the case with many Germans. I interact daily in open-source projects I'm involved with. While they are direct most of them seem very respectful and welcoming. From time to time some people might seem a bit more arrogant but I think that there might be a cultural bias too at play here.
Kudos for assisting your colleagues with better communication in English, I think this practice could be useful for many organizations with people that speak English as a foreign language.
I think its an issue of inter cultural communication.
My impression is that in Germany you bear the burden of proof that you actually have a problem, that its not your own fault, and that you did your part trying to fix it.
You usually can get good support in Germany ( even at government offices ) , IF you show up with your Leitordner ( legendary German ring binders) with all the receipts, all the possibly relevant account numbers, transaction ids and a detailed analysis of your own problem.
There is nothing that signals to German support staff that you have to be taken seriously like a ring binder, preferably with color coded markers at the margins and lots of punch pockets.
If you think this is satire, try it the next time you have an in person appointment...
> Doesn't surprise me, German companies always know better and will try to prove you wrong
Yep. Its a plight that afflicts almost all engineers around the world, and German companies are even more afflicted by it due to the German engineering culture...
We had a similar problem at Hetzner, we were using their consumer grade systems and this was over 6 years ago. To Hetzner's credit though, after we decided it had to be a hardware problem (dmesg would log something about cpu states before every crash) we sent in a ticket and within a reasonable timeframe (I think within a day or a couple, it was long ago so don't know for sure) a Hetzner sysadmin went into the BIOS and changed some feature (I suppose he disabled some sleep mode) and the systems ran perfect after that.
I chalked it up to our decision to run our databases on consumer grade hardware and didn't give it a second thought.
I had a weird performance issues on Hetzner virtual machines where network bandwidth would drop to 50mbit/s like it was being throttled. My application had a tendency to use very little bandwidth most of the time but when updates were available we needed 10-40GB in a day or so and I always suspected we were getting throttled but support never admitted it. On average we were below their resource limits but clearly at times we were above the average. Support was pretty dismissive and arrogant and I don't regret moving off to netcup who in comparison the experience has been flawless.
> I contacted support again and received an arrogant email
Yep. The attitude of the engineering support is something that Hetzner needs to improve a LOT on. They tend to treat customers as if they are treating members of their own open source or engineering community - scolding the customers when they think the customer (community member) is being unreasonable etc.
Its a great thing that eng. support is just a ticket and 20-45 minutes away at Hetzner. But the attitude needs a lot of improvement. Imagine that you are a startup facing some quirky issue that affects your business in the middle of the night and having to deal with attitude and scolds from datacenter engineers...
@ducktective -- You asked about being able to charge your account €20 or soomething similar. Yes, this is possible with bank transfer. You can find our bank details at the bottom of every invoice. Please enter your invoice or client number as a reference to your bank transfer. You can add more than €20 if you would prefer. And then later, if you decide to close your account, we will transfer the money back to you within 14 business days.
You can find the answer this this question and others here: https://docs.hetzner.com/accounts-panel/accounts/payment-faq...
--Katie
We're a long time Hetzner dedicated server customer currently transitioning from their dedicated servers in Germany to their US Cloud product (for reduced latency) which we can highly recommend as it was the best US Cloud provider we've found that works out to be an order of magnitude less expensive than equivalent specs on Azure/AWS and also includes 20TB free bandwidth that would cost a fortune in AWS/Azure's artificially inflated egress costs [1].
The UX behind managing instances is delightfully pleasant where new instances are available faster than any other cloud provider we've used, within seconds of creating an instance you can immediately login with your configured SSH keys. Another nice feature is being able to "rescale" your instance to higher specs after a restart [2], so you can confidentially start with a small instance that just meets your current workload knowing that you can easily scale up your instances as your workload increases.
AWS RDS was the only critical service keeping us on AWS, a service we no longer need in our new Apps which we're building with SQLite thanks to the effortless replication in Litestream [3] that we're using to replicate to Cloudflare R2 - another great value S3 alternative with $0 egress fees [4] where you can get even greater value & performance when hosting behind their free CDN.
Thanks so much for the detailed recommendation! We're thrilled that you're with us, that you appreciate our low prices, that you find our Cloud Console pleasant to use, and that our rescale feature helps you grow so easily! --Katie
Please do a managed Kubernetes next. I couldn't convince any of our customers to switch to Hetzner because they'd need to do "everything themselves". A managed Kubernetes instance would instantly make Hetzner an alternative for at least 75% of our customers. And honestly it's quite a cheap way to earn a bonus on your server instances.
Edit: And if you do manged Kubernetes and managed Kafka Instances the number would go up to like 95%. Oh and those Videos with der8auer? Really awesome to see, do Linus Tech Tipps or Level1Techs next!
Kubeone on Hetzner is honestly one of the smoothest non-managed experiences I've ever had with Kubernetes - better than some managed ones. Includes all of the things Hetzner already provides for Kubernetes (so I'm sure they're working on their own) - internal networking support, load balancers, volumes. And you scale worker nodes with a CRD too.
That's wicked cool, thank you. For me and a little more experimental customers, this is a nice option but for the more legacy customers out there, if it doesn't have an SLA, it doesn't exist. I know it doesn't make sense but it's how big companies tend to work.
Hetzner S3 Storage would be great. I know Hetzner is slow moving when it comes to Software but it's so obvious that they would have explosive growth if they did that and most of it is available as FOSS Software already
So I'm working on adding this on top of Hetzner -- it's service #2 and the beta should be out by the end of the month (Redis is first!) -- if you'd be willing to beta test I'd love to have you try it.
IMHO this has to be offered by the platform, not by some other party/intermediary. Of course I could run minio, swift, etc by myself, but this is not the point.
Thanks for sharing this, hopefully after the expansion finishes (and maybe some dedicated servers appear in the US), Hetzner could offer object storage natively!
I have seen the "apps" in image selection in the Hetzner cloud console, so I'm convinced they're at least thinking about going up the value chain.
There are some other pieces missing.
The cluster-autoscaler implementation is quite basic. It doesn't have a way to managing multiple clusters inside the same project and no way to add custom tags. That makes it harder to track resources.
The permissions/security is quite simple. Read/write tokens with full permissions on the project is very wide. Adding S3 like service without more security would be interesting.
These are the most important ones, but I remember there were a few more nice to have feature (like server groups for scaling up an down and the ability to retrieve the userdata, image and ssh keys for a running server.
If we're taking requests, I would love a Fargate-like service. It would be absolutely fantastic if Hetzner provided a service where users uploaded a Docker image, and Hetzner took care of handling the networking bits and running it, and gather metrics.
Something like this is on my roadmap, would you mind telling me a bit about the metrics and the scale you'd expect? Would you expect always on or more of an ephemeral container?
Combining what the original comment was, they also want some AppRunner-style ergonomics -- I'd like to see just how much of cloudwatch/monitoring would be expected to be available. Basic things like CPU and memory aren't too hard but it really does depend on how much of the VM (firecracker/etc) one would expect to be able to see, as well as higher level metrics like (RPS/Errors, etc).
AppRunner, Fargate and Cloud Run have different ergonomics and specifics but thanks for this outline. v1 is likely to be pay per container but other than that feels doable.
> would you mind telling me a bit about the metrics and the scale you'd expect?
It will depend mostly on what the service offers.
If the service only supports running a single isolated container without any scaling whatsoever then it would be helpful if we could monitor básico stuff like CPU and memory utilization, and also network traffic, free disk space, and also disk IO. If the service supports auto-scaling then it would be helpful to track all resource utilization rates along with all alarms and events involved. Auto-scaling also implies load balancing thus if that's the case then it would also be helpful to track the basic load balancing indicators, as well as request logs.
In the end it really depends on what services you're planning on offering, and how you'll charge for it. As a user I would need to monitor any metric which is directly and indirectly involved in determining cost, and on top of that I need to monitor performance.
> Would you expect always on or more of an ephemeral container?
The most pressing need would be always on containers to be able to go the lift-and-shift onboarding route to managed services, but ephemeral containers sound like function-as-a-service and those are pretty exciting as well.
They could do it like GCP or Scaleway - provide a managed Kubernetes service, and provide managed Knative on top for the Container as a Service service.
For Kubernetes, maybe give kOps a try. It's quite close to managed. The Hetzner integration is quite good (I spent about 6 months getting it ready for my company). The community support is great also. It still needs autoscaling added though.
https://symbiosis.host uses Hetzner for their underlying nodes as I understand. It certainly explains them being the cheapest managed Kubernetes offering I've ever seen.
Whenever I hear Aiven, I think of an incident I witnessed 2nd hand in January 2020. They accidentally terminated the services for (at least) one big customer and had to restore them from backups. This lead to data loss in production (Kafka topic data and configuration gone) and was a huge mess for the customer to clean up. Of course they abandoned Aiven after that.
You might be interested by Elestio, we support 13 managed DB and 170 other open-source software. We also support hetzner including this new Hillsboro region.
> Cloudflare R2 - another great value S3 alternative with $0 egress fees
I don't understand their price page. They claim $0 egress fees, but their free "Class A operations (mutate state)" and "Class B operations (read state)" have a mothly cap. After that you pay by the number. Isn't that an egress free?
One time I hosted a new movie (mkv, 23gb+), that cost like $17 to “rent” on Amazon prime… so my extended family could just take the link, download it, or just watch it in their mobile browser or laptop.
The egress of the streaming alone (not downloading), while they aircasted, definitely adds up much more than one would expect!
No, because you pay by the operation, but by the amount of data in that operation. S3 also has operations costs, but those are separate from the egress costs, which are throb the roof.
Does anyone know what the actual margin on outbound data / egress is?
S3 is charging $0.023 per GB, right?
IIUC - there's not a fixed cost for sending 1 GB of data. The cost differs based on where you send from and send to. So it'd be hard to have a really good estimate - but I'm wondering if anyone has a good ballpark figure.
My understanding is that transferring data on the same continent (vast majority of traffic) should be <$0.0005 - meaning the margin is really high.
I know it's more in Amazon's interest for your cloud usage to be as inefficient as possible - so they can charge you as much as possible and get as much margin as possible.
However, this product doesn't seem to make sense to me.
Why would you even pick your S3 regions? Shouldn't AWS balance your data for you across continents so that your data egress is automatically optimized?
To put numbers to the relativity, for a somewhat smaller datacenter in Central Europe with a 100Gbit/s connection, the running Cost of transit is somewhere around 0.00004€ per GB, on a theoretically fully sustained connection(not realistic, also with no redundancy or hardware).
Peering is basically free(after initial buildout) with around 3k€ per 100Gbit/s (https://www.ams-ix.net/ams/pricing). This all excludes cost of investment for initial buildout and hardware.
On a scale on amazons level this becomes even cheaper, since operating your own network is cheaper than buying transit.
Hetzner charges 1,19€ per TB, amazon 90$(on the first 10 TB). So amazon probably doesnt really care about the price differences per region and went with a mixed calculation, since their margins are so absurdly huge.
also, major parts of the cost are in hardware and staffing/development, of which amazon has way more, due to the demands of SDN.
First thing on signup: YOU HAVE TO DO ADDITIONAL IDENTITY VALIDATION, yes, even though we took your home address and credit card and phone number already.
No thanks, I'll keep using Digital Ocean or someone who doesn't make me jump through hoops.
Interestingly, I've never used Digital Ocean because when I went to sign up (in my memory it would be circa 2015, but my memory is not what it once was) you had to give them something like your Twitter or Github name, which I didn't feel like sharing.
If your fraud detection cant do anything with name, address, phone number, and credit card and you need Drivers License/Passports then you probably need to pay for a better one.
What do you expect a German company to do with a copy of your drivers license or passport? Is this one of those situations where it contains your your social security number and it therefore becomes direct access to your bank account?
The only thing I imagine someone could do with the copy of my passport is pretend to be me when they sign up for a similar service.
eBay did this to me. They let me enter everything and then said they can't validate my ID without any way to fix that even. It's a nice way to provoke people for sure.
Thanks for the notification. I came to ask if they were still doing this. I found this to be an exceptionally shady practice. They took my information first, then they wanted some ridiculously personal information that they didn't need. I assumed at first that I had gone to the wrong URL. I don't even know how that additional information would have helped them with their "identification". The only thing it did was expose the information of legitimate users to being stolen.
Your address and phone point to a location, not necessarily a valid identity but I understand you may be hesitant to share personal info with just anyone. If you are a EU citizen you can ask them exactly how long your data is retained and who may have access to it (it’s actually published on their website). They use it only as a 2nd level verification to prevent spam (not shared with 3rd parties). As a German/EU company, Hetzner is subject to all regulatory requirements for handling of personal information and so when leaving, you can also request the deletion of your data.
This is a problem with Hetzner yes. Last time they wanted a copy of my ID. I blacked out my social security number as this is considered private for Dutch citizens (even the police advises people to black this part out) and it took some arguing for them to accept it. I sent them the police advisory and that helped.
Recently I signed up for something else from Hetzner (needed temporary storage) and they didn't request anything even though I had closed my account before so I created a new one. So perhaps they have mended their ways.
Mind that sending it does not even guarantee that you'll pass that validation, I sent them my ID and they still locked me out for apparently no reason.
I'll add a +1 to managed database product. We've migrated all workloads except main application servers to Hetzner. Having a managed DB service (with backups, point-in-time recovery, etc) would have us quitting AWS in an instant.
I emphasize that I personally did not run any such tests, yet. But was wondering, since this is Hetzner thread, that maybe someone can share their experience, in particular comparing AWS's gp2/gp3 based deployments vs Hetzner's volumes.
I would also add that on AWS, for example, you can nearly seamlessly expand EBS volumes in size without downtime. Last time I checked, not an option on Hetzner - you must take care of expanding the file system yourself. Which makes sense, as Hetzner is more basic service, but it's worth remembering that there are various differences like that, when comparing the day to day operations between such providers.
Network volumes always have lower iops than local disks. Thats expected, except when you pay a huge price.
> I emphasize that I personally did not run any such tests, yet. But was wondering, since this is Hetzner thread, that maybe someone can share their experience, in particular comparing AWS's gp2/gp3 based deployments vs Hetzner's volumes.
AWS GP3 has a baseline performance of 3000 iops, and from there on you have to pay for each iop/s. To reach the performance of hetzner volumes you have to add 20$/month. But with AWS you can get higher than the hetzner value. But the iops performance of the dedicated machine is not possible with gp3 (max. 16k, compared to hetzner 30-40k for local disks).
With io2 devices you can get higher than 16k iops, and higher than the hetzner local disk.
Hetzner Cloud is great, but I do wish they'd offer dedicated servers in the US as well, even if it was just a fraction of their European offering.
Hetzner Cloud pricing is great, but their dedicated servers are even cheaper. For example, they offer a 16-core (32-vCPU) Ryzen 9 5950X for €103 with 128GB RAM and 2 x 3.84 TB NVMe SSDs. They offer a cloud server with 32 vCPU, 128GB RAM, and 600GB of storage for €296 - nearly triple the price and those CPU cores are probably not as good since they're likely Zen2 cores rather than the Zen3 cores of the Ryzen 9 5950X.
I know, benchmarks aren't everything, but the Ryzen is getting 1,432 per vCPU while the EPYC is getting 766.
The AX51 costs €59 for 8 Zen2 cores (16 vCPU) and 64GB RAM while the CCX41 costs €154 for 16 vCPU and 64GB RAM.
I know, the cloud servers come with flexibility, hourly billing, and no set-up fees. I also know that their cloud pricing is very good. Still, I wish I could get a few AX101s in the US. 3 AX101 servers would be €310/mo, each with 16 Zen3 cores, 128GB RAM, and 2x 3.8TB of storage.
iirc Hetzner once had GPU offerings in Germany but it was discontinued. I guess it was hard to model a business case on short living hardware (from a business perspective as Hetzner hardware usually runs for a couple of years) and mad pricing/sourcing due to all the blockchain/ML hype. Today I guess eneregy is also an issue.
Exactly. Because it seems transparent. It would just be better if the customer service given through the company's support channels were adequate. Not these veiled attempts to market via customer support.
When I was moving my server to a new hoster Hetzner was on the top of the list, but they refused me as a customer.
As it was going to host (among other things) my mail server, I could not use my primary (self-hosted) email address - because that's asking for trouble if I ever run into issues. It seems using Protonmail triggered something on their side.
They asked me for a copy of my ID card, which I happily provided. But they still refused my account, without explanation. Oh well, their loss. I've been at OVH now for a few years without any issues. OVH's product is definitely worse, but at least they'll actually let you use them.
hi there, I am very sorry that we could not approve your account. We also do not publish a list of things that may or may not make your account accidentally appear fake. So I can't confirm whether or not it was the Protonmail or something else that may have triggered a review of your account. I am very sorry for the lack of transparency on this. I understand that it is frustrating and disappointing to be rejected as a customer. We are purposefully non-transparent about what triggers a review of new accounts. Why? If we published a list like this, it would very quickly become much easier for scammers and spammers to create realistic-looking accounts that they could use to abuse our products, and naturally, we don't want that. I glad that you have found another provider who could use to host your mail server. --Katie
I’m not sure if reading this response made me feel better or worse.
Clearly if you are rejecting legitimate customers for arbitrary reasons there’s still some work to do on your approval process.
I mean, I get that there are bad actors, and that it’s rationally better for Hetzner to have false positives than false negatives. But it just feels wrong. Legitimate customers shouldn’t be rejected until they prove they’re malicious.
This level of account paranoia (which is infamous at this point; it's one of the primary things mentioned every time Hetzner comes up) is one of several reasons why Hetzner is doomed to be a second-rate provider. If you have a credit card, you can get an AWS account. I've never provided any kind of ID to any American hosting provider. They wait until after you've done something bad to ban you.
It's more expensive to provide AWS-type "assume you can be trusted" service at a low-cost provider - and Hetzner is one of the lowest-cost big providers.
Low-cost attracts more fraudulent customers, using stolen credit card numbers. Hosting is particularly bad for this compared with other low-cost services, because of the community of people who want to use rented servers for DDOS and such, ideally without paying or being tied to a real identity.
It also attracts people who will do a card chargeback if the server isn't what they wanted or after they've used it for some temporary event. Some people don't appreciate that chargebacks are expensive for the low-cost supplier, and some people don't care.
Low-cost also means the penalty cost of credit card chargebacks is a higher proportion of income, even if the number of them was the same. It might be so much higher that the business couldn't be profitable at the prices it offers if it didn't aggressively filter which customers it takes on.
Could some "insurance", paid extra, be a solution to this? As in saying to Hetzner, hey, I'm a legitimate consumer, willing to pay extra for, say, a year, so that you do some additional checks (EDIT: in advance) and/or switch me to a better tier of more relaxed "fraud detection", offering some hot line, like between Russian and American generals, to help avoid any accidental, hm, deletions?
I worked for them a couple of years ago. I can confidently say that as of 2 years ago, the only thing images of your face would've been used for is verifying if it's an actual human face (e.g. not a photo, mask, etc.) and performing a facial match with the photo on your document. Also, at least 2 years ago every identification flow had a human review, to weed out false positives and negatives. I'm fairly certain that these things have stayed the same, as the guys running it are a good bunch of people, and don't have any ulterior motives for using AI besides moving the SLA from humans to AI.
Same thing here, signed up, got randomly flagged, provided all of the required documents and was still rejected with no option of using the service. The first and only time I had an experience like that with a business.
The answer of course is zero, but would you please stop posting unsubstantive comments? You've been doing it a lot, unfortunately, and we eventually have to ban such accounts.
This is certainly interesting. I've been using DO for something like 8 years and I really like them.
I don't plan to switch but Hetzner offers a CPU optimized server with 4 CPU cores and 8 GB of memory for $17 USD / month. DigitalOcean offers the same thing for $56 USD / month. It's hard to ignore how much of a difference that is.
Even the smaller'ish instances have a huge difference. Hetzner has a 2 CPU core 4 GB of memory server for $6.75 USD / month. DO offers the same thing for $24 USD / month and that's using the worse grade CPU. If you pick the higher end AMD CPU it's $28 a month (which sounds similar to what Hetzen is using but ~4x less price). For perspective DO's $7 USD / month price point gives you 1 CPU and 1 GB of memory.
Edit: As someone brought up in the comments below, it's possible that the Hetzner price ends up being ~25% cheaper due to not charging VAT. This will depend where you live. I converted Euros to USD on Google based on their public pricing page https://www.hetzner.com/cloud as a US site visitor.
And the difference is even bigger if you get a bare metal instance.
Performance is insane if you compare it to dedicated VMs or worse cloud VMs.
In one of my Java toy projects I got http responses to sub 10ms, that's includes, querying a small amount of data from postgres and responding as JSON. The tls handshakes might have been reused through, now that I think about it. The number was taken from the network tab while switching around routes in the pwa
The bare metal instances generally have nvme storage, so you get incredible IO
As a reference from a recent test: I went with the smallest OVH bare metal server that was discounted to 30€/month the other day (normal price 60€) and got about 2k with hdparm.
Running the same test on my 45€ hetzner instance got me over 3.1k.
And the hetzner instance also got twice the memory (64gb), though that didn't impact this particular metric.
The price increase of Digital Ocean is what prompted us to evaluate different US Cloud providers in which we found Hetzner offering by far the best value [1], what's even nicer was that the prices for the instances ended up being ~25% cheaper than what they're advertising, e.g. their 4x vCPU / 8GB RAM / 160GB HDD is advertised at €17.27 but when creating instances of them in their cloud console it only ends up costing €13.10. Not sure why that is, perhaps it's the difference of their hourly vs monthly cost.
@nickjj -- That might be a difference in which VAT applies to your location. If you saw an advertisement in German, or perhaps an English ad in the UK or somewhere else in Europe, you might see a price difference similar to this if your actual region is one where we are not required to charge VAT. We also have a list of what VAT rates apply to specific locations: https://docs.hetzner.com/accounts-panel/accounts/payment-faq... --Katie
That's a good point. The USD prices in my comment were from taking the publicly listed Euro price while visiting the site in the US and converting it to USD on Google. I didn't sign up and try to create these resources. I've updated my original comment to reflect this.
That's good to know the prices are even more of a difference. Hopefully this sparks DO into being more competitively priced. If someone wants to throw up a few servers somewhere on the cloud and doesn't care about managed features, it's really hard not to consider using Hetzner for that.
For those curious about what it actually looks like in a Hetzner datacenter, der8auer made a video about their Falkenstein, Germany location: https://www.youtube.com/watch?v=5eo8nz_niiM
Lovely, but as a customer from Brazil i they never accepted me on their verification step. I even sent a picture from my passport, did the whole face scan verification but nothing. Gave some pretty sensitive information, got a rejection in return lol.
Again, I am sorry that we could not accept your accounts. Every day, we also reject a large number of accounts from German and European users if there are red flags. (And unfortunately, there are some real situations that are logical that can lead still to false red flags.)
For example, I have an unusual last name, especially for here in Germany, and when I have done personal shopping, I have had a number of online e-shops reject my new accounts because that looked suspicious or because they didn't think an address I wanted to send something to was real. So I understand that your frustration is real, and I am sorry for it. --Katie
You are the only business I have ever tried to use that straight up rejected me with no explanation, and obviously it's a huge issue as in just this one thread there are multiple people saying the same thing, so there must be thousands if not tens of thousands of legitimate customers you've turned down.
Does anyone have some experience regarding the bandwidth between Hetzner's old US location, Virginia, and the big cloud providers (AWS/GCP/Azure)? My use case requires lots of ingress into these clouds. I am currently using Cloudflare's R2 object storage but I'm not satisfied with the bandwidth as it fluctuates between 40 MB/s and 200 MB/s when loading to AWS.
Last time I checked through Vultr[0] from South America, Ashburn was significantly faster than Digital Ocean in SFO. Though, not sure if this helps you.
> The new location at Hillsboro will host Hetzner Cloud servers mounted with AMD processors
A kind of interesting statement, given that Intel has a large presence in Hillsboro (>15K employees IIRC). Not that this would matter to Hetzner, but still curious.
At no point did I say or imply anything to that end. I found it curious, as their other DCs offer servers with Intel and AMD processors, while in this new DC they specifically call out using AMD processors.
344 comments
[ 3.1 ms ] story [ 359 ms ] threadI'd definitely vote on that!
but you are hosting cloud offering on some physical servers already, so you can start selling these servers as dedicated..
I love their pricing. I’d love it even more if I could announce my IPs on their dedicated offerings.
Perhaps it could be an option to let people back up their words with money up front? I.e. put $10k in the account, proves you can probably afford more than 10 servers.
What I'd like to have is something like Linode's start scripts feature which I've always found to be very nicely implemented.
If you really really need startup scripts, it'll be trivial to add it to your own image. Packer (and similar tooling) makes it trivial to create your own images as well.
Hetzner offers "managed bare metal", for extremely competitive prices. But, in my experience, exactly what I'd expects: "pay peanuts, get peanuts". I love, and chose them for some proof of concepts, early phase and hobby projects. But would not choose them for anything that requires serious stability and availability.
Not because they have flakey or even unpredictable service, but because the trade-off is that "bare metal" requires more work done by me, more responsibility for me, less options for quick failover and so on.
Just like for some situations, a server in your attic is the perfect fit, yet for others a managed cloud infra is the perfect fit, hetzner has some sweet spots.
Since then they priced their boxes competitively relative to DO. A little bit less. That's still holding.
Also note that at that time hosting in Europe wasn't a popular option, unlike what we see now.
Dedicated servers there were an option way before DigitalOcean, if I remember correctly (I recall at one point having some Linode and some Hetzner servers, DigitalOcean was not in the picture).
Early example: a weirdly memorable ad captaign around 2000 that for many years occupied the single most expensive computer related print ad slot in Germany (decisive spending) with a series of ads that seemed not quite "high production values", but also not deliberately grungy, a weird "definitely trying to be high gloss perfection, but somehow not quite there" (like in-house best effort or some local design house, certainly not the big-name agency you'd expect for ads on that slot). More recent example: their hardware seems to be a continuation from early-Google style "desktops on shelves" that's now a custom rack design (still noticeably lower density than typical 16") that's all about finding good price/reliability spots in cheap CotS parts, e.g. according to certain "begins the scene" blogger visits they sort for publicity, price-optimized custom versions of desktop mainboards (same PCB but not placing any parts they don't need). Chances are company with big investor backing would either go all standard rack parts (from a supplier like Dell or something like that) or go all in designing their own.
Very happy with both the cloud and dedicated servers, but it's weird how bad the communication is on the object storage thing. If it's not going to happen, why not say so?
Wasabi is a great, inexpensive object store for anyone who's looking for that. I've had a really good experience with them (we have a little over 80TB of data there), and have found their support to be top-notch especially when you consider the price.
It wouldn’t even needed to be integrated to their API and billing.
Storj pricing is basically unbeatable. $4/TB/month, $7/TB/month for egress bandwidth which (I heard, yet to try) can be saved further if you put something like cloudflare in front of your bucket.
Speaking as someone who set up their own minio cluster (on Hetzner) as a way to have object storage at the lowest cost possible, if Storj was an option 3 years ago, I would have saved me quite a bit of money and time.
[0]: https://storj.io
Wish they would add managed databases to their cloud.
I currently have hybrid DigitalOcean / Hetzner setup to take advantge of load balancer and managed database at DO. If Hetzner provided some of those, I'd gladly switch.
I am not managing replicated postgresql myself ever again.
Am I you?
I've been there for 2y, pretty happy with them so far.
+1 for managed Cloud DBs. I'm surprised they don't exist already, given that they have managed DBs on their cheap web hosting platform. Shouldn't be a big step to make that available in Cloud.
Another request I have is virtual routers, so I only need 1 dedicated IP address and can NAT (or whatever) everything else. I get that I can do this with a small Cloud Instance and a private network, but those things are a pain to manage and I'm sure Hetzner could do a better job than myself :-)
IT is basically 2 things, at its core:
* compute
* storage
Hetzner offers cheap compute, which is great because modern applications can have stateless web/app servers.
Now, the missing part is storage, and that's much harder and riskier since it's state, inherently. A server dies, you lose stuff if you misconfigure it.
So people want a fully managed IT solution.
It's simple and it's obvious conceptually, just hard.
And I guess Hetzner won't do it because it's hard thus expensive to do.
Postgres (and later some exciting flavors like Neon) is actually the only database I plan to support, at least for right now -- MySQL and MongoDB are not even on the roadmap really.
I might make an exception for RethinkDB since I loved it so dearly.
Don't get my hopes up... I haven't used RethinkDB in ages and have fully moved to Postgres though
But also...
https://www.rethinkdb.cloud
Don't know who is behind it -- a company named MostlyTyped (not sure if this has old RDB engineers in it), but warms my heart that the repo is still seeing commits and usage.
I want hetzner to concentrate on the core capabilities: servers, storage.
I want Hetzner to provide a PLATFORM to many different database-as-a-service on top of Hetzner. I want choice, and I don't want Hetzner doing what AWS does: stifling competition by releaseing their own flavor-of-open-source-database that puts the core projects sponsoring the open source code out of business.
Since the EU datacenters seem to be operated by EU companies and the US company is merely a sibling subsidiary of Ensoxx, which itself is also an EU company, this should provide sufficient isolation to prevent interference from US agencies short of direct sabotage or espionage (since the EU staff is not in the chain of command of the US company).
So for a definitive answer you probably want your lawyers to talk to Hetzner's lawyers but at face value this is at least miles ahead of any US-based cloud provider, which in all honesty is still the default solution for most EU-based companies despite this ruling.
Conclusion:
In summary, you as a customer do have influence - to a certain extent - on shaping who has access to the data on your servers. EU and US authorities do have to follow the laws and legal procedures in requesting data. However, this may give you a false sense of security since some authorities have been known to stretch or violate agreements. If you require a web hosting company that has absolutely no connections to the USA, then unfortunately, we may no longer be the best choice for you. Since Hetzner US LLC is part of the Hetzner Group, there certainly is a connection. We hope that we have explained things clearly from our point of view using the two above case studies.
So, because Hetzner is not owned by a US company, stuff like the CLOUD act doesn't apply to them. So, if you have a contract with the German entity of Hetzner and use a German server, you should be fine in terms of GDPR.
Any of the big cloud providers can claim that they comply with EU legislation, but they also have to comply with US-legislation and if 3-letter agency wants to have some data from one of their subsidiaries in EU, then they can/will decide which contract to breach.
I read Hetzners statements as being that they can no longer guarantee that they will not be forced to do the same - but that can be my reading of their statement that is wrong.
If I already had them as hosting-partner for a solution that fell under Schrems II, I would have them confirm this, to be sure.
Hetzner US does not have a European subsidary and therefore cannot violate GDPR (assuming US personal can't access EU customer data).
Hetzner HQ is in Germany and is not allowed to enforce the CLOUD Act outside the US
But if I was under legal/contractual obligations, with Hetzner as my hosting provider, I would have their legal department confirm this.
Since Hetzner found the need for appending the paragraph I referenced, they must have become aware of something.
Now that they are entangled with US law there might be an incentive to be as a cooperative as possible.
Yet, Hetzner is still a "better" option (with regards to data protection) than any of the big US-based cloud providers.
Imho, as soon as you do business with the US or trade in US Dollars, you need to play nice with the relevant authorities.
If I understood it correctly, Hetzner is now "infected" in the same way as the three US cloud providers are. The Schrems II verdict and Cloud ACT basically concludes that no European company can exist in the US and vice versa without having to deal with the same pesky legislation.
An alternative could of course be that Hetzner created a new US based company where the EU parent Hetzner company only holds a minority ownership in the new US-based company. The EU based parent company in turn then "sells" its technology to the new US company. This way, the arrangement becomes more reminiscent of how IBM has sold its mainframe to European companies...
What makes you think that a European company operating within US jurisdiction would not be subject to the same laws?
If the European company receives a request from the US authorities for information, they need to follow the same legislation as the US companies do. Just because it's a subsidiary won't help. The authority will say "we want to know everything you know about the following person, please give us the information, otherwise...". The authority will not distinguish whether it is a subsidiary or the parent company.
Of course have the choice to just ignore the request from US authorities, but then you have to be aware of the consequences, i.e. quickly give up and shut down the subsidiary and stop trading with US dollars.
This is the root of the problem. CLOUD act has been ruled illegal in the EU just as you said, but it is also illegal not to comply with CLOUD act in the US. And companies operating on both continents in practice need to comply with both laws, regardless of whether it is a parent company or a subsidiary.
At least that's how I interpret it...
But it really depends on how infectious just owning a company is, which I have no idea. But my gut-feeling is that it shouldn't be too infectious, since otherwise just buying a single share of a company operating in another country would put you into legal peril (who controls the subsidiary here is not really relevant, since the Cloud act wants to swim in the opposite direction in your scenario, therefore it shouldn't matter if it's 0.01% or 100%).
Hetzner Europe is owned by Hetzner Group, a German company. Hetzner US is also owned by that German company. Hetzner Europe isn't owned by a US company, it's just a sibling to one.
In the 2022 annual financial statement they do mention expanding to the US, though they don't go into the legal details. As the link posted in the cousin comment mentions, though:
> Hetzner US LLC, as a subsidiary of Hetzner Online GmbH, provides data center services within the USA for the parent company, Hetzner Online.[2]
So there is no US owner.
[0]: https://www.northdata.de/Hetzner+Online+GmbH,+Gunzenhausen/A...
[1]: https://www.unternehmensregister.de/ (enter "ENSoXX Holding AG" in the text field)
[2]: https://docs.hetzner.com/general/general-terms-and-condition...
78M profit on 290M revenue
(See [0] above)
_ edited figures _
There are some Youtube videos of people getting tours of their data centers. It's a very custom setup, keeping both component costs and energy costs down. Their earliest servers were basically tower PCs on shelves, their more recent generations are more akin to custom rack designs with inhouse-assembled servers, with a datacenter design that exploits natural convection to do a lot of the cooling.
a friend of mine told some support staff got 20k EUR in bonuses last year because of that
probably they figured out, it's easier to give some of the profits away than pay a higher tax
Choices for EU only companies are getting smaller, and this is a real let-down.
Also, track record of at least some of them is not the best, if we are to believe, for example, comments on HN. I've made initial research about most of them. Just to give some examples about Scaleaway: prices moving up by 75% without notice (https://news.ycombinator.com/item?id=25999148), zero notice of the removal of the ability to start ARM64 instances (https://news.ycombinator.com/item?id=22923413).
I want multi node, multi AZ hetzner k8s clusters ;)
I already run cluster in GER and US and what has happened in the last years has been amazing, but its not all the way there just yet!
And those failed communications between provider and customer are why everyone is frustrated with the big corpos (google, cloudflare, stripe, etc.) which Hetzner is apparently bound to become.
This is not to excuse bad support experiences. I just think we should appreciate it when there is still a channel to a real human being willing to be helpful and engage constructively, because it's getting increasingly rare these days.
Hetzner is on my radar for a long time as an alternative to simpler deployments on AWS. It's great they are monitoring internet forums and try to be helpful, but also very frustrating to read such problems apparently still occur.
We recently had a mission- and time-critical issue with play store and our first reaction was to contact our gcp account manager and old google friends because we knew play store communication sucks. It should not be this way.
Let's hope I don't need another "one-time" server replacement because of your faulty servers, because I already used mine. 10 hours of downtime with angry customers on my tail sounds fun.
Doesn't surprise me, German companies always know better and will try to prove you wrong, whether you're their customer or supplier. Such is the business culture, I think.
Kudos for assisting your colleagues with better communication in English, I think this practice could be useful for many organizations with people that speak English as a foreign language.
My impression is that in Germany you bear the burden of proof that you actually have a problem, that its not your own fault, and that you did your part trying to fix it.
You usually can get good support in Germany ( even at government offices ) , IF you show up with your Leitordner ( legendary German ring binders) with all the receipts, all the possibly relevant account numbers, transaction ids and a detailed analysis of your own problem.
There is nothing that signals to German support staff that you have to be taken seriously like a ring binder, preferably with color coded markers at the margins and lots of punch pockets.
If you think this is satire, try it the next time you have an in person appointment...
The rest is entirely correct, though.
Yep. Its a plight that afflicts almost all engineers around the world, and German companies are even more afflicted by it due to the German engineering culture...
I chalked it up to our decision to run our databases on consumer grade hardware and didn't give it a second thought.
Yep. The attitude of the engineering support is something that Hetzner needs to improve a LOT on. They tend to treat customers as if they are treating members of their own open source or engineering community - scolding the customers when they think the customer (community member) is being unreasonable etc.
Its a great thing that eng. support is just a ticket and 20-45 minutes away at Hetzner. But the attitude needs a lot of improvement. Imagine that you are a startup facing some quirky issue that affects your business in the middle of the night and having to deal with attitude and scolds from datacenter engineers...
The UX behind managing instances is delightfully pleasant where new instances are available faster than any other cloud provider we've used, within seconds of creating an instance you can immediately login with your configured SSH keys. Another nice feature is being able to "rescale" your instance to higher specs after a restart [2], so you can confidentially start with a small instance that just meets your current workload knowing that you can easily scale up your instances as your workload increases.
AWS RDS was the only critical service keeping us on AWS, a service we no longer need in our new Apps which we're building with SQLite thanks to the effortless replication in Litestream [3] that we're using to replicate to Cloudflare R2 - another great value S3 alternative with $0 egress fees [4] where you can get even greater value & performance when hosting behind their free CDN.
[1] https://servicestack.net/blog/finding-best-us-value-cloud-pr...
[2] https://bizanosa.com/how-to-upgrade-resize-hetzner-cloud-ser...
[3] https://docs.servicestack.net/ormlite/litestream
[4] https://www.cloudflare.com/products/r2/
Edit: And if you do manged Kubernetes and managed Kafka Instances the number would go up to like 95%. Oh and those Videos with der8auer? Really awesome to see, do Linus Tech Tipps or Level1Techs next!
https://github.com/kubermatic/kubeone/tree/main/examples/ter...
Though I'm sure if you ask and they have the money for it lying around, Kubermatic will sell you an SLA.
For folks wanting to learn more, we have this nice getting started tutorial covering all the cloud providers we support including Hetzner: https://docs.kubermatic.com/kubeone/v1.5/tutorials/creating-...
And for folks asking about SLAs, support, and stuff like that, I recommend checking out our KubeOne Managed Offering: https://www.kubermatic.com/products/managed-kubermatic-kuber...
I'm also happy to answer any questions y'all might have. :)
Proof: https://forum.hetzner.com/index.php?thread/21421-docker-cont...
You can find the link in my profile or here[0]
[0]: https://nimbusws.com
I have seen the "apps" in image selection in the Hetzner cloud console, so I'm convinced they're at least thinking about going up the value chain.
Would you have wanted a hypothetical discussion though (directed at grandparent), I would be up for that. Email in my profile
If we're taking requests, I would love a Fargate-like service. It would be absolutely fantastic if Hetzner provided a service where users uploaded a Docker image, and Hetzner took care of handling the networking bits and running it, and gather metrics.
Upload a docker image, specify container size (1cpu 2gb)
go live
scale from 1rps to 1000 rps any time
stateless
pay per request or pay per container
AppRunner, Fargate and Cloud Run have different ergonomics and specifics but thanks for this outline. v1 is likely to be pay per container but other than that feels doable.
Superb!
> would you mind telling me a bit about the metrics and the scale you'd expect?
It will depend mostly on what the service offers.
If the service only supports running a single isolated container without any scaling whatsoever then it would be helpful if we could monitor básico stuff like CPU and memory utilization, and also network traffic, free disk space, and also disk IO. If the service supports auto-scaling then it would be helpful to track all resource utilization rates along with all alarms and events involved. Auto-scaling also implies load balancing thus if that's the case then it would also be helpful to track the basic load balancing indicators, as well as request logs.
In the end it really depends on what services you're planning on offering, and how you'll charge for it. As a user I would need to monitor any metric which is directly and indirectly involved in determining cost, and on top of that I need to monitor performance.
> Would you expect always on or more of an ephemeral container?
The most pressing need would be always on containers to be able to go the lift-and-shift onboarding route to managed services, but ephemeral containers sound like function-as-a-service and those are pretty exciting as well.
Free disk space would imply elastic block storage or something similar so I’ll need to give that a think!
V1 is very likely to be always on so great that it’s the core use case for you!
[1] https://www.ayedo.de
https://registry.terraform.io/providers/hetznercloud/hcloud/...
https://elest.io/
Disclaimer: I'm the CTO
I don't understand their price page. They claim $0 egress fees, but their free "Class A operations (mutate state)" and "Class B operations (read state)" have a mothly cap. After that you pay by the number. Isn't that an egress free?
One time I hosted a new movie (mkv, 23gb+), that cost like $17 to “rent” on Amazon prime… so my extended family could just take the link, download it, or just watch it in their mobile browser or laptop.
The egress of the streaming alone (not downloading), while they aircasted, definitely adds up much more than one would expect!
S3 is charging $0.023 per GB, right?
IIUC - there's not a fixed cost for sending 1 GB of data. The cost differs based on where you send from and send to. So it'd be hard to have a really good estimate - but I'm wondering if anyone has a good ballpark figure.
My understanding is that transferring data on the same continent (vast majority of traffic) should be <$0.0005 - meaning the margin is really high.
I know it's more in Amazon's interest for your cloud usage to be as inefficient as possible - so they can charge you as much as possible and get as much margin as possible.
However, this product doesn't seem to make sense to me.
Why would you even pick your S3 regions? Shouldn't AWS balance your data for you across continents so that your data egress is automatically optimized?
Is that how Cloudflare works?
Bandwith cost highly depends on the scale and target network. Cloudflare has a good blogpost on relative bandwith cost: https://blog.cloudflare.com/bandwidth-costs-around-the-world...
To put numbers to the relativity, for a somewhat smaller datacenter in Central Europe with a 100Gbit/s connection, the running Cost of transit is somewhere around 0.00004€ per GB, on a theoretically fully sustained connection(not realistic, also with no redundancy or hardware). Peering is basically free(after initial buildout) with around 3k€ per 100Gbit/s (https://www.ams-ix.net/ams/pricing). This all excludes cost of investment for initial buildout and hardware. On a scale on amazons level this becomes even cheaper, since operating your own network is cheaper than buying transit.
Hetzner charges 1,19€ per TB, amazon 90$(on the first 10 TB). So amazon probably doesnt really care about the price differences per region and went with a mixed calculation, since their margins are so absurdly huge.
also, major parts of the cost are in hardware and staffing/development, of which amazon has way more, due to the demands of SDN.
No thanks, I'll keep using Digital Ocean or someone who doesn't make me jump through hoops.
The only thing I imagine someone could do with the copy of my passport is pretend to be me when they sign up for a similar service.
Recently I signed up for something else from Hetzner (needed temporary storage) and they didn't request anything even though I had closed my account before so I created a new one. So perhaps they have mended their ways.
I emphasize that I personally did not run any such tests, yet. But was wondering, since this is Hetzner thread, that maybe someone can share their experience, in particular comparing AWS's gp2/gp3 based deployments vs Hetzner's volumes.
I would also add that on AWS, for example, you can nearly seamlessly expand EBS volumes in size without downtime. Last time I checked, not an option on Hetzner - you must take care of expanding the file system yourself. Which makes sense, as Hetzner is more basic service, but it's worth remembering that there are various differences like that, when comparing the day to day operations between such providers.
Network volumes always have lower iops than local disks. Thats expected, except when you pay a huge price.
> I emphasize that I personally did not run any such tests, yet. But was wondering, since this is Hetzner thread, that maybe someone can share their experience, in particular comparing AWS's gp2/gp3 based deployments vs Hetzner's volumes.
AWS GP3 has a baseline performance of 3000 iops, and from there on you have to pay for each iop/s. To reach the performance of hetzner volumes you have to add 20$/month. But with AWS you can get higher than the hetzner value. But the iops performance of the dedicated machine is not possible with gp3 (max. 16k, compared to hetzner 30-40k for local disks).
With io2 devices you can get higher than 16k iops, and higher than the hetzner local disk.
https://aws.amazon.com/de/ebs/general-purpose/
Hetzner Cloud pricing is great, but their dedicated servers are even cheaper. For example, they offer a 16-core (32-vCPU) Ryzen 9 5950X for €103 with 128GB RAM and 2 x 3.84 TB NVMe SSDs. They offer a cloud server with 32 vCPU, 128GB RAM, and 600GB of storage for €296 - nearly triple the price and those CPU cores are probably not as good since they're likely Zen2 cores rather than the Zen3 cores of the Ryzen 9 5950X.
https://www.cpubenchmark.net/cpu.php?cpu=AMD+EPYC+7502P
https://www.cpubenchmark.net/cpu.php?cpu=AMD+Ryzen+9+5950X
I know, benchmarks aren't everything, but the Ryzen is getting 1,432 per vCPU while the EPYC is getting 766.
The AX51 costs €59 for 8 Zen2 cores (16 vCPU) and 64GB RAM while the CCX41 costs €154 for 16 vCPU and 64GB RAM.
I know, the cloud servers come with flexibility, hourly billing, and no set-up fees. I also know that their cloud pricing is very good. Still, I wish I could get a few AX101s in the US. 3 AX101 servers would be €310/mo, each with 16 Zen3 cores, 128GB RAM, and 2x 3.8TB of storage.
https://www.hetzner.com/dedicated-rootserver/matrix-ax
As it was going to host (among other things) my mail server, I could not use my primary (self-hosted) email address - because that's asking for trouble if I ever run into issues. It seems using Protonmail triggered something on their side.
They asked me for a copy of my ID card, which I happily provided. But they still refused my account, without explanation. Oh well, their loss. I've been at OVH now for a few years without any issues. OVH's product is definitely worse, but at least they'll actually let you use them.
Clearly if you are rejecting legitimate customers for arbitrary reasons there’s still some work to do on your approval process.
I mean, I get that there are bad actors, and that it’s rationally better for Hetzner to have false positives than false negatives. But it just feels wrong. Legitimate customers shouldn’t be rejected until they prove they’re malicious.
Low-cost attracts more fraudulent customers, using stolen credit card numbers. Hosting is particularly bad for this compared with other low-cost services, because of the community of people who want to use rented servers for DDOS and such, ideally without paying or being tied to a real identity.
It also attracts people who will do a card chargeback if the server isn't what they wanted or after they've used it for some temporary event. Some people don't appreciate that chargebacks are expensive for the low-cost supplier, and some people don't care.
Low-cost also means the penalty cost of credit card chargebacks is a higher proportion of income, even if the number of them was the same. It might be so much higher that the business couldn't be profitable at the prices it offers if it didn't aggressively filter which customers it takes on.
Or would that just not scale?
That's their courteous treatment. They asked me to consent to an AI scanning my face!
edit: Specifically with this startup,
https://www.idenfy.com/identity-verification-service/
I too provided US State issued identification, which still resulted in my account being denied.
I generally think highly of the company, and want to pay them for their services; but they made it impossible for me.
https://news.ycombinator.com/newsguidelines.html
I don't plan to switch but Hetzner offers a CPU optimized server with 4 CPU cores and 8 GB of memory for $17 USD / month. DigitalOcean offers the same thing for $56 USD / month. It's hard to ignore how much of a difference that is.
Even the smaller'ish instances have a huge difference. Hetzner has a 2 CPU core 4 GB of memory server for $6.75 USD / month. DO offers the same thing for $24 USD / month and that's using the worse grade CPU. If you pick the higher end AMD CPU it's $28 a month (which sounds similar to what Hetzen is using but ~4x less price). For perspective DO's $7 USD / month price point gives you 1 CPU and 1 GB of memory.
Edit: As someone brought up in the comments below, it's possible that the Hetzner price ends up being ~25% cheaper due to not charging VAT. This will depend where you live. I converted Euros to USD on Google based on their public pricing page https://www.hetzner.com/cloud as a US site visitor.
In one of my Java toy projects I got http responses to sub 10ms, that's includes, querying a small amount of data from postgres and responding as JSON. The tls handshakes might have been reused through, now that I think about it. The number was taken from the network tab while switching around routes in the pwa
The bare metal instances generally have nvme storage, so you get incredible IO
As a reference from a recent test: I went with the smallest OVH bare metal server that was discounted to 30€/month the other day (normal price 60€) and got about 2k with hdparm. Running the same test on my 45€ hetzner instance got me over 3.1k. And the hetzner instance also got twice the memory (64gb), though that didn't impact this particular metric.
[1] https://servicestack.net/blog/finding-best-us-value-cloud-pr...
That's good to know the prices are even more of a difference. Hopefully this sparks DO into being more competitively priced. If someone wants to throw up a few servers somewhere on the cloud and doesn't care about managed features, it's really hard not to consider using Hetzner for that.
Last time I checked through Vultr[0] from South America, Ashburn was significantly faster than Digital Ocean in SFO. Though, not sure if this helps you.
[0]: https://sao-br-ping.vultr.com
Simple UI, certainly robust infra (for my case at least) and the best, by far, prices in the market.
A kind of interesting statement, given that Intel has a large presence in Hillsboro (>15K employees IIRC). Not that this would matter to Hetzner, but still curious.