I'm having some servers at Hetzner (beside AWS and DigitalOcean) for the last 15 years and never had 8 hours of continuous downtime, so this really astonishes and scares me.
Could you point to a Hetzner post where this has happened?
(so I can reevaluate if I might move to OVH).
You are just like the loudest and most popular guy in the bar.
How do you propose to "back up" such a claim after 4-5 years without the Internet archive (which was mentioned there before the GP changed his post in response to your mud slinging)?
For the record, I've had significant problems with network connectivity being constant at OVH's BHS datacenter. Even when presented with evidence that they're having downtime issues (repeated customer mentions on Twitter, New Relic reports, etc.), OVH refused to honor their SLA or even acknowledge the problem. I had a week with 96% uptime because of their network link problems.
[parent commenter complained about 8 hour long downtimes and otherwise bad experience with Hetzner. He/she has since replaced the comment with insults to other HN users.]
Counter experience: 4 years with Hetzner from small VPS to huge machines and based on traceroute servers in different sections/buildings of their datacenters. No hardware issues or hourlong downtimes you describe (other than myself messing up software RAID configuration).
The more providers offer something like this, the merrier. I understand that this isn't a layer 7 solution, but that has it's downsides as well - Cloudflare (or any other reverse proxy) will MITM all your TLS traffic, for example.
It's also time to address the elephant in the room: AWS. "Oops, you got DDOS'ed? Here, have a $50k invoice"
CloudFlare also regularly speaks about attacks and mitigations, therefore is helping the community to build better defences. Other providers stay shy and never disclose their magic. We believe DDoS is an internet wide problem and one of the ways to solve it is to spread the mitigation know how.
I think their work is fantastic, but I really don't like that every site I visit meets me with CF's captcha page. It's not only annoying per se, but the privacy implications are unsettling. I welcome the competition.
This was never the intention. Part of the problem is inertion - cf operates large and complex application that was designed back when we had only a handful of customers. Part of the problem is technical - the privacy-centric anti-abuse technologies don't exist yet.
Please do help us fix this. Report issues, help us understand when we have incorrect IP reputation. Help us find captcha accessibility problems. And maybe - join the team to actually code the fix.
Problem is, your "incorrect IP reputation" concept is fundamentally flawed. As an example, I noticed that most VPN exit nodes have "incorrect IP reputation", which means if I want to browse the internet without my government spying on me, I have to wade through all your CAPTCHAs.
Keep in mind, unless you're using your own personal VPN off a self-hosted machine, you're likely to be sharing your IP address with other (potentially) malicious actors trying to hide their tracks
“fundamentally flawed” is not synonymous with “not supporting the style of anonymity which I prefer”. There's no evidence supporting the assumption that those VPN exit nodes’ IP reputation is actually incorrect rather than earned by the behaviour of other customers.
We went through this in the 90s where a few people were upset that they couldn't send email directly from their dialup connection, because they were still thinking of the world as it was in 1993 before spam became so prevalent and anyone who ran a mail server was constantly trying to deal with thousands of dialup IPs trying to deliver spam. What actually worked was that people changed the way they worked to use things like authenticated SMTP relays so you could simply block entire dialup ranges rather than try to come up with a spam-blocking AI.
The browsing system could be improved by something like a CloudFlare login system or long-term persistent authentication storage so you'd only see a CAPTCHA once a week. Unfortunately, most of the complaints come from very pro-anonymity users – which is a legitimate position but also means that it's a community which is going to be significantly more likely than average to have things like cookie & JavaScript blocking, so the complaints would simply shift to “I shouldn't have to create an account!” or “Why can't I disable JavaScript and cookies for your site!?!”
I stand by my assertion — the idea that an IP address can have "reputation" is fundamentally flawed. The SMTP example that you cite is actually a good one: the whole reason why we started assigning "reputation" to IP addresses is because we could not be bothered to improve SMTP over the last 40 years or so and it still assumes we live in a world of trust.
CloudFlare (and everyone else for that matter) should stop assigning "reputation" to IP addresses. An IP address is a network location, think of it as a temporary storage box which could be occupied by anyone and anything. We should move beyond coloring boxes.
So what's your alternative? People use addresses because they're a strong signal with relatively low false-positives (not many people are tor/VPN users with cookie blocking). How do we do better short of deploying a login system or a unique client identifier which cannot be disabled?
This is an inherent problem with numerous different people using the same IP addresses (VPN, Tor, etc). And might be a bigger problem than Cloudflare can resolve on their own. Cloudflare has to protect its network and its customers like a normal business, so VPN and Tor IP addresses will naturally be viewed with suspicion.
Also, I think the captcha solution is better than what most administrators do, which is to block those IPs outright.
There is probably no solution to this problem. Criminals regularly use those same services you're using.
Cloudflare's job is to block potential attacks, and blocking or CAPTCHA-restricting a subnet tied to a large number of attacks against their infrastructure seems reasonable.
Either switch VPN providers, or deal with the CAPTCHAs.
Tor is used for abuse. If you want the privacy* you get with Tor the price you pay is that sites and services will check that you're not one of the abusers. At least it's not a flat-out ban. Any other large network on such a small IP space (as an example maybe a NAT-using mobile ISP) would be checked just as harshly.
* Or any other reason you're using it.
Edit: If I'm wrong can you please reply explaining why? I'm having an emotional rollercoaster of up and downvotes here! Please do feel free to correct me if I'm wrong, we all need learnin'
You're looking at it purely from the user side. Look at it from the admin side. Tor is basically a massive open proxy, and by blocking it or throwing up human checks like captchas, you eliminate a significant source of spam and abuse.
There's not much to be done about this otherwise - a Tor user is sharing a network with a significantly higher than usual amount of the bad elements of the internet.
You can put captchas where they belong—on comments etc. Instead, Cloudflare punishes people simply browsing; they nuke the 99.99999% of visitors that have zero intention of interacting with the page beyond doing few GETs.
Oh and don’t forget they’ll even put the captchas on subdomains, like img.domain.tld. Go visit stackoverflow via VPN/TOR and watch how the site has no styling/images even if you do their stupid captcha on stackoverflow.com. They’ll still serve you another one on static{1-50}.stackoverflow.com. Fun.
A few GETs spread out the right way and repeated often enough is a DDOS.
There's a very good reason why Cloudflare does this. Cloudflare isn't "punishing" anybody, their job is to protect the people that use their service. Tor (or any other open proxy) is a massive source of bogus and/or abusive traffic.
Of course they’re punishing legitimate visitors. If there weren’t any humans getting captchas there wouldn’t be any need for captchas in the first place.
It wouldn’t be so bad, except Cloudflare doesn’t have any idea what they’re doing. Like the static.domain.tld issue I’ve mentioned—not even documented. This breaks websites in a very, very, bad way.
Next we have the lovely security setting called “Essentially-off.” Which does absolutely nothing to the captcha system. Apparently “essentially” means “not really,” according to Cloudflare.
Also forget VPN/Tor. I’m getting consistently captchad by Cloudflare just visiting mostly american-only websites from residential networks in Europe.
They’re either incompetent or just plain malicious. Right now I’m swaying towards the latter :/
I run a site (not behind CF), and Tor is used heavily to scrape our uncacheable and extremely heavy to render pages for abusive/content theft purposes.
Using a massively shared IP space of any kind is only available by submitting to a poor experience.
Nobody is saying "hey this guy has privacy, lets make this experience a pain in the arse". They're saying "Man this set of specific IPs are really hammering my system looking for WordPress exploits and we're not even running that".
To treat Tor better than regular traffic would be to discriminate against Tor, which would invite more abuse of Tor.
Pure rough estimates, but consumer ISP subnets are probably ~1-3% malicious, while with Tor's rotating IPs, if you look over the past 5 years odds are at least 40% of exit node IPs have been associated with malicious activity at least once.
It makes perfect sense to add additional verification challenges for Tor users.
Also, anecdotal, but I work in infosec for a large US company and see a lot of traffic and read a lot of logs every day. The majority of all spam, fraud, and scan activity comes from Tor, or consumer-advertised VPN services, or various European VPSs. Residential ISPs (excluding Chinese and Russian ones) make up almost no part of it, relative to those others.
I'm pretty sure Cloudflare's challenge criteria is sourced from real data they've generated from traffic they've seen. They aren't trying to single out Tor users.
> It makes perfect sense to add additional verification challenges for Tor users.
I’m actually perfectly fine with that. The issue is how Cloudflare does it. They should just offer something like X-Cloudflare-Fraud-Score header or even <x-cloudflare-protected-frame /> tag (they’re already doing SSI), but instead they just nuke the hell out of everything.
The way they do ‘protection’ is openly hostile to open web, or even neutral web. Whether this affects Tor disproportionately or not is a minor point.
As a site operator you basically must use Cloudflare and highly aggressive policies regarding Tor.
I like Tor as a concept, the problem is that there is no way to stop people massively abusing Tor.
Whatever you want - nazi/hate speech, swatting, trolling, DDoS (by hitting expensive render paths or by forcing cache bypasses) - operate any kind of site with user interactions and you will get messed around with by mostly Tor-using scum, since trolls have understood by now that exposing their real IP will lead to them being v&d.
I don't like Cloudflare as a SPOF for half the internet, but for now they seem to be the only one able to reduce the impact of crap you have to deal with as a site op to a manageable level.
As a site operator, I have Tor (T1) whitelisted on all of my CloudFlare websites and have no additional spam problems or anything else. CloudFlare doesn't even catch all of my malicious traffic (spammers, mostly) -- even StopForumSpam doesn't do a 100% perfect job.
Counter-anecdote: I run a fairly small community and deal with an absurd amount of problems (spam, child pornography, threats, attempted attacks) sourcing from Tor and VPNs like PrivateInternetAccess.
I'm very glad CF exists. It certainly doesn't catch all bad traffic in my case either, but it helps. I don't think any cloud security service can ever stop all spam or bot activity.
I love CloudFlare, but understand the concern some people have.
With the amount of sites they have behind them, and the amount of visibility and data they can extract, it is actually a bit scary to think what will happen if they get compromised or "do evil".
But so far, they seem to take the open and ethical approach and actually follow the "do no evil" mantra. CloudFlare is another Google in the making and I hope they keep like that.
I work for a hosting company. Can confirm that CloudFlare's posts help us building our own defenses and I'm grateful that they're sharing their knowledge.
Calling this "the elephant in the room" is very insightful. It surprises me that AWS is so often recommended as the best choice for new businesses without large capital. AWS automatically scalable infrastructure changes DoS outcomes from 'your server is down due to DoS', to 'your company is out of business due to DoS' (and for small company it doesn't even need to be a large DDoS, shell script on a single machine with decent link can generate substantial AWS cost).
I've seen people state, in comments here and in blog posts, about DDoS attacks and also the resulting charge from Amazon - but in every case that I've seen the person goes on to say 'after talking with Amazon they dropped this charge'. Do you have any examples of companies or individuals being help liable by Amazon for these costs? Totally prepared to accept I'm wrong, just going on my experience so far.
> but in every case that I've seen the person goes on to say 'after talking with Amazon they dropped this charge'.
Negotiating the bill afterwards is an experience most people want to avoid, I think. I guess many would prefer a service where they don't receive the huge bill to begin with, even with other downsides.
Calling a reverse proxy company "MITM" seems kind of silly to me. It's not an attack, it's a service provider doing what their customers ask them (pay them) to do.
I know what it stands for and it is the name of an attack. Try Googling "man in the middle" and see what all the results say. It is not a neutral term.
When a service provider decrypts and filters traffic for you, it's usually just called decrypting and filtering traffic.
It's amazing how you can redeclare the meaning of words and distort reality just to fit your narrative.
Stripping encryption implies that there was, at some point, some sort of encryption between the browser and the website itself, which there never ever was. In the Cloudflare architecture the browser never ever interacts with the website, always with the Cloudflare servers. When you've got the small lock, it never implied that you were connecting to the website, but always that you were connecting to what lies behind the domain name, which is the Cloudflare servers. As an additional point, the website owner has always agreed to use this architecture; they're not somehow victim of some kind of attack from a spooky middleman. They ask (heck, some of them even pay) for it. It is part of the way they want to serve content.
You should tone down the sentimental analysis and keep to the facts. Cloudflare is a reverse proxy service the website owner uses; it is part of their infrastructure.
The name on the certificate is not Cloudflare’s. It’s origin’s. There’s no legitimate mechanism for reverse proxies to operate HTTPs sites, other than to act as a dumb pipe. Calling it MITM is therefore entirely appropriate. Whether this protocol abuse is sanctioned by origin or not is entirely beside the point.
Simply decrypting and re-encrypting web traffic in an intentional and authorized way is not MITM and should not called that.
So, why do people call CloudFlare MITM?
1) They honestly don't know that MITM is just the name for an attack, not a neutral term.
2) They think any mid-route decryption/re-encryption is bad and dangerous.
3) They are mad about some aspect of CloudFlare's service, and call it MITM to make it sound worse to everyone else.
Folks in the 1 and 2 camp just need to get educated. Folks in the 3 camp might have good points on the substance, but they are contributing to the misunderstandings of 1 and 2.
Assumption: End-to-end encryption to the name on the certificate.
Reality: Someone else (Cloudflare) claims to be the name on the certificate, whereas in reality they’re not.
That’s the textbook definition of MITM attack, whether you like it or not.
Of course all CDNs work like that. No one’s claiming otherwise. Cloudflare is just bringing all the ways internet is fundamentally broken to the forefront. Their unapologetic and extensive abuse of those design flaws (captcha pages) are a constant reminder of just how a hopeless mess the internet is.
And where is the website hosted? If you're on AWS, guess what, Amazon is decrypting the traffic, not the website owner.
Calling Cloudflare MITM but not Amazon is just demonstrating that you haven't thought carefully about how professional service providers work together to deliver a website.
The certificate authenticates that you are seeing the website that the website owner intended you to see. If the website owner hires Cloudflare as a professional service provider, that premise is not violated!
The difference is Cloudflare is perfectly content to forward the backbone traffic unencrypted over public internet. In fact, they advertise this malfeature.
Anyway as I stated before the bar for calling it MITM is simply the intended behaviour of the protocol (HTTPS). Which is very much violated. Whatever negative or otherwise connotations the term has doesn’t change the fact that it is technically correct. Really all I’m arguing here. There’s plenty of better reasons to hate Cloudflare anyway.
> The difference is Cloudflare is perfectly content to forward the backbone traffic unencrypted over public internet. In fact, they advertise this malfeature.
This puts you into segment #3 in my list above. And I agree with you: the CloudFlare "flexible" SSL that sends last-leg traffic in the clear over the open Internet is a terrible idea.
But while that terrible idea opens up the potential for MITM attacks, it is not, itself, a MITM because the website owner has authorized CloudFlare to do that.
And of course there are levels of CloudFlare service that do properly re-encrypt the last leg to the origin.
> Anyway as I stated before the bar for calling it MITM is simply the intended behaviour of the protocol (HTTPS). Which is very much violated. Whatever negative or otherwise connotations the term has doesn’t change the fact that it is technically correct. Really all I’m arguing here.
You need to understand that you are not correct on the technicality. None of the protocols used in HTTPS are violated by decrypting or encrypting a session using the proper keys, even if it happens several times along the way.
Not every bad idea is a technical protocol violation. Not every bad idea is equivalent to an attack.
Abusing the term "MITM" makes it harder to talk with specificity about actual MITM risks. If CloudFlare itself is always a "MITM", then how do we explain why their Flexible SSL service is riskier than their Full/Strict SSL service? The former makes it easier for some mid-point to impersonate or alter the origin website. But now what do we call that? An "actual" MITM? A "real" MITM?
Let's just leave MITM to mean an attack. It's a lot clearer that way.
I think not having explicit support for proxies in HTTPs is a fundamental design mistake, so seeing how you’re keen on putting me in a camp, I’d like to volunteer for that one.
The term MITM is appropriate because it shows the whole thing is a charade. If handing off secret keys to third parties with absolutely no transparency past the first hop is the standard modus operandi then the entire protocol is a big fat joke. The only purpose, as far as I can tell, is to further centralize the web. First it was ICANN with domain registration, now it’s certificate authorities. Soon you’ll need a special permission from government to host a website.
HTTP(s) needs to be burned to the ground and replaced with something better. Liberty and freedom are at stake. How’s that for FUD.
I'm from Germany but Online.net is the way better alternative to Hetzner for me. Server grade hardware, cheaper, no compromises, internal network, etc. etc. I think they even hired a lot new support people recently so you get an answer pretty fast. Hetzner always has this "cheap feeling" even though the hardware looks good on paper.
You can get server grade hardware with the PX line at Hetzner, which is a bit more expensive of course. And an internal network is possible as well from what I read, though you have to pay for the Flexi pack because it counts as a modification of your server.
The internal network is a joke, it only works if you have all of your servers in one rack and you need to rent a switch to connect all of them. Online.net has a real RPN where you can add SAN storage to it etc etc.
The PX line is a Fujitsu Desktop PC with a server CPU and ECC Ram. For me that's a compromise I don't really like.
Also the network is a bit shitty since they don't route through DTAG (largest german carrier), you have to pay a premium so that they activate it for you, and that's only one thing that feels weird about Hetzner. Of course I understand the reasoning behind it, but still, why should I choose Hetzner if I can get a way better solution from Online.net?
If you need xx TB storage servers then you will not find any cheaper hardware + storage + network option in EU than in hetzner. I have storage servers there.
Yep, true. I use a used dedicated server for backup with FreeBSD w/ ZFS and 2 3 TB disks in mirrored ZFS. I pay 36 EUR/m for this setup. I think you must be lucky to get a similar setup for the same price from online.net.
> Also the network is a bit shitty since they don't route through DTAG (largest german carrier)
Because DTAG is a bunch of <insert choice swear word here>. What they did and continue to do to site/net ops, is amounting to extortion. It's good that Hetzner stands up to Telekom, lots of other net ops simply pay the extortion fee.
The main reason why we migrated our servers from Hetzner to OVH is Hetzner's hard drive policy. Basically, if one of your drives fails, they will swap it for… another failing drive. You will keep getting various refurbished or re-tested drives. Basically, if it passes their test, it is considered a good drive, even if SMART shows a history of errors.
Obviously, these drives do not last very long, so you end up with hardware that fails more and more often as time passes. Makes you look bad in front of customers.
I guess it's a matter of pricing — Hetzner is relatively cheap, and you get what you pay for. But you should know what you're getting.
You can pay around €50 to get a new drive - and you can skip the proof that the drive failed or is failing. For us we always choose this option as soon as we see SMART problems arising. Even with occasional drive failures the overall costs are so much lower than cloud hosting or self-owned hardware.
I'm in the UK and have used Hetzner for servers for a few years. I've found them to be extremely responsive for support and remarkably good value. (Also, not being locked in to a 6-12 month contract is great.)
But I'll check out online.net, so thanks for that ;)
Have you been using online.net for long? Their offer looks interesting except for availability rate 99.5%. That's almost 4 hours of downtime every month. within availability range.
I have couple of dedicated servers there. No downtime. But i had problems with support when one of hard drives was EOL and was causing problems. They launched with Live CD and said that everything is fine, i needed to show them logs from dedicated hp raid software and after that they replaced the hard drive. Overall besides that one incident i had no problems at all.
Yes, for several years. I am also using Scaleway (their bare metal cloud) which is dirt cheap, you get a 8 Core Atom (4000 Passmark score) with 32 GB Ram and a 250 GB direct attached SSD for 22 EUR / month.
You can get more uptime (99.95%), more IPs, better and faster support with their business service (35 EUR / month).
They have very competitive pricing and I haven't had major issues with them, polite and fast support as well.
The only thing that rubbed me the wrong way is that they downright refuse to delete your credit card from their system when you terminate your (fully paid) account.
But they are not the only ones (not that this makes it good)... Linode did the same to me, I asked kindly after their latest data breach and they refused to do so.
Checking billing I've had my longest server there since 2014-06-02 so just over 2 years.
In that time I've had* one network issue < 1 hr and one instance where the server rebooted unexpectedly. Any other downtime has been either on purpose by me or accidentally by my hand. I can't even recall any scheduled maintenance that's taken my server offline.
* That I've noticed in my elsewhere-hosted monitoring.
My only complaint if I can even call it that is that the price of the server doesn't keep up with the specs they're selling - i.e. if you can get a server with <x> resources for cheaper than my server with <x-1> resources my price should come down to avoid me churning it). Having said that I don't know anywhere that actually does that and for reliability reasons it's probably better to cycle machines out when they get old enough for it to be worth the savings.
So they're willing to provide a SLA with 4 hrs of downtime every month. This does not however mean that you will have 4 hrs of down-time per mont - it's a managed risk.
Also, depending on the nature of your application, you can often get away with that kind of downtime. Something trivial (yet extremely popular) like Twitter had lots of down-time and it made the users love the service even more.
The claim that downtime "made the users love the service even more" is probably a stretch, but Twitter's error page became a bit of a phenomenon in its own right for a while:
What caught my eye is, if you scroll down long enough on dedicated servers page [1], you will eventually see GaaS - "Geek As A Service". This made me giggle far more than I am wiling to admit :)
I just compared both services to find a new VPS provider and decided in favour of Hetzner. Why do you think online.net looks better?
My main problem with online was the poor IOPS performance due to non-local SSDs. But maybe my test was just on the wrong node?
Had a few machines on hetzner for years and rarely had issues. The few issues i had with them i got fast replies.
I did try online.net few years ago and had a lot of network issues (it wasn't stable, kinda like when you have a misconfigured vps and your neighbour is eating all the bandwidth at times). Have they improved on that?
Looked pretty good untill I saw the €30/mo for windows licensing :( I pay €7,50 for the same windows license @ TransIP; wonder why there is such a big difference (Windows 2012 R2 Standard)
They allow you to install your own windows license to avoid that recurring license fee. We use a (non-public) server there for internal development and use an MSDN license on that. It involved a one time 25€ fee for their involvement in the setup process.
In some cases cheaper. Often lower setup fees. You can choose to have your setup fee split over your first 6 months.
On the other hand their support is not great. I've heard it's all PC-grade hardware rather than server-grade. They complain Canonical tried to charge them out the backside for Ubuntu licensing when in reality it'd be nothing if they'd stop using a modified, unsupported and un-maintained Ubuntu derivative.
Hit or miss really, depends what you're after and what you're personally ok with.
My personal experience with Hetzner is also mixed, two of the four servers we had had disk failures a few months into the contract. Luckily you are only bound for a month. The other two servers on the other hand are great - a good price for powerful hardware and fast support.
I've spent a lot of time pricing low end hardware and online.net is currently the winner for cores and ram per dollar. Unfortunately for me their servers are not on the same continent as my customers.
But whats their policy on DDoS? Since this is the topic you are posting this on just saying Hetzner "feels cheap" doesn't make that any less important.
They don't have good reputation but I am Hetzner customer. I have run Windows Server and some smaller Linux boxes more than 2 years and I haven't had any problems. What I have understood from comments that you need to take care of everything yourself but that is really the same case in EC2 and Azure in the end of the day.
Wow thanks for the input. I am wondering if they are seeing an uptick in business for compliance reasons which is one of the reason I am asking actually.
We run a SaaS business there on ~25 servers with Hetzner. Besides small issues we are pretty happy. Feel free to email me if you want to know more details.
Funnily I saw a thread on the customers forum at forum.hetzner.de today about why Hetzner does not have a better reputation social-media wise. Seems not to be on their priority list. There is even an English subforum, you may try to register there and ask for experience by other customers.
Social media is a wet rag in Germany: few companies see the point to invest with the relative low user engagement. It's typically seen as a distraction and/or too compromising on privacy.
There is also the 'German' view of customer service to consider (i.e. the customer is not always right).
They target their offer a bit differently than other providers and this puts off some people, who don't appreciate the tradeoffs.
The details and tradeoffs are a bit too many to list here, but basically when they can cut corners to lower prices, while still keeping server-grade performance and uptime, they do not hesitate to do it.
It's also a bare metal / every server for itself / full DIY offering, without expensive gear such as SAN, blade servers and such. So if you're not prepared to roll your own virtualization, failover, backup, provisioning, etc solutions, you are better off going somewhere else.
That being said, the network service and server specs you get per your buck are second to none. The support is also adequate, if you understand the limitations of their offering.
Source: been a customer for 10+ years, currently have 10 servers with LXC and ZFS-on-Linux, hosting 100s of websites, mail domains, and custom applications, including a full backup solution based on ZFS send / recv. Would cost me a fortune on any other provider.
Hetzner is solid but you have to understand what you're paying for, because that's what you'll get.
They'll answer competently. I sent a spam complain there once and got a sensible reply from the CEO.
But if you buy a product where they supply hardware and you do all the root work yourself, and then ask them to help you recover after a file system problem, they'll brush you off and you might consider their response rude.
In 4 years we had network issues < 1 hour, twice. One was a real issue, the other a planned downtime since they switched network gear.
One time we had to switch a bad hard drive. From support ticket to new hard drive and rebooted machine approx. 20-30 minutes. On a Friday evening at 2030.
Edit: be aware this is a bare metal server. You have to do everything yourself.(monitoring, raid integrity, smartd). And do not forget to request a remote KVM console(LARA), if you plan to reboot and you are not sure the kernel will boot.
Solid. I've been running my private servers with them for years now. Had problems with company work, specifically with them replacing failed hard drives with "shows errors, but barely-passed-testing" hard drives (see my other comment), but you don't normally hit that problem unless you have multiple physical servers.
Having servers with Hetzner for the last 15 years I'm a happy customer.
Caveat: (Root servers) They won't help you find problems or solve problems. They don't monitor your hardware. If you've got hacked and send spam, they close you down. If you have harddrive problems, you need to show them otherwise they will not easily swap drives. It's bare bone, though if they are responsible for the problems, their support is fast when notified. They are also helpful, e.g. when moving servers to a different data center, keep old servers so you can migrate to new ones etc.
We used to run some basic infrastructure on Hetzner. The support was appalling – any requests for help were swiftly met with short answers such as "Unfortunately we can not help you here". That makes me question how responsive and understanding Hetzner's staff will be in case of an on-going DDoS that their automated systems are unable to detect and take care of.
What sort of scenario would benefit from the announced DDoS protection? People are surely not running websites on commodity hardware.
What type of support issues? I had servers with hetzner for years and I never had any problems, which also means I never contacted support, so I can't say.
I'm trying to figure out if you were expecting support for things that most dedicated, colo and cloud providers wouldn't support (OS updates, install nginx, ....) or if these were clearly provider issues (hardware and network).
> People are surely not running websites on commodity hardware.
Huh? That's been the growing trend for...decades? It feels like the two places where pure server-grade equipment is used, high end dedicated hosting and collocation, are losing market. I'd go as far as to say high-end dedicated hosting is flat out dying (yay!).
It didn't come across in my comment – my amazement was regarding WordPress, etc. websites being run on commodity hardware without proper backups. E.g., get that ludicrous 128GB machine and run Apache on it with a single website for years. What could possibly go wrong?
The most recent issue we had with Hetzner was when we requested a LARA remote console to connect to a failing machine. The standard keyboard on the remote is German and much of the non-Latin keys are remapped. It was impossible to log in to a shell. All we got from support was "This is a standard Generic 104 key layout". That was it.
Before that Hetzner rebooted a whole raft of machines and on one of them this corrupted ext2. We asked what prompted the reboot (not bothered by the dead server) and we got (as above) "Unfortunately we can not help you here".
There were many more incidents before that and at the end it was just too much to swallow so we moved away.
Just out of interest, what's your use case for Hetzner?
A central location to take writes. Writes to this location go into a database and a durable, at-least-once, queue (writing to the queue is as important as writing to the database).
Then you put your API servers in different geographies (and preferably different hosting vendors). They listen to the queue and update their own storage (which could itself be a full relational database).
Edges aren't caching layer, since all the data is expected to be there (so, no miss). But certainly some of the data could be loaded in this way. The edges can also take some writes and send it back to "central", but you have to not care too much about the consistency (and maybe even accuracy) of that specific data.
Throw a latency-aware DNS on top of it (anycast, geo) with a short TTL and good health checks and you don't care too too much about uptime, allowing you to focus on raw price and performance.
Ironically, the only time I remember a serious outage (at a single location) was during a DDoS against a specific location at a much more expensive provider (not-rackspace-but-the-other-one-you're-thinking-about) which, of course, null routed us.
That'd be somewhat worrying, DDoS protection shouldn't be dropping non DDoS packets. If the chinese bruteforcers can't get to your server there's a very good chance your users can't either.
I always liked Hetzner and I have a few dedicated servers there. Their service was always fast and very competent. However they are not soo cheap compared to other hosters anymore.
We are switching away from hetzner managed server as we had ~30 hours of downtime this sunday. They suddenly swapped our valid ssl certificate with a self signed one. They offered no immediate support (i called them three times) and open tickets were left unanswered for another ~20 hours.
As i ranted about their customer service on twitter, they finally took care of it. So please stay away from them.
Free? How about mandatory? I hate blackboxes in my communication channel that decide which communication is acceptable and which isn't. Plus all the security vulnerabilities and other bugs that the complexity of such systems probably brings with it, making it harder and harder to debug network problems.
And all that is especially true in the case of Hetzner, who have repeatedly demonstrated how to build hilariously broken networks. Even ignoring that they were vulnerable to ARP spoofing for a long time and other things that have since been fixed, just some blunders of their current offerings: virtual servers behind v4 NAT (yes, you can't make that shit up ...) and IPv6 assignments of a single /64, even for dedicated machines (with an option to extend it to a /56 for money). So, if their new blackbox screws something up, what are the chances that they will care?
Can you actually use that traffic though? I can run AWS bandwidth at max for the entire month and (assuming I'm not spamming or whatever) I just get a bigger bill.
With places like Hetzner you may be shut down for abusing their services if you use that limit too frequently.
nah, wrong. If you go over the allowed bandwith, you can get either limited to 10Mbit/s or pay 1,37€ per TB. In comparison, this is pretty cheap, but that is about waht you pay, when you buy transit.
Source: https://wiki.hetzner.de/index.php/Traffic/en
Peering has recurring costs too, it's just not going to the other peer (generally), the recurring costs are for things like space at the exchange, exchange fees for dedicated circuits to the peer, etc.
I see great future for companies like Hetzner which are providing dedicated hardware with reasonable prices. While Amazon and Azure are making the live of virtual server providers difficult, I believe it is harder for them to compete with companies like Hetzner without eating their own business.
What these companies need are just additional services and nice management tools which make the whole dedicated server and network stuff look more like what we see on AWS or Azure.
My assumption is that at some point more people will come to the conclusion that part of their workloads such actually run on cheap dedicated hardware. On Hetzner €60/month buys you 4 cores, 32GB, 480GB SSD, 30TB traffic. Compare that to Azure A2, which at €64/month gives you 2 cores, 3.5GB, 60GB.
I used Hetzner, they have one of the worst support teams out there. Everything is next to unlimited and blah blah blah but the reality is really painfull for their customers including myself as a former one.
I wouldn't buy DDoS protection from Hetzner, not to talk about using it for free. There is no such things as free ddos protection which means tons of free bandwidth, this cheap marketing.
Hetzner announced this a couple weeks ago on their German language forums. What's concerning is they still have not stated their filtering capacity and the bandwidth and throughput protection levels being provided. If it's only 5 gbit and 500k PPS of filtering it won't be all that helpful.
online.net has serious bios problems. their ddos protection is a joke. yes they have arbor but they are not able to use it right. unmetered isn't unmetered ! they will kick you if you use to much traffic. their connection to amsix was very bad for weeks, its now good but they first oversell the network and do the upgrades to late. look on lowendtalk how bad they are
If I remember correctly, didn't this host have their entire address space banned from a couple IRC networks because their support wouldn't act on abuse reports?
This is what I expect from any serious hosting company know: DDoS protection for every customer as part of the service. It's part the hosting duties and should not be offloaded to any third party company, say Cloudflare.
That's a great move by Hetzner. Glad to see that OVH is not the only main player doing it anymore.
The only issue with both is that they don't handle l7 DDoS, which seems to be getting more common. I also don't like that they leverage TCP rst's for syn floods, but I guess thats better than going down.
But so far, for l7 attacks you still need ddos mitigation strategies or something like CloudFlare.com or https://sucuri.net in front of your site.
I'd have to say that webtropia.com or even myloc.de great service great value myloc has some real nice bells and whistles. webtropia has great vps and the dedi servers too. I'm at 2 locations east side USA and Deutschland. Check it out, I'm happy.
Does it mean Hetzner won't pull the cable off your server now if/when it's under heavy DDoS? I read horror stories about customers being attacked and Hetzner disabling their servers. Never experienced it myself, but I host my playground server with them and chose OVH for a more serious project due to DDoS policies..
I wish that Hetzner would provide me with DDoS protection from THEIR users. I've blacklisted their entire IP space for several different customers because of the relentless onslaught of malicious and aggressive attacks originating from their networks. They have never responded to a single abuse notification.
174 comments
[ 2.7 ms ] story [ 241 ms ] threadCould you point to a Hetzner post where this has happened? (so I can reevaluate if I might move to OVH).
I'd not put too much stock in this, Hetzner isn't perfect but almost a full day of unscheduled downtime does not match my experience.
How do you propose to "back up" such a claim after 4-5 years without the Internet archive (which was mentioned there before the GP changed his post in response to your mud slinging)?
Counter experience: 4 years with Hetzner from small VPS to huge machines and based on traceroute servers in different sections/buildings of their datacenters. No hardware issues or hourlong downtimes you describe (other than myself messing up software RAID configuration).
Nowhere did the original comment mention "otherwise bad experiences", in fact it mentioned the good value that you get for the price.
The "insults" were clearly directed at the downvoters, thus at a small but nasty subset of HN users.
The more providers offer something like this, the merrier. I understand that this isn't a layer 7 solution, but that has it's downsides as well - Cloudflare (or any other reverse proxy) will MITM all your TLS traffic, for example.
It's also time to address the elephant in the room: AWS. "Oops, you got DDOS'ed? Here, have a $50k invoice"
CloudFlare also regularly speaks about attacks and mitigations, therefore is helping the community to build better defences. Other providers stay shy and never disclose their magic. We believe DDoS is an internet wide problem and one of the ways to solve it is to spread the mitigation know how.
Examples:
- DNS attacks https://www.youtube.com/watch?v=UcAygzNSxlI&t=2h13m20s
- Iptables is great https://www.youtube.com/watch?v=pCVTEx1ouyk
- Our DDoS mitigation pipeline https://www.youtube.com/watch?v=XiK4643YdOk
- BPF for DNS https://blog.cloudflare.com/introducing-the-bpf-tools/
- BPF for SYN https://blog.cloudflare.com/introducing-the-p0f-bpf-compiler...
- Kernel bypass with netmap https://blog.cloudflare.com/single-rx-queue-kernel-bypass-wi...
- NTP attacks https://blog.cloudflare.com/technical-details-behind-a-400gb...
- DNS amplification https://blog.cloudflare.com/deep-inside-a-dns-amplification-...
- Recent attack trends https://blog.cloudflare.com/a-winter-of-400gbps-weekend-ddos...
- L7 attack with ad networks https://blog.cloudflare.com/mobile-ad-networks-as-ddos-vecto...
Please do help us fix this. Report issues, help us understand when we have incorrect IP reputation. Help us find captcha accessibility problems. And maybe - join the team to actually code the fix.
We went through this in the 90s where a few people were upset that they couldn't send email directly from their dialup connection, because they were still thinking of the world as it was in 1993 before spam became so prevalent and anyone who ran a mail server was constantly trying to deal with thousands of dialup IPs trying to deliver spam. What actually worked was that people changed the way they worked to use things like authenticated SMTP relays so you could simply block entire dialup ranges rather than try to come up with a spam-blocking AI.
The browsing system could be improved by something like a CloudFlare login system or long-term persistent authentication storage so you'd only see a CAPTCHA once a week. Unfortunately, most of the complaints come from very pro-anonymity users – which is a legitimate position but also means that it's a community which is going to be significantly more likely than average to have things like cookie & JavaScript blocking, so the complaints would simply shift to “I shouldn't have to create an account!” or “Why can't I disable JavaScript and cookies for your site!?!”
CloudFlare (and everyone else for that matter) should stop assigning "reputation" to IP addresses. An IP address is a network location, think of it as a temporary storage box which could be occupied by anyone and anything. We should move beyond coloring boxes.
Also, I think the captcha solution is better than what most administrators do, which is to block those IPs outright.
Cloudflare's job is to block potential attacks, and blocking or CAPTCHA-restricting a subnet tied to a large number of attacks against their infrastructure seems reasonable.
Either switch VPN providers, or deal with the CAPTCHAs.
* Or any other reason you're using it.
Edit: If I'm wrong can you please reply explaining why? I'm having an emotional rollercoaster of up and downvotes here! Please do feel free to correct me if I'm wrong, we all need learnin'
There's not much to be done about this otherwise - a Tor user is sharing a network with a significantly higher than usual amount of the bad elements of the internet.
Oh and don’t forget they’ll even put the captchas on subdomains, like img.domain.tld. Go visit stackoverflow via VPN/TOR and watch how the site has no styling/images even if you do their stupid captcha on stackoverflow.com. They’ll still serve you another one on static{1-50}.stackoverflow.com. Fun.
There's a very good reason why Cloudflare does this. Cloudflare isn't "punishing" anybody, their job is to protect the people that use their service. Tor (or any other open proxy) is a massive source of bogus and/or abusive traffic.
It wouldn’t be so bad, except Cloudflare doesn’t have any idea what they’re doing. Like the static.domain.tld issue I’ve mentioned—not even documented. This breaks websites in a very, very, bad way.
Next we have the lovely security setting called “Essentially-off.” Which does absolutely nothing to the captcha system. Apparently “essentially” means “not really,” according to Cloudflare.
Also forget VPN/Tor. I’m getting consistently captchad by Cloudflare just visiting mostly american-only websites from residential networks in Europe.
They’re either incompetent or just plain malicious. Right now I’m swaying towards the latter :/
Nobody is saying "hey this guy has privacy, lets make this experience a pain in the arse". They're saying "Man this set of specific IPs are really hammering my system looking for WordPress exploits and we're not even running that".
To treat Tor better than regular traffic would be to discriminate against Tor, which would invite more abuse of Tor.
Man just read up I'm essentially just summarising this anyway: https://blog.cloudflare.com/the-trouble-with-tor/
[1] https://luminati.io/
It makes perfect sense to add additional verification challenges for Tor users.
Also, anecdotal, but I work in infosec for a large US company and see a lot of traffic and read a lot of logs every day. The majority of all spam, fraud, and scan activity comes from Tor, or consumer-advertised VPN services, or various European VPSs. Residential ISPs (excluding Chinese and Russian ones) make up almost no part of it, relative to those others.
I'm pretty sure Cloudflare's challenge criteria is sourced from real data they've generated from traffic they've seen. They aren't trying to single out Tor users.
I’m actually perfectly fine with that. The issue is how Cloudflare does it. They should just offer something like X-Cloudflare-Fraud-Score header or even <x-cloudflare-protected-frame /> tag (they’re already doing SSI), but instead they just nuke the hell out of everything.
The way they do ‘protection’ is openly hostile to open web, or even neutral web. Whether this affects Tor disproportionately or not is a minor point.
I like Tor as a concept, the problem is that there is no way to stop people massively abusing Tor.
Whatever you want - nazi/hate speech, swatting, trolling, DDoS (by hitting expensive render paths or by forcing cache bypasses) - operate any kind of site with user interactions and you will get messed around with by mostly Tor-using scum, since trolls have understood by now that exposing their real IP will lead to them being v&d.
I don't like Cloudflare as a SPOF for half the internet, but for now they seem to be the only one able to reduce the impact of crap you have to deal with as a site op to a manageable level.
Trolls are destroying the Internet.
I'm very glad CF exists. It certainly doesn't catch all bad traffic in my case either, but it helps. I don't think any cloud security service can ever stop all spam or bot activity.
{blinking text|viruses|spam|IE6|blackouts|myspace|visual basic|2400bps|java|javascript|EMP}
rumors of its imminent demise might be exaggerated.. ;)
With the amount of sites they have behind them, and the amount of visibility and data they can extract, it is actually a bit scary to think what will happen if they get compromised or "do evil".
But so far, they seem to take the open and ethical approach and actually follow the "do no evil" mantra. CloudFlare is another Google in the making and I hope they keep like that.
thanks,
So, thank you!
Negotiating the bill afterwards is an experience most people want to avoid, I think. I guess many would prefer a service where they don't receive the huge bill to begin with, even with other downsides.
When a service provider decrypts and filters traffic for you, it's usually just called decrypting and filtering traffic.
Stripping encryption implies that there was, at some point, some sort of encryption between the browser and the website itself, which there never ever was. In the Cloudflare architecture the browser never ever interacts with the website, always with the Cloudflare servers. When you've got the small lock, it never implied that you were connecting to the website, but always that you were connecting to what lies behind the domain name, which is the Cloudflare servers. As an additional point, the website owner has always agreed to use this architecture; they're not somehow victim of some kind of attack from a spooky middleman. They ask (heck, some of them even pay) for it. It is part of the way they want to serve content.
You should tone down the sentimental analysis and keep to the facts. Cloudflare is a reverse proxy service the website owner uses; it is part of their infrastructure.
Indeed.
It's almost as if metaphorically speaking they were standing there between them... like a man... in the middle...
So, why do people call CloudFlare MITM?
1) They honestly don't know that MITM is just the name for an attack, not a neutral term.
2) They think any mid-route decryption/re-encryption is bad and dangerous.
3) They are mad about some aspect of CloudFlare's service, and call it MITM to make it sound worse to everyone else.
Folks in the 1 and 2 camp just need to get educated. Folks in the 3 camp might have good points on the substance, but they are contributing to the misunderstandings of 1 and 2.
Reality: Someone else (Cloudflare) claims to be the name on the certificate, whereas in reality they’re not.
That’s the textbook definition of MITM attack, whether you like it or not.
Of course all CDNs work like that. No one’s claiming otherwise. Cloudflare is just bringing all the ways internet is fundamentally broken to the forefront. Their unapologetic and extensive abuse of those design flaws (captcha pages) are a constant reminder of just how a hopeless mess the internet is.
Calling Cloudflare MITM but not Amazon is just demonstrating that you haven't thought carefully about how professional service providers work together to deliver a website.
The certificate authenticates that you are seeing the website that the website owner intended you to see. If the website owner hires Cloudflare as a professional service provider, that premise is not violated!
Anyway as I stated before the bar for calling it MITM is simply the intended behaviour of the protocol (HTTPS). Which is very much violated. Whatever negative or otherwise connotations the term has doesn’t change the fact that it is technically correct. Really all I’m arguing here. There’s plenty of better reasons to hate Cloudflare anyway.
This puts you into segment #3 in my list above. And I agree with you: the CloudFlare "flexible" SSL that sends last-leg traffic in the clear over the open Internet is a terrible idea.
But while that terrible idea opens up the potential for MITM attacks, it is not, itself, a MITM because the website owner has authorized CloudFlare to do that.
And of course there are levels of CloudFlare service that do properly re-encrypt the last leg to the origin.
> Anyway as I stated before the bar for calling it MITM is simply the intended behaviour of the protocol (HTTPS). Which is very much violated. Whatever negative or otherwise connotations the term has doesn’t change the fact that it is technically correct. Really all I’m arguing here.
You need to understand that you are not correct on the technicality. None of the protocols used in HTTPS are violated by decrypting or encrypting a session using the proper keys, even if it happens several times along the way.
Not every bad idea is a technical protocol violation. Not every bad idea is equivalent to an attack.
Abusing the term "MITM" makes it harder to talk with specificity about actual MITM risks. If CloudFlare itself is always a "MITM", then how do we explain why their Flexible SSL service is riskier than their Full/Strict SSL service? The former makes it easier for some mid-point to impersonate or alter the origin website. But now what do we call that? An "actual" MITM? A "real" MITM?
Let's just leave MITM to mean an attack. It's a lot clearer that way.
The term MITM is appropriate because it shows the whole thing is a charade. If handing off secret keys to third parties with absolutely no transparency past the first hop is the standard modus operandi then the entire protocol is a big fat joke. The only purpose, as far as I can tell, is to further centralize the web. First it was ICANN with domain registration, now it’s certificate authorities. Soon you’ll need a special permission from government to host a website.
HTTP(s) needs to be burned to the ground and replaced with something better. Liberty and freedom are at stake. How’s that for FUD.
The PX line is a Fujitsu Desktop PC with a server CPU and ECC Ram. For me that's a compromise I don't really like.
Also the network is a bit shitty since they don't route through DTAG (largest german carrier), you have to pay a premium so that they activate it for you, and that's only one thing that feels weird about Hetzner. Of course I understand the reasoning behind it, but still, why should I choose Hetzner if I can get a way better solution from Online.net?
On the other hand, if you need 24x 1 TB SSD storage with a 10gbit RPN connection: https://pbs.twimg.com/media/CblcJ2pUUAAtbhK.jpg :-)
Because DTAG is a bunch of <insert choice swear word here>. What they did and continue to do to site/net ops, is amounting to extortion. It's good that Hetzner stands up to Telekom, lots of other net ops simply pay the extortion fee.
Obviously, these drives do not last very long, so you end up with hardware that fails more and more often as time passes. Makes you look bad in front of customers.
I guess it's a matter of pricing — Hetzner is relatively cheap, and you get what you pay for. But you should know what you're getting.
But I'll check out online.net, so thanks for that ;)
You can get more uptime (99.95%), more IPs, better and faster support with their business service (35 EUR / month).
The only thing that rubbed me the wrong way is that they downright refuse to delete your credit card from their system when you terminate your (fully paid) account.
But they are not the only ones (not that this makes it good)... Linode did the same to me, I asked kindly after their latest data breach and they refused to do so.
In that time I've had* one network issue < 1 hr and one instance where the server rebooted unexpectedly. Any other downtime has been either on purpose by me or accidentally by my hand. I can't even recall any scheduled maintenance that's taken my server offline.
* That I've noticed in my elsewhere-hosted monitoring.
My only complaint if I can even call it that is that the price of the server doesn't keep up with the specs they're selling - i.e. if you can get a server with <x> resources for cheaper than my server with <x-1> resources my price should come down to avoid me churning it). Having said that I don't know anywhere that actually does that and for reliability reasons it's probably better to cycle machines out when they get old enough for it to be worth the savings.
Also, depending on the nature of your application, you can often get away with that kind of downtime. Something trivial (yet extremely popular) like Twitter had lots of down-time and it made the users love the service even more.
http://www.theatlantic.com/technology/archive/2015/01/the-st...
I think the main lesson here is that your error messages are part of your UX and need to be treated as such.
What caught my eye is, if you scroll down long enough on dedicated servers page [1], you will eventually see GaaS - "Geek As A Service". This made me giggle far more than I am wiling to admit :)
[1]: https://www.online.net/en/dedicated-server
https://console.online.net/en/server/selfie/Al5dpTp6Awp3
Their support was responsive but ultimately unable to do anything.
Had a few machines on hetzner for years and rarely had issues. The few issues i had with them i got fast replies.
I did try online.net few years ago and had a lot of network issues (it wasn't stable, kinda like when you have a misconfigured vps and your neighbour is eating all the bandwidth at times). Have they improved on that?
How does OVH compare to the above two?
On the other hand their support is not great. I've heard it's all PC-grade hardware rather than server-grade. They complain Canonical tried to charge them out the backside for Ubuntu licensing when in reality it'd be nothing if they'd stop using a modified, unsupported and un-maintained Ubuntu derivative.
Hit or miss really, depends what you're after and what you're personally ok with.
Funnily I saw a thread on the customers forum at forum.hetzner.de today about why Hetzner does not have a better reputation social-media wise. Seems not to be on their priority list. There is even an English subforum, you may try to register there and ask for experience by other customers.
There is also the 'German' view of customer service to consider (i.e. the customer is not always right).
They target their offer a bit differently than other providers and this puts off some people, who don't appreciate the tradeoffs.
The details and tradeoffs are a bit too many to list here, but basically when they can cut corners to lower prices, while still keeping server-grade performance and uptime, they do not hesitate to do it.
It's also a bare metal / every server for itself / full DIY offering, without expensive gear such as SAN, blade servers and such. So if you're not prepared to roll your own virtualization, failover, backup, provisioning, etc solutions, you are better off going somewhere else.
That being said, the network service and server specs you get per your buck are second to none. The support is also adequate, if you understand the limitations of their offering.
Source: been a customer for 10+ years, currently have 10 servers with LXC and ZFS-on-Linux, hosting 100s of websites, mail domains, and custom applications, including a full backup solution based on ZFS send / recv. Would cost me a fortune on any other provider.
They'll answer competently. I sent a spam complain there once and got a sensible reply from the CEO.
But if you buy a product where they supply hardware and you do all the root work yourself, and then ask them to help you recover after a file system problem, they'll brush you off and you might consider their response rude.
Edit: be aware this is a bare metal server. You have to do everything yourself.(monitoring, raid integrity, smartd). And do not forget to request a remote KVM console(LARA), if you plan to reboot and you are not sure the kernel will boot.
Caveat: (Root servers) They won't help you find problems or solve problems. They don't monitor your hardware. If you've got hacked and send spam, they close you down. If you have harddrive problems, you need to show them otherwise they will not easily swap drives. It's bare bone, though if they are responsible for the problems, their support is fast when notified. They are also helpful, e.g. when moving servers to a different data center, keep old servers so you can migrate to new ones etc.
What sort of scenario would benefit from the announced DDoS protection? People are surely not running websites on commodity hardware.
I'm trying to figure out if you were expecting support for things that most dedicated, colo and cloud providers wouldn't support (OS updates, install nginx, ....) or if these were clearly provider issues (hardware and network).
Huh? That's been the growing trend for...decades? It feels like the two places where pure server-grade equipment is used, high end dedicated hosting and collocation, are losing market. I'd go as far as to say high-end dedicated hosting is flat out dying (yay!).The most recent issue we had with Hetzner was when we requested a LARA remote console to connect to a failing machine. The standard keyboard on the remote is German and much of the non-Latin keys are remapped. It was impossible to log in to a shell. All we got from support was "This is a standard Generic 104 key layout". That was it.
Before that Hetzner rebooted a whole raft of machines and on one of them this corrupted ext2. We asked what prompted the reboot (not bothered by the dead server) and we got (as above) "Unfortunately we can not help you here".
There were many more incidents before that and at the end it was just too much to swallow so we moved away.
Just out of interest, what's your use case for Hetzner?
A central location to take writes. Writes to this location go into a database and a durable, at-least-once, queue (writing to the queue is as important as writing to the database).
Then you put your API servers in different geographies (and preferably different hosting vendors). They listen to the queue and update their own storage (which could itself be a full relational database).
Edges aren't caching layer, since all the data is expected to be there (so, no miss). But certainly some of the data could be loaded in this way. The edges can also take some writes and send it back to "central", but you have to not care too much about the consistency (and maybe even accuracy) of that specific data.
Throw a latency-aware DNS on top of it (anycast, geo) with a short TTL and good health checks and you don't care too too much about uptime, allowing you to focus on raw price and performance.
Ironically, the only time I remember a serious outage (at a single location) was during a DDoS against a specific location at a much more expensive provider (not-rackspace-but-the-other-one-you're-thinking-about) which, of course, null routed us.
That actually is such a growing trend that one of the more popular NoSQL databases is called...
Cluster Of Unrealiable Commodity Hardware Database,
aka CouchDB.
If this has kicked it, I guess that might explain it.
30 hours in a 24 hours day? :O
And all that is especially true in the case of Hetzner, who have repeatedly demonstrated how to build hilariously broken networks. Even ignoring that they were vulnerable to ARP spoofing for a long time and other things that have since been fixed, just some blunders of their current offerings: virtual servers behind v4 NAT (yes, you can't make that shit up ...) and IPv6 assignments of a single /64, even for dedicated machines (with an option to extend it to a /56 for money). So, if their new blackbox screws something up, what are the chances that they will care?
AWS charges a whooping 90$ per TB, it keeps me wondering why their traffic is SO much more expensive than Hetzner's...
With places like Hetzner you may be shut down for abusing their services if you use that limit too frequently.
(Not on a recurring basis anyways -- yes, there's the one-time capital costs and such.)
What these companies need are just additional services and nice management tools which make the whole dedicated server and network stuff look more like what we see on AWS or Azure.
My assumption is that at some point more people will come to the conclusion that part of their workloads such actually run on cheap dedicated hardware. On Hetzner €60/month buys you 4 cores, 32GB, 480GB SSD, 30TB traffic. Compare that to Azure A2, which at €64/month gives you 2 cores, 3.5GB, 60GB.
The only issue with both is that they don't handle l7 DDoS, which seems to be getting more common. I also don't like that they leverage TCP rst's for syn floods, but I guess thats better than going down.
But so far, for l7 attacks you still need ddos mitigation strategies or something like CloudFlare.com or https://sucuri.net in front of your site.
thanks,
That just creates a wrong impression.