Interesting perspective, but it seems like this is just an ad for easyDNS and their "Proactive Nameservers," though I couldn't imagine a better time than the misstep of a behemoth of a competitor in this space. Not to detract from the more important discussion about the internet's dependence on Cloudflare overall.
It may be just an ad for easyDNS' Proactive Nameservers [1] product but it provides a roadmap for one possible solution to this type of problem. From a quick reading of the marketing info, the solution can be summarized as "Provision, Monitor, and Fail-Over DNS Name Servers across multiple DNS-as-a-Service providers". The question I have is whether the following constraint is artificially introduced or not:
> We must be your domain registrar for this to work...
IIRC, Netflix OSS published some tools quite some time ago to support multiple DNS providers but I don't know/remember if they tackled the availability problem. The question comes down to build vs buy and whether the solution is general enough to warrant an Open Source Software solution.
> The only requirement to use Proactive Nameservers is that we have to be your registrar, because we need to connect to the registry to update your nameserver delegation.
So I guess technically this could be achieved with an API for your domain settings.
> IIRC, Netflix OSS published some tools quite some time ago to support multiple DNS providers but I don't know/remember if they tackled the availability problem.
The classic way of doing this is AXFR (your own DNS server is a "hidden master" and the DNS providers are the slaves).
The problem is you won't be able to have redundancy at the registrar level, but that has historically at least been less of an issue.
They want to be the registrar to be able to update your NS records. But ... that's not really important nor needed (So the answer to your question yes, it's likely artificial). Just use two anycast-ed IPs/domains. (Like Cloudflare.)
The magic happens at BGP level.
I considered CF as a domain registrar, but they don't allow setting the NS records. So you must use them. (They basically use sane no-nonsense domain registration as a way to gain leads for their main product. Pretty smart actually, because it's a great high-level add-on for their main product, but they just went ahead and made that the bait for everyone.)
Anyway, ideally, if you add 2 separate sets of NS servers to your NS records then you eliminated this SPoF, great. Sure, it's your job to keep them updated, and in sync (preferably, to avoid problems like half of your users landing on a different CNAME/IP/etc).
And recursive nameservers will handle the failover.
I meant that easyDNS should handle the BGP for its clients, without requiring their clients to use them as registrars.
There's HE.net's free DNS, and though they don't explicitly advertise as, it's anycasted. (Check via https://tools.keycdn.com/ping , try 216.66.80.18 [ns5.he.net].)
easyDNS has to be the registrar because only your registrar can change your nameserver delegation with the registry. This is, in essence, the registrar's job. To maintain your domain record and info, including nameserver delegation, with the registry.
You could do it with BGP, but it is non-trivial and you need your own ASN to do that.
But you can just add multiple DNS providers yourself. I mean you can add the namservers of both easyDNS and cloudflare. EasyDNS just automates this.
In theory they could simply create a few subsidiaries, let's call them saferDNS1,2,3 and have them build completely different redundant DNS architectures, and add then add the resulting nameservers.
That said, it'd be good to see an actual domain that uses this "proactive" feature to see what easyDNS is doing.
I'm not a DNS expert, so... It's not really that simple is it? If you have multiple nameservers I thought they get equal weight, don't they?
So if you have Cloudflare + (ex:) NS1, and you're using Cloudflare for caching, you need your NS1 records to return Cloudflare proxied IPs normally, but origin IPs under failure conditions. That's a lot of infrastructure.
It also fails completely if you're relying on Cloudflare for DDoS protection and IP obfuscation because a failure means your origin IPs get exposed. That's assuming Cloudflare DNS being down means Cloudflare proxying is down too. It might not be the case, but I think you'd have to plan for it.
Then there's also Cloudflare's detection of nameservers. I haven't tried it with more than Cloudflare's nameservers set for a domain, but if your domain doesn't actively use their nameserver they'll drop your site from their system. So, at the very least, you can't use Cloudflare as a secondary DNS provider (at least the last time I checked).
I was talking about just the DNS layer, but ... you can periodically check what IP Cloudflare would return and return that. (There's also the apex-CNAME record type, ALIAS or DNAME, I don't remember right now, but PowerDNS supports it, and you can set up the resolver to use CF's NS.)
Of course CF doesn't support anything like this, but it works well (because they use quasi fixed, static anycasted IPs for the HTTP(S and TCP?) proxying/load-balancing too), even if it's hacky as hell.
If they have any active verification of nameservers, and if they disable the proxies if they detect something bad, then ... it won't work obviously :)
But technically there's nothing amazing in being a registrar of a domain. So both CF and easyDNS are just stubborn in the name of user experience (consistency).
... all in all, working around any SPoF (reliably) will usually require exponentially more resources/engineering/care.
> Proactive Nameservers is a patent-pending system that optimizes the nameserver delegation for your mission critical domain names.
That's a huge negative and I can't believe they think it's a good marketing point. I don't want a patent encumbered, non-standard solution for critical infrastructure.
> We must be your domain registrar for this to work.
So they're updating the domain record at the registry level to facilitate failover? That's the only scenario I can think of where they _need_ to be your registrar. Assuming that's the case...
I've always seen 24-48 hours quoted as the worst case wait when updating nameservers at the registry. I've never seen an explanation of how it works, what's allowed to be cached, how long it actually takes to update, etc.. How do they do it in a way that's suitable for failover? Do they have a special SLA with registries?
How would the registries handle a deluge of nameserver updates? Imagine a Cloudflare scale failure and corresponding registry updates. Would the registry servers be able to handle it?
I'd love to see a technical explanation of how their proactive nameservers system works.
If only DDOS attacks were taken seriously and their perpetrators punished accordingly (and maybe if the network had better ways of self-defense) instead of companies and websites having to fend for themselves (or having to resort to solutions like Cloudflare).
It's interesting to me that Cloudflare doesn't really have any competition with a similar business model. I suppose the free plan requires quite a lot of spending before the upgrades offset it.
Depends on what you want but a very solid free CDN and DNS option is: host your site on netlify and use dns.he.net for your nameservers.
Another good DNS option is dnsimple.com or, indeed, EasyDNS. For even more redundancy, use one provider as your domain registrar and another for your nameservers (and set short TTLs for your zones so you can re-point IPs quickly if you need to).
For the other things Cloudflare offers on their free tier, I'm not sure what good alternatives exist (there must be some, I'm just not familiar with them outside of the obvious AWS alternatives).
Edit: one caveat with above advice, I have no idea if netlify use cloudflare behind the scenes...
There are other large CDNs like Akamai. They just don't compete with Cloudflare in the consumer sector. It probably doesn't matter for them because enterprise contracts are where all the money is at.
AWS, Azure and Google Cloud all offer competitive alternatives for the small business sector. Personally, I find the actual cost of CloudFlare very difficult to reason about in advance of actual use. This is in part because it's easy to miss a bit of the a la carte model you need.
Full disclosure, I work at Google on the SRE team for Cloud CDN. If you want a credible alternative for the CDN part of Cloudflare, our product is extremely fast. We were all very sad for Cloudflare and watched the whole affair closely, as we've got a few customers that use our services alongside Cloudflare's.
> Is it realistic for a small-medium sized business to have more than one DNS provider?
Yes: as the weblog post points out, you can have EasyDNS as your master with their multiple DNS servers, and then also have (e.g.) Route53 slaved to EasyDNS and have those in addition to EasyDNS in your records.
Okay here's the thing. I'm okay with people bashing other (competing) companies when they do wrong. However, I believe it is somewhat childish and uncalled for to bash another company, because of a mistake.
First of all "I use easyDNS so I didn't notice it at all tbh" is not only a childish assertion, it's borderline a falsehood. You DO NOT offer the same services, nor the same scale. (No, VOD would not work if your VOD provider used Cloudflare's offering.)
Second of all, as some have noted in other comments, you are very welcome to get just as big as them if you can offer similar (excellent) service and similar extremely competitive pricing. Otherwise, keep working on your offer and stop going for low hanging fruit like bashing the competitor for an outage when they literally might handle 1000x your traffic, and perhaps offer 20x the services your offer.
Honestly I didn't notice the domain name, and I actually thought the author was being quite understanding by saying things like "This is inevitable and unavoidable and entirely excusable. Everybody blows up, every DNS provider in existence will experience downtime. No exceptions." and that they use Cloudflare themselves.
This is obviously subjective, but to me it didn't come across as "they suck use us" but rather pointing out the inherent flaws in this quite popular SPOF and cautioning to avoid it.
I think the root cause (which, IMO, you correctly point out) is lost on many modern developers.
For whatever reason there's this modern idea that if a company A is paying money to company B for a service, that company B will handle all the 'hard stuff' for them.
The end result is we have a lot of applications/infra built with SPOFs, in some cases known, but in many, swept under the rug and abstracted away to passing the buck in case of a large failure (i.e. major AWS/Cloudflare/Azure outages).
You also see this at times when vendors pitch internal software solutions. I've been at more than one shop where a vendor's 'silver bullet' turned into a SPOF time-bomb because nobody considered this company's solution could fail. After all, the sales presentation said it had %nines%!
"Your app will go down when half the Internet goes down" is not that big of a deal to most software companies, because:
1. no one's going to blame me if my app goes down when half the Internet is also down but they are going to blame me if my custom solution to the same thing causes an outage,
2. there's no way my custom solution is going to achieve the same uptime. AWS/Cloudflare/Azure are not perfect, but whatever I roll for myself is almost certainly going to be much less perfect.
Regarding 2, that’s exactly what the blog post was advocating for: Redundancy. They weren’t saying give up on Cloudflare (as they mentioned, they use it themselves).
They do blame you though, most people won't be aware of the real issue, when cloudflare went down, the trending things on twitter were #spotifydown and #applemusic
> However, I believe it is somewhat childish and uncalled for to bash another company, because of a mistake.
The thing that they're bashing is not the mistake, which happens to everyone. They're bashing SPOF:
> EasyDNS was unaffected because while we do use Cloudflare to soak up large DDoS attacks against our nameservers, we don’t use them across all of our nameservers. I think somewhere in my book I wrote “DNS providers have a near-pathological aversion to SPOFs” (Single Point of Failures). Maybe only we do.
Well if you are going to rant, you might want to rant and quote what they ACTUALLY said, not what you conceived in you mind what they were saying
You indicate they said " "I use easyDNS so I didn't notice it at all tbh" but NO WHERE IN THAT ARTICLE was that statement
The actual quote is
"We’re familiar with Cloudflare’s DDoS service for DNS providers, because we use it ourselves. Fortunately easyDNS was not impacted by the outage (I didn’t even notice it, tbh),"
This is a MUCH different statement than you attempt to cast out as call "childish". He is stating that the services they use of CloudFlare was not impacted by the outage
If only DDOS attacks were taken seriously and their perpetrators punished accordingly (and maybe if the network had better ways of self-defense) instead of companies and websites having to fend for themselves.
That is the problem with massive centralization even if it is market level and internally Cloudflare (or any other big fish) does decentralization/fail-over of their own. Many of these companies should have had fail-over to competitors at least for reliability.
The problem with near market monopolization, oligopoly, even the singularity, the fail-case is catastrophic and may even wipe out decentralized, diffused, dispersed, decoupled system solutions that can't make it due to so much relative size from the big fish that it squashes them along the way. The bigger the ship the longer it takes to turn.
This Cloudflare issue is like the recent Facebook SDK startup crashes where everyone has a single point of failure on Facebook SDK where people should be using or able to use the OpenGraph API directly as they need which is more robust to the app that uses it, it won't crash on startup.
In business it is a goal to centralize to grow, in nature and robust systems it is more differentiation and decentralization to survive. There will always be a push and pull between these two forces.
Systems and markets are like gardens. The garden must be maintained, new seeds planted and helped to grow from small to mid-sized, mid-sized plants the bulk of the garden, and then the larger plants need to be culled back when they get too big to not take the mid-sized and then all the resources from the new seeds/small plants. The problem is we have allowed the top end to take over the garden and when they fail they fail spectacularly. The bigger the scale the bigger they can fail.
> Many of these companies should have had fail-over to competitors at least for reliability.
How would you even set such a thing up? I fear that you might get a couple of collusionary companies that bail each other out and smaller providers might just be left out to dry…
Many DNS providers have the ability to do AXFR "zone transfers", so you can sync your records to a secondary provider and you would add a secondary set of nameservers for redundancy. Unfortunately Cloudflare doesn't offer this unless you pay for their Enterprise plan (they started offering it earlier this year).
I do love using CloudFlare for DNS, lots of great features and generally works well, but I wish they would support AXFR for the lower tiers. I've been working on a solution for this using the CloudFlare API, but we'll see how well it works out.
I think they want to price discriminate: the costs from AXFR should be minimal, as it is an old technology, very optimized and low bandwidth.
However, cloudflare decision turns cloudflare "free" offering into a free SPOF. They should extend this offer to their free users, who could then use the secondary DNS that most hosts/domain name sellers provide for free to the IP that is proxified by cloudflare. It could even be limited to the case of proxy or cloudflare DNS failure, so that cloudflare could still price discriminate (make AXFR fail, unless cloudflare is down, like a dead man's switch)
Amazing that this isn't "free" and folks have to resort to syncing records across DNS providers with proprietary, vendor-specific APIs. AXFR is standard. It's not just Cloudflare... AWS and Azure don't support it either.
Fwiw, you could use stack overflow DNS Control to manage your DNS records and upload them to many providers. Then the only thing you'd have to do is flip a line of code to fallback to the other DNS provider.
Back in the day (I haven't run my own mail server for years and years) there was a company called Secondary MX.
That's all they did. They weren't a mail host or provider, there was no UI, nothing. All they did was allow you to specify them as, well, a secondary MX, so if you were offline they'd cache your inbound email until you were back. Simple. Efficient.
The article posted actually talks about EasyDNS's implementation of exactly this.
> At easyDNS we experienced so much pain from this reality that we created a system to automate flipping DNS providers at the first sign of trouble.
> We call it Proactive Nameservers, and we’re the only company in the world doing it for some reason. Maybe this is because in order to provide a service like nameserver failover, it means a company has to admit to its customers the reality that their own nameservers may at some point, fail.
I imagine the implementation could look similar to how cell phones can use other carriers networks (for 911) when they don't have signal from their own carrier.
You needn't use your real name, of course, but for HN to be a community, users need some identity for other users to relate to. Otherwise we may as well have no usernames and no community, and that would be a different kind of forum. https://hn.algolia.com/?sort=byDate&dateRange=all&type=comme...
The question is: Can we do better? A natural monopoly is a good thing. It just means that the natural monopoly needs to build its own redundancy. Cloudflare total failure isn't common.
I wouldn't classify a natural monopoly as a good thing. It mostly signifies high barriers to entry for competitors and a market that's more susceptible to failure, as seen here.
I guess but there's not really a shortage of commercial CDNs. A market with lots of competitors but a clear winner says more about consumer preference and network (ha!) effects than it does about Cloudflare's moat.
If you have a "monopoly" which is this case just means you're the market leader by a wide margin, but in theory this could mean near total control of a market but the reason you have this total control is because your customers choose to buy from you over your competitors simply because you're better and or cheaper then this is a good thing. Customers are getting what they want. And the market leader in this situation will have a hard time abusing their position because any they'll have competitors nipping at their heels if they slip up.
I would personally phrase it pedantically as not that the natural monopoly is good but that it exists because they are good. A fine distinction and a baked in assumption admittedly that if they were to no longer be good they would no longer dominate. Technically there are some other variables like if an extended stay on top would atrophy any competitors or not and the time scale operated upon.
On a century scale individual murderers aren't a huge concern to society because they are either dead or infirm by then.
I think for the sake of a diverse community and economy, monopolies should be difficult to achieve even naturally.
But for this specific example, a robust technical redundancy doesn't stop CloudFlare from going out of business. A technology company going out of business is pretty much the norm. Incumbents are a relatively new phenomenon for technology (sans some key exceptions), and I don't think CloudFlare is an incumbent. They are an accessory, and your business would probably run without them.
Cloudflare isn't a monopoly. Cloudflare doesn't even have a particularly strong moat when compared to Fastly or Akamai. Cloudflare doesn't even have any network effects.
I must disagree, and wish I had more data on how many sites host with cloudflare because it's free.
I just checked, fastly is $50 per month, and Akamai doesn't list pricing so let's assume it's more.
Cloudflare has had network affects from integration with wordpress and cpanel too I believe for some time now.
without cloudflare your site can be taken offline by any random person willing to spend $20 for ddos sellers.
The free plan is a pretty big moat imho, especially if 'half the internet can fail..' - I doubt 10% of those sites would be using cloudflare if there was a monthly fee pushed on them.
Admittedly, my cloudflare usage is only about 30 sites, so my data point is small. The few hosting and design clients I have and their budget constraints are not indicative of 'half the internet' - but I don't believe fortune 500 and SV sites are either.
How do you do a fallback from cloudflare failing when they’re your dns provider too? Any redirection around would take too long to change wouldn’t it? They’d be back up and running before it was implemented. What’s the right approach?
I don't think cloudflare supports such a configuration.
The DNS is part of the load balancing, they serve different IPs based on location of the DNS query.
Edit: Apparently they do support a CNAME configuration if you pay for one of their business plans. That gives you the option to quickly switch away (if your TTL is low enough) but will impact performance by having to fetch the CNAME every 60 seconds.
Does cloudflare actually do geo-loadbalancing via DNS A records now? For years they only did anycast, unlike, say, Akamai, which hands out different IPs for each POP.
Actually, I'm not sure if they do any DNS geo-loadbalancing. I've seen it report different IPs from different locations at times, but that could be something else.
But I'm pretty sure they use DNS do other loadbalancing and DDoS mitigations.
For example, if a site is under attack, they can send it to different IP addresses to keep it away from other sites. Or if someone is directly targeting a cloudlflare IP with a DDoS, they can redirect all sites to other IPs and just blackhole that IP.
Where do you put your DNS servers? On-prem is going to be less reliable than cloudflare which has a 100% uptime SLA. I doubt your local ops team can compete.
True, however at AWS at least, customers are specifically told "multi-cloud doesn't allow you to fully leverage the benefits of AWS", whatever that means.
It makes sense that cloud companies are inclined to keep customers from giving money to competitors, but they way they sell it and structure services, reserved instances, and enterprise discounts is such that you basically are putting all of your eggs in one basket.
> "multi-cloud doesn't allow you to fully leverage the benefits of AWS", whatever that means.
One of the selling points of cloud providers is managed services like SQS. If you run a multi-cloud architecture, you either can't use managed services, or have to build abstraction layers on top of them (and only use the features that exist in both cloud providers' versions of the managed service).
If you want to use a managed service that only exists on AWS, then that's obviously incompatible with a fully multi-cloud architecture.
> If you run a multi-cloud architecture, you either can't use managed services, or have to build abstraction layers on top of them (and only use the features that exist in both cloud providers' versions of the managed service).
And this is, of course, why they do everything they can do discourage it. Because if you do that, not only are you not reliant on them for availability, you can switch more of your business to the other provider(s) based on current pricing, and they do not want that big time.
I thought that was something trivially obvious stated about fully leveraging the benefits - that they don't control other's clouds and thus they can't use all of the same sorts of performance or efficency boosting tricks. (People would be way more mad if they did as it would require hacking into say Azure to gain root access.) Not a domain expert but it is my interpretation.
If you make the engineering decision to go multicloud for whatever reason those are inherent trade offs you need to be aware of. They have their own agenda of course in addition to any actual fundamental "real" in bulk efficiencies that price reflects.
The problem is customers choosing to put all their eggs in one basket.
Until relatively recently absolutely none, and now almost none of the tooling allows effective multi-cloud or hybrid cloud/private.
Basically the cloud providers work very hard to prevent the commodification of their services with special incompatible service offerings, lock-in, interdependency, deep and opaque APIs for integration, and networks of training and certification that position change as a direct threat to people's job security.
Cloud providers today are basically Microsoft in the 90s.
Much as open source challenged Microsoft, I would say that the world now needs open infrastructure tooling that positions hybrid and multi-cloud as first class infrastructure architectural cases in order to displace established cloud provider hegemony. Even then we will have to fight the hardware and real estate economies of scale available to large established cloud providers.
I wrote some observations about this space based significantly on HN community comments prior to the rise of Docker a few years ago: http://stani.sh/walter/pfcts/ ... click 'original' ... the conclusions still seem timely.
> Until relatively recently, the cloud didn't exist.
Depends:
> IBM and other mainframe providers conducted this kind of business in the following two decades [after 1961], often referred to as time-sharing, offering computing power and database storage to banks and other large organizations from their worldwide data centers. To facilitate this business model, mainframe operating systems evolved to include process control facilities, security, and user metering.
[…]
> In 1998, HP set up the Utility Computing Division in Mountain View, CA, assigning former Bell Labs computer scientists to begin work on a computing power plant, incorporating multiple utilities to form a software stack. Services such as "IP billing-on-tap" were marketed. HP introduced the Utility Data Center in 2001. Sun announced the Sun Cloud service to consumers in 2000.
[…]
> In spring 2006 3tera announced its AppLogic service and later that summer Amazon launched Amazon EC2 (Elastic Compute Cloud).
Cloudflare has done really well in a short amount of time, but I'm sure Akamai is way bigger. Cloudflare is best known because they offer free/cheap tools for individuals and this post is targeted at developers. Most big enterprises I've worked with are on Akamai.
Compared to the endless content marketing Cloudflare posts [1]? It's an ad, but they're still right. That's just good content marketing (informative, relevant, and perhaps you buy something because of it).
The more people that use anything other than Cloudflare, the better. I had this conversation with people back in ~2010 about Facebook and they all ignored it. They will again, but this time the consequences of centralization will be even worse.
It won't just be one single website that goes shitty with blockages and manipulation and censorship. It won't even be just the web. When Cloudflare achieves their goal of deep packet inspection at every peering and transit point it'll be the end of the internet as we knew it and the slow transition to just another cut apart "China-net (tm)".
That is a fair point; SPoF depends on what level you're looking at. A RAID array removes the SPoF that is a single disk, but still leaves a SPoF in the RAID controller or CPU or power supply; a ceph cluster can withstand the loss of a whole rack but could still fall to certain software bugs. Likewise, "cloud" companies are internally redundant, right up to the point where they aren't. It depends on how you scope the question.
Doubly ironic since in doing this, they created a system where a protocol is the SPoF; namely nonsensical or false BGP advertisements can quickly kill the internet as a whole if done correctly.
That's not actually a contradiction. When an individual website is considered as a system, it will have multiple points where the failure of a component inhibits the system. It's possible to have both cloudflare and your database as "SPoFs" and that "single" is not meant to imply everyone only gets one.
It's absolutely true that if us-east-1 in AWS has a bad day, a significant fraction of the American digital economy will shut down. For some companies, the same is true of Azure and Google's various comparable offerings.
I read your post as skeptical. Why would you be skeptical? If you care about keeping your product up, you absolutely should have a fallback for cloudflare if you're a customer of theirs. Now, you might not care (and actually, for most folks I submit you need not care), but the folks making sure Ambulances get timely push notifications and realtime driving instructions probably care quite a bit.
it's a series of single points of failure on different levels, so, not multiple points of faiure, but much worse, single point after single point, which means the house of cards fails as often as any of them. The internet of 2020 is a monolith
This is why AWS is puts effort into blast radius reduction.
The Physalia paper is a wonderful read that explains both the justification and high-level approach to implementing said isolation [1].
Another great example is shuffle sharding [2], notably used in Route53.
The end goal of this is that aws outages would be isolated to subsets of aws customers where possible. So instead of half the internet failing it's only 10% (obviously the exact numbers depend heavily on implementation).
What's funny about this outage is I'm sure many of of us (myself included) used this window to analyze large services and determine an increase in major Cloudflare customers and presumably, revenue. Even ISPs like T-Mobile faced issues due to the Cloudflare outage! The situation has exposed just how critical Cloudflare is.
I went ahead and bought calls ahead of NET earnings next month. Cloudflare is becoming an increasingly bigger part of the internet backbone. Purely speculating here, but I wouldn't put it past AWS or another large player acquiring them soon.
> I wouldn't put it past AWS or another large player acquiring them soon.
I really hope it doesn't come to this sadly. I'd be okay with DO or somebody who isn't as massive doing a merger. Maybe they can pull resources to make each other even successful and maintaining reasonable independence.
> an increase in major Cloudflare customers and presumably, revenue. Even ISPs like T-Mobile faced issues due to the Cloudflare outage!
Careful about this methodology. Some services at my org were impacted despite not being direct CloudFlare customers. They had external dependencies that used CloudFlare.
So it's much bigger proverbial 'blast-radius' lest something happen to CloudFlare? Can you elaborate a bit on that part? I'm interested in knowing more.
On an unrelated note. Thank you so much for PortableApps.com! It was invaluable at university when studying in the library. To this day, I still use it for utilities that don't need to be installed on my desktop.
I am also grateful for PortableApps.com. Along with Scoop and Homebrew, they make using a system without root or admin privileges a really nice experience, in both Windows and Linux.
My sincere thanks to John and all the PortableApps.com contributors.
You're welcome, I'm glad it's helped and helping you! I try to keep it growing and relevant with a more synced cloud folder/work or school laptop/keep work and personal separate bent these days.
Honest question: I've never really understood the back of the napkin math of how Cloudflare functions economically, which I feel would go a long way towards my understanding of why/how they were able to become such an integral and generally positive part of the Internet.
Did they have some crazy in to get cheap bandwidth? Did they bet big on bandwidth prices falling? Did they figure something else out that nobody saw? Do they just to a tremendous job of migrating sites from free to paid plans?
Off topic, but any regrets on getting involved in content policing? It always sat with me as wrong and a disturbing precedent that an internet backbone service such as yourselves would make it their business to shut down unsavory yet legal speech.
Maybe GP is referring to 1.1.1.2 and 1.1.1.3, the DNS resolvers that filter malware and malware+adult content respectively? Both are optional alternatives to the unfiltered 1.1.1.1 DNS service if so...
Cloudflare CEO Matthew Prince spoke to CNBC upon banning Daily Stormer: “We were worried that people would say, ‘We won’t work with you anymore,’” Cloudflare CEO Matthew Prince told CNBC.
“We had to have the conversation now because at some point we’ll be a public company. We had to prompt that discussion,” said Prince, who added “we want to be ready internally by July 2018,” for a possible stock offering.
I don't know if it's how it played out, but it's easy to see how increased scrutiny could have lead to the illegal content being highlighted and them getting dropped because it was then reported (possibly many times by many parties that wanted the site shut down for other reasons).
In that case, they still would have been taken down because of the illegal content, even if at an (extremely simplistic) surface level the timing seems to indicate otherwise.
Nazism is pretty popular on HN, comes with the territory when most of the board's users are well off westerners. Doesn't hurt that the mods are way more concerned with anti-capitalist rhetoric. Not surprising considering who owns the site though.
There isn't always a line, some businesses are in the business of selling products and services and not in the business of policing content. And if some of their customers are pedophiles or the like then have the police arrest the pedophiles and leave the fry cooks and gas station attendants out of it, even if the pedophiles eat food and drive cars.
We wouldn't be having this conversation if they had won the war - my family would be dead and I would never have been born. Excuse me for not protecting their right to speech (which leads to action).
Nazism isn't a point of contention anymore. We know what their rhetoric leads to (because they were allowed to rise and showed the world what the consequences are). That's why I said they lost the war. It's imho a pretty strong argument to draw the line right there, to ensure the sacrifices were not made for nothing.
No. The reason why freedom of speech (or freedom of expression, if you're Canadian like me) is important in a government context is because the government (in theory) has a monopoly on violence. Businesses and private individuals should not be expected to uphold and enforce freedom of speech because requiring them to do so puts them at odds with themselves.
An employer expected to uphold freedom of speech must then require their employees to work with or for people who believe they are lesser people, not even people, or should be the victims of abuse, violence, and genocide. That is an untenable position from a human rights perspective. I as a private individual do not want to do business or associate with people who use freedom of speech as a platform to preach hate or ignorance, and that is my choice.
I am happy to see companies kick bad actors to the curb, whether they are fashionable nazis being deplatformed, gamergaters harassing women and minorities, and I am frustrated when I see legitimate journalists being censored by those same platforms. It's not cognitive dissonance to support stopping bad actors while rallying to support good actors, it's a recognition of the fact that our rights are open to abuse and that the current system isn't great at coping with abuse.
I certainly don't think that requiring businesses and platforms to provide guarantees around freedom of speech is a good idea, and most platforms that have attempted to or succeeded in doing so have turned into the cesspools of the internet.
> The reason why freedom of speech ... is important
This view is (IMO) far too narrow. Freedom of expression, particularly political expression, is absolutely essential to the functioning of western society as it currently exists. Personally, I also see it as an ideal to be pursued in and of itself regardless of any functional need for it.
Government regulation has a high potential for abuse for a number of reasons (the monopoly on violence is merely one) so it makes sense to take steps to constrain it in certain critical cases. Note that this doesn't imply anything about private entities; it is simply a logical consequence of a functional need or ideal in the context of our current system.
As to private entities (businesses, etc) things vary based on context. Certainly I don't have any wish (for example) to force YouTube to host pornographic content. However there may well exist cases where broader freedom of expression (either the functional need or the ideal) requires protection against private entities.
California actually has such a law - an employer is not permitted to take actions that would "influence or tend to influence" their employees political activities outside of the workplace. This can get very complicated (as you might imagine) when an employer takes an official political stance on an issue.
US telecoms are also subject to regulation of this sort (ie common carrier laws). Personally I think that infrastructure companies (Cloudflare and other CDNs, as well as those providing the physical layer) ought to be subject to something broadly similar. It would help protect usage of and access to the underlying infrastructure for everyone by shielding the relevant companies from negative public opinion (public outrage campaigns accomplish nothing if the company is legally required to serve all customers).
> ... have turned into the cesspools of the internet
Broadly, I think you are tending to conflate infrastructure providers with social networks (and similar end user sites). There are important differences (for example) between FedEx and an online retailer, even though they might both be involved in getting a physical product to you.
> Government regulation has a high potential for abuse for a number of reasons
Can you identify any government regulation that is enforceable where that enforcement isn't rooted in the monopoly on violence? Note that violence doesn't just mean capital or corporal punishment, it also depriving individuals of their possessions (for example, fines, asset forfeiture, etc), or their freedom. Each case where government can impose regulation carries the full weight of the threat of violence. If not for that threat of violence, the regulation would have no meaningful impact.
> However there may well exist cases where broader freedom of expression requires (...) protection against private entities.
I agree with you, but I don't think the answer here is to give government a mandate to dictate a policy of freedom of expression, I think the correct step is to evaluate whether a business has grown sufficiently large to constitute a threat to expression in society, and I think the model of being a monopoly is close to that. Disney is a stellar example; they don't own all of the movie studios, but their enormous size and ownership of the intellectual property rights to such a huge swath of cultural history for the last century, and the way they use those rights is growing increasingly harmful. I don't think it's right for government to say that Disney should start distributing adult films, but I certainly think it's appropriate to introduce regulations governing how a company like Disney can negotiate for, or more practically, dictate terms to movie theatres and retail outfits to carry or distribute their products.
> Broadly, I think you are tending to conflate infrastructure providers with social networks (and similar end user sites)
I do, but that is a separate discussion around the way that major platforms, cloud service providers, content studios, transit providers, and online retailers have become interconnected through acquisitions and competition that it has gotten to the point where I treat any online service as a "temporarily disenfranchised critical infrastructure". This is largely due to the idea that for many online services the end goal is to either achieve enough market share to become a competitor to another internet company, or to be acquired trying.
Suppose a guy walks into a grocery store with a shaved head and swastika tattoos. He wants to buy a bag of apples. The store owner doesn't like the cut of his jib and asks him to leave. That's fine, the guy goes to the grocery store across the street and buys the apples there.
The store owner across the street doesn't care what kind of tattoos a man has and is willing to sell apples to anybody. That's fine too, right? But then an angry mob descends on the store, filling the entrances and harassing the customers, accusing the store owner of being a complicit racist for doing business with Nazis. The store owner does some simple math and realizes that selling a bag of apples a month isn't worth this kind of trouble, so they start to turn the man away too.
Then the angry mob does the same thing to anyone who will do business with the man at all. He can't buy food, electricity, clothes, medicine, nothing, or the merchant will be the subject of a harassment campaign which will cost them more than they could get from his business.
This is clearly just as coercive as anything the government could do, if not more. Even they're not allowed to deprive you of food and medicine.
And as much as we might like the idea of making life impossible for Nazis, the target is only the current whim of the mob. Nothing guarantees that a given target is actually evil.
Or that they're paying enough attention to notice that the guy with the shaved head and swastika tattoos is a Buddhist monk and not a Nazi at all.
Which is why, when the government does it, they have an obligation to give you a fair trial and access to an attorney and respect your constitutional rights. None of which the mob gives you.
But yep, this is pretty much my thinking as well - that being ruled by the whims of the mob is hardly desirable from a societal standpoint because it ultimately results in many of the same issues that government restrictions on expression do.
This is a bad analogy. Cloudflare is not inspecting individual packets for hate speech. They are refusing to do business with an organization that negatively affects their brand (The Daily Stormer). They should have the right to make that choice as a private entity.
"They should have the right to make that choice as a private entity."
No they should not.
And the analogy works: if you're getting filtered on the basis of your content - at the packet level or not - then it's fundamentally against Net Neutrality.
Wait until the PR team at Verizon decides they don't want to publish your content because you're too vocal about BLM. Or, they will only support you if you do support BLM, or something rubbish. And now your VPN, your server host, caching technology provider, Telco, wireless provider, Visa/Amex, Video-conf provider - it's completely absurd.
FedEx won't ship 'PlanB' because it's a 'controversial' medicine? But they will in 3 states?
USPS will ship condoms everywhere but not in Utah where the local Union forbids it?
Alaska Big Oil gets their local VPN owners to ban Greentech related sites?
Trump's buddies on the Board of AT&T get them to threaten anyone hosting 'fake news' about Trump?
California Teachers Union Pension Fund presses Cloudflare to ban all hosting of anything related to law enforcement?
And FYI nobody is acting 'morally' - they're scared executives just trying to do whatever to hush people up and continue making money - a system which hands arbitrary power to arbitrary groups. This is not what anyone wants.
For services that are inherently 'content neutral' - the content should not be allowed to be a basis of discrimination.
For Social Media it's different, as there is an inherent association between the platform and it's users, but not for Cloudflare, or AWS or Verizon, Gmail for example.
There is no end to the insanity otherwise; we need basic, smart and clear regulation.
Edit: I should add 'and that's just the US'. Imagine when a very vocal, organised group wants to ban Arabs living in what is commonly referred to as 'Palestine' from using the term 'Palestine'. Or Serbian authorities from hosting content using the term 'Kosovo' in any way that reflects its supposed 'autonomy'. Or Greek companies ganging up on Macedonia's usage of the term 'Macedonia'. Or Greens in Germany from banning pro-Nuclear energy content. There are at least a handful of Tweeters who would want those things. It gets infinitely messy, very quickly.
There is clear regulation in many countries: nazi speech gets you thrown in jail, the rest is mostly fine. See njmerious european countries like Germany that aren't some dystopia hellscape. That reflects a lot of what happens voluntarily today anyway, but it can't of course be formalized into regulatory law in the USA, because such regulation is restricted by the first amendment.
Your panic is unwarranted anyways, because many of those countries do allow free speech restrictions, yet no runaway crazy banning has happened yet. Its just a fictional slippery slope made up to protect hate speech. Words don't protect against tyrany anyways; actually fighting against tyranny does.
"Its just a fictional slippery slope made up to protect hate speech"
Ah, and there is the ugly, authoritarian, Stalinist argument right there.
"We only ban, denounce, destroy those who are GUILTY of Hate Crimes, so how could you have any worry about that?"
People are banned all day long on Facebook, Twitter, YouTube, Stripe, comments sections etc. for any and all sorts of made up reasons that have nothing to do with hate speech. And FYI you can say things that are essentially hate speech, but in a political context may not be, so it's fine i.e. "White People Are a Disease" is a common thing on Twitter. No problems there apparently.
And speaking as a former German resident, they have some real Nazis there, none of this kind of 'made up suff' like in the US.
This is a problem that needs to be addressed.
'Freedom of Expression' is the #1 Amendment to the US Constitution and the #1 thing in the relatively new Canadian constitution (#11 in EU) for a very good reason: it took thousands of years for 'every day people' to be able to speak their minds without being shot.
We can't leave this issue up to corporate CEO's, mostly trying to read the Twitter tea leaves.
I agree completely. I think democratizing speech so that it cannot be controlled by any source of power, government or financial, does have real importance. But its not the most important right, compared to life or wellbeing: instead, its important because it supports what's actually important.
But I have a message of good news: it turns out speech can be lightly restricted while leading to less authoritarianism or cronyism, as can be seen from numerous examples, by simply restricting speech in a way that is kept controlled by the specific people who are subject to it.
I don't have a lot of faith that we will resolve this, but the 'Net Neutrality' vs. 'Ban Someone I Don't Like' hypocrisy is dizzying.
The positions are ideological and naive - they are obviously contradictory - it's only that when they are framed in certain ways, often 'straw-man-ish', do people think they have clarity on them.
For example, it's easy to understand why we don't want Verizon charging customers different prices based on the nature of the content. It's easy for us to want to 'ban Nazis' from Reddit. But both of those are unrealistic ideals (i.e. straw men) because neither contemplate the broader application. It's a very slippery slope.
It's not a big deal that Nazis get banned from somewhere, and maybe it's not a big deal that Verizon wants to design services such that your profitable business costs 2x on their network - but those bounds expand very rapidly.
On the front pages of CNN and Fox right now we have a completely ridiculous war over 'Goya' canned foods because of some arbitrary comments the founder made about Trump.
This issue is only going to get much worse and more complicated over time.
We have basically no choice but to define what kinds of businesses can use what thresholds, and try to infer what those thresholds are.
> This issue is only going to get much worse and more complicated over time.
Slippery slope is a fallacy. Humans aren't required to follow the precedent of previous decisions like SCOTUS justices are. People eventually see the problems with previous choices once the consequences hit a tipping point.
We are having multiple moral panics right now. They will pass.
People will eventually get more tolerant of others making mistakes on social media; it will accelerate once more of us know someone who directly loses their job/status/etc to a moral panic. Also, there are examples of people who were targeted by moral panic mobs (eg. Colin Kaepernick) who survived the outrage and overcame it.
Once enough people get {"cancelled", fired, boycotted, excommunicated, etc}, these calls carry less and less effect over time and social media mobs lose their power.
> On the front pages of CNN and Fox right now we have a completely ridiculous war over 'Goya' canned foods because of some arbitrary comments the founder made about Trump.
Those are bad gauges of anything except outrage-for-clicks. Those two webpages rile up their core viewership to generate eyeballs and collect web-ad revenue. Trump only got to the front of the 2016 Republican primary because the rest of the field was boring and he riled up moral panics and owned most of the news cycles since.
> but those bounds expand very rapidly
Not always. And this ignores the observable fact that there is almost always a regression to the mean. There are social frictions which prevent these moral panics from burning too long.
The 'slope' has been 'slipping' for at least 10 years, and here is no indication that it's going to let up.
The opposite of 'fallacy' - it's literally happening all around you:
1) We now have major corporate support for 'social claims' with brands backing (and cashing in) on celebrities for their politics, not their skill at 'whatever', and supporting some causes surrounding a lot of public disruption. These have led to some groups trying to 'ban' others.
2) The attempted influencing of electoral outcomes via 'bad information' from outside agents, agitators, along with a plethora of information - with no guidance other than the whims of CEOs. There's all sorts of 'banning' on FB and YouTube and it's all pretty grey, often it can have major consequences.
3) CEO's brought to congressional hearings for these issues -
this is a sign that 'ban' culture is now a top concern.
4) Weekly national 'pop culture wars' over which brand has transgressed which group, promoted widely in the press, resulting in calls for bans of some products or services.
5) 'Social concern' as the primary legitimacy for populism, even among those who are supposed to be popular for their acting, music, or athletics. In the last 10 years - ever 'celeb' has embarked upon a 'personal branding campaign' to imbue themselves with moral authority. There was always a little bit of this - but now it's 'the thing'. Go and have a look at social media, try 'Kristen Bell'. Every few posts are about some kind of moral concern. Otherwise: they could get the ban. A host of globally respected individuals had to sign an piece in Harper's. This is basically 'shocking' to anyone of a certain age, because intellectuals and their ilk used to be the one's fighting for freedom of expression against the authorities - now, 'ban' culture has infiltrated their organisations and become a populist issue.
6) There are disruptions and protests around what would be normal, common, mundane events: 'The Joker' review by the Daily Beast was entitled: "Everything About the Joker was Absolutely Infuriating". The 'infuriating' parts had nothing to do with the film, but rather the supposed politics of the film. The DB and others wanted the film to not be released is it were. There are weekly arguments over who can play who in film, supposed issues of representation, wars over the composition of selection committees - often resulting in the 'cancelling' of individuals.
Twitter and social media have created the ability for agitators to move giant waves of people in an emotional, populist direction, resulting in quite a lot of effort to ban, cancel, disassociate.
It's far worse than ever before, and it's going to be worse before it gets better.
I see no pathway for it 'getting better' anytime soon, because the pathways to outrage have formed, hardened and are now part of our culture.
Individually - yes - most of us are getting sick of it - but that won't change the fact of the activity in the news, on Twitter, and journalists 'calling for the resignation' of so and so for this and that with the basic moral impetus of 'someone on Twitter said it, therefore, that's how America feels'.
CloudFlare, Facebook - your startup - does not want to deal with this, there needs to be some kind of collective clarity.
1, partly 5) Who are you to be the arbiter of what is 'supposed' to make people popular? It doesn't work that way. Can you read the mind of Kristen Bell and know she acts out of fear?
2, 3) No guidance other than the whims of CEOs... oh and congress. How does that add up?
4) On a slippery slope from yellow journalism to...
I'm sorry for the low-effort post, but something like this is quite tricky to respond to. Much like your view of the thing you're railing against, any legit points you may have drown in a sea of outrage completely lacking perspective and full of non sequiturs.
I wonder if there are people out there who don't want internet infrastructure to be run by the government specifically because the government would protect freedom of speech "too much".
Cloudflare was purely acting out of market based fear, there wasn't a hint of moral impetus. Literally he said: "I don't want people saying they won't work with us" - which is giving into the mob.
Where is the ACLU on this?
We were all screaming for Net Neutrality just a couple years ago.
It's up to communities and governments to make decisions on content, it would actually help if the government made it illegal for CloudFlare to refuse service to someone so long as they were within certain guidelines, thereby absolving businesses of this issue.
Imagine literally the marketing and PR teams of Verizon, Facebook, Cloudflare, AWS, Google, your rando VPN provider, getting to decide if they 'think they might not like you' or not, it's just too much.
For marketplaces like AppStore, it's fine. But for other services, this is not going to work. It's not the job of your Telco or Garbage Pickup do decide if your public statements are cool/uncool enough for their Instagram.
> Cloudflare was purely acting out of market based fear, there wasn't a hint of moral impetus.
IIRC, the reason DailyStormer pissed off CloudFlare is because the users claimed (lied) about CloudFlare was somehow participating / sponsoring the site/activities. Sports teams have non-disparagement clauses; I don't see this as much different.
The only thing I'm not clear about is if it was just a rando user on DS that said the thing or some moderator/admin who can reasonably be said to represent the organization that runs the site.
Also, Macy's and Starbucks are allowed to deny their business to individuals at their discretion (subject to Equal Rights and Americans with Disabilities laws). Last I checked, being a Nazi is not a protected status (perhaps I'm wrong).
> We were all screaming for Net Neutrality just a couple years ago.
Different concept. This is about who a vendor chooses to allow as a customer. Common Carrier status is perhaps a closer comparison, but I think AT&T is allowed to drop a customer if they violate the AT&T ToS / contract.
Also, the CEO of CloudFlare went out of his way to publicize that this was a problem for the health of the internet and to start a conversation. Society didn't walk away from CloudFlare (eg. "vote with your feet/wallet") and Congress didn't choose to create any laws.
> Common Carrier status is perhaps a closer comparison, but I think AT&T is allowed to drop a customer if they violate the AT&T ToS / contract.
That would make net neutrality meaningless, surely. Just put something in the ToS that says you agree not to actually use your internet service.
> Also, the CEO of CloudFlare went out of his way to publicize that this was a problem for the health of the internet and to start a conversation. Society didn't walk away from CloudFlare (eg. "vote with your feet/wallet") and Congress didn't choose to create any laws.
Which is exactly the problem. America has given up on free speech, and CloudFlare was a prime mover in that shift.
When they're speaking for themselves? Sure. When they're acting as a common carrier? No. It's too bad regulation hasn't kept up with the realities of how important the Internet is.
> Society didn't walk away from CloudFlare (eg. "vote with your feet/wallet") and Congress didn't choose to create any laws.
Neither of those things mean you did the right thing. It’s really easy to pick on a widely unpopular minority group and not get laws passed against you or lose a noticeable amount of customers.
> Neither of those things mean you did the right thing.
I find it hard to believe the right thing would involve either CloudFlare tolerating libel (my interpretation) about them by a customer or that we should always tolerate an unmitigated amount of free speech (at least the obviously political/religious speech originally envisioned) no matter the cost to {business, society, decency, morals, etc}.
What is "the right thing" to you in this situation?
Citation for the CloudFlare / DailyStormer incident:
> The tipping point for us making this decision was that the team behind Daily Stormer made the claim that we were secretly supporters of their ideology.
This is more about enforcing ToS and maintaining reputation than "an internet infrastructure company cancels DailyStormer because of their ideology".
The complicating factor is that CloudFlare was not the only internet infra company to drop them. DS were rapidly dropped or denied accounts from other companies during this news cycle, so they were effectively kept offline because none of the large infra companies they approached wanted to deal with the issue in a "free speech over all other concerns" kind of way.
This misses the point to an impressive degree. No one here is disputing the current legality and I think everyone here understands the difference between government and private services.
The argument is that we've come to depend on privately owned backbone infrastructure in much the same way we depend on publicly owned roads. Furthermore, the operators of that infrastructure have shown themselves to be vulnerable to public outrage. Therefore, it's reasonable to ask if additional regulations might be a good idea.
A good analogy here might be to privately operated toll roads. Surely such a system shouldn't be allowed to discriminate against you based on (for example) your bumper stickers?
No, you can use any other service you want. CF has no private toll roads, in fact you can make your own 'toll' road right know. But if you have your private toll road you can forbid any bumper sticker you don't want on YOUR road.
> if you have your private toll road you can forbid any bumper sticker you don't want on YOUR road
If this existed in reality somewhere in the US (or wherever you happen to live) are you seriously saying that you would be ok with it?! Do you really not see the necessity of having neutral infrastructure?
> CF has no private toll roads
They are largely analogous. They transport your traffic from point A to point B for a fee. Yes, competitors exist. No, you can't just go build your own any more than you can simply whip up a viable Twitter or Facebook clone in a weekend. Claiming otherwise is either to be profoundly misinformed or to advance an argument in bad faith.
> > we've come to depend on privately owned backbone
> No, you can use any other service you want.
... which is also private and subject to the same pressures due to public outrage. Disputing that we have come to depend on privately owned basic infrastructure in general because we could switch to depending on a different piece of privately owned infrastructure is to again miss the point entirely. There is not (to the best of my knowledge) a government operated CDN or network backbone which you could make use of in order to avoid such concerns.
> There is not (to the best of my knowledge) a government operated CDN or network backbone which you could make use of in order to avoid such concerns.
The line between private and public gets a bit blurry when talking about edge routing and CDNs. Especially edge routing.
Then go to your Government and ask them to make such a Service, look there is just no point arguing about that, as a private firm you don't have to serve everyone, especially groups that are potentially bad for your brand.
Why is it a precedent? Nobody was under the illusion that it isn't technically possible for them to shut down some customer they don't like.
Nor was/is there any debate about the legality of doing so,
So the only reason for continuing to work for/with violent anti-semites was that they wanted to. Until, at some point, they changed their mind.
HN has a strange infatuation with this idea of avoiding responsibility by pretending to be powerless. But it's neither morally sound nor logically or legally coherent to pretend to be bound by some principles that are entirely of one's own making.
Precedents aren't limited to the legal system. Their actions changed the expectation (ie established a new precedent) for their behavior going forward.
> So the only reason for continuing to work for/with violent anti-semites was that they wanted to.
Or entertaining broader ideals surrounding freedom of expression? Or perhaps seeing the inherent dangers of living in a society dominated by public outrage? Or something else entirely?
> Until, at some point, they changed their mind.
Due, by their own admission, to financial concerns. Concerns which wouldn't exist if they and all their competitors were legally required not to discriminate based on otherwise legal content.
> HN has a strange infatuation with this idea of avoiding responsibility by pretending to be powerless.
I don't see that anywhere in this comment section? Rather, I see people objecting to actions which are legal but which they feel should not be.
II love cloudflare. It has really helped out with several sites/projects that I have worked on and the service is top notch. I am also an investor. I tend to invest in stuff which I use a lot or trust/respect the employees. Weirdly what made me really invest is the level of geekiness on the company. I remember seeing you guys using a lava lamp wall to generate entropy and just thought "that's awesome". I just wanted to say don't change. Your lz4 implementation, aes gcm golang optimizations have directly benefitted me. Coupled with really high quality post mortem articles, articles on interesting things like compression, encryption, networking, a few random articles (the privacy focussed file system recently comes to mind, written at the same time I was making my own distributed fs) just leave me with a lot of respect for the culture there.
I was worried a bit when I saw the initial IPO that the free tier would leave (despite promises it wouldn't) but that doesn't seem to be the case. Literally the only bad thing I've seen on the site is recently you switched from recaptcha to a new one that I had a real tough time with logging in today - it was a bit glitchy on my pc. The only suggestion I thought of as well would be a simple "maintenance mode" similar to the "I'm under attack mode" which would allow those of us without super-ha to quickly toggle on something to pop up a "sorry server"/site is down form maintenance page without having to mess with our proxies/web servers.
Anyway I know this comes across as totally kissing-ass but I just wanted to say thanks to someone who actually works there. Everyone fat fingers stuff every now and again,don't sweat it.
If you use Cloudflare workers, you could put together a maintenance page with a simple script + Cloudflare Workers KV as a toggle switch. It would cost $5/mo if you're not already using workers.
Sounds lame but this story really touched me. I recently visited a family member who's nearing the end and it broke my heart to see them in that condition.
I'm not sure what to take from this. But thank you for sharing it
Wow, thanks for linking to the article. What an awful story. Makes you think about the wisdom of undertaking major elective surgery if it's not absolutely required in the short term.
Nothing beyond the normal "cognitive reserve" findings that high education and broad cognitive engagement makes the brain pathology take longer to reduce your functioning to a dementia state.
Never heard of him but I read this article in one sitting (extremely hard thing for me) and I felt some unexplained connections with this story. A Beautiful Mind is another book which I very often go back to and go through similar emotions.
> I remember seeing you guys using a lava lamp wall to generate entropy and just thought "that's awesome".
To be sure (and for the sake of internet rando completenessism), it does look like CF waited until the original SGI patent on the technique ran out. :)
> To be sure (and for the sake of internet rando completenessism), it does look like CF waited until the original SGI patent on the technique ran out. :)
I work at hCaptcha, we run CF's captcha. If you're having problems in the future, popping open a debugger and capturing the results can help us figure out what's going on. But also: browser, OS, site, ...?
Right now: there is an issue with Safari users on the most recent iOS and OS X, where 3rd party cookies have now been disabled by default. We're working on a solution.
If that's your issue, you can fix on your side in the short-term by not using Safari, or by enabling 3rd party cookies.
I cannot believe apple decided to roll that out in the middle of COVID. I got hit pretty hard with it and am finally rolling out a fix for my own stuff.
Congratulations with growing so big. Why aren't there dozens of lookalikes by now?
I mean CDNs that will let you override the origin's cache instructions and do a decent job of DDOS protection, and whose feature list otherwise looks a little like Cloudflare's.
CDNs are expensive to build, and often not very useful to customers until you've built out a large portion of it (actual hardware required, you can't just run it atop AWS). On top of this, much of the money is in Enterprise. So you've got to compete with Akamai, AWS, Fastly, Incapsula, Cloudflare, and several other notable ones to get any customers to speak of.
There are smaller CDNs out there. You can find them readily enough.
Depending on how you integrated it and how deeply, it may be trivial. Certainly going from one DNS-based CDN to another can be pretty easy - a Cloudflare / CloudFront swap could be quick.
Which suggests to me that CDNs are already a commodity in some ways.
One answer - It's much, much harder to build and operate a global network infrastructure that it might seem. It's also even harder to invent some sort of "killer feature" or other genuine innovation on the experience. You're likely not using Cloudflare simply for commoditized pipes alone, but for other features or designed experiences in their offering.
A second answer - there are a bunch of bottom-barrel commoditized pipe services. You likely haven't heard of them because they're so generic. They've existed before Cloudflare, and more will be created in the future https://www.citrix.com/products/citrix-intelligent-traffic-m...
Don't take this as being glib, but isn't this the entire point of funding rounds? Demonstrate growth and talk about a few approaches to creating a business model that you're investigating, which then brings investors on as they're betting you'll find one that'll succeed?
We believe our platform disrupts several large and well-established IT markets. The key markets that are addressed by our platform include VPN, internal and external firewalls, web security (including web application firewalls and content filtering), distributed denial of service (DDoS) prevention, intrusion detection and prevention, application delivery controls, content delivery networks, domain name systems, advanced threat prevention (ATP), and wide area network (WAN) technology. From our analysis based on IDC data, $31.6 billion was spent on those products in 2018, which is expected to grow to $47.1 billion in 2022, representing a compound annual growth rate of 10.5%. We also are actively developing new products to address adjacent markets including compute, storage, 5G, and Internet of Things (IoT) that are not included in the estimate of our addressable market.
That’s just general market analysis. It doesn’t say anything about what their hedge is. Or was it simply that the existing actors were that inefficient/price gauging.
LOL, I'm sure it is, but referring someone that is wondering about the "back of the napkin math" to an approximately 250 page S-1 filing is sort of ludicrous.
I don't really know, but I guess the biggest feature in the past was protection against DoS attacks.
Bandwidth isn't expensive compared to what most people are paying for it. Cloudflare is paying for the infrastructure as it is to handle attacks, so why not use it? As long as it doesn't affect paying customers then it's great marketing.
> Did they figure something else out that nobody saw?
I may not have a full scope of the history, but my own experience with DDoS protection was quite different. Whilst providers offered anti ddos protection through GRE tunnels and dedicated machines behind DDoS appliances and a heavy null route hand, Cloudflare had a simple few-click solution that worked at the web application level making things a lot easier and, also, allowing for features like caching, and thus, CDN benefit from a global network. Further, they've maximized performance on their machines, and as a result, Cloudflare is wicked fast.
Cloudflare does what it does really well and has built additional services on their global network that make a lot of a sense and provide a lot of value.
This has been my experience too, CF is just so easy to migrate to and to configure once you're there.
If AWS was able to offer a single click "DDOS protection and CDN" feature with similar pricing and features as Cloudflare then I'd consider it since most of our infrastructure is on AWS but at the moment they don't offer anything nearly as competitive. Just the Cloudfront bandwidth costs alone would dwarf our total infrastructure costs.
Indeed. I think I understand the argument claiming Cloudflare is a SPOF. However, if you're actually dealing with DDoS attacks with any kind of regularity, you're likely to have better uptime with them than without.
Cloudflare is a relatively cheap "lite CDN" for developers who want caching without putting in any work. This becomes a gateway drug for Cloudflare's more expensive plans once you outgrow the free plan. It quickly adds up; I worked for a company that wasn't even on an enterprise plan and was spending hundreds of dollars a month on Cloudflare just because they had a lot of domains.
My reservation with Cloudflare is the concept of letting a third party MitM my SSL traffic. That and it's more expensive than a cheapo CDN like Stackpath if all you really care about is CDN (and Cloudflare isn't even really a good CDN, just a quick hack to speed up small static files).
In my experience, it's extremely rare to find a CDN that doesn't expect to do TLS termination. My understanding of TLS is that it's exceptionally difficult to cache content if you cannot see into the requests.
Right, with Stackpath they do TLS but not necessarily on the primary domain. You don't have to point your nameservers if you don't want to. So you can set them up on a subdomain and use their fake SSL to serve purely static files (if the file doesn't exist on CDN at the time it's requested then their system will pull it directly from your server via a private subdomain, serve it to the client and store it for next time)
So in this way it's possible to setup CDN with shared SSL for purely static files but not the app server itself; you don't have to give the keys to the whole kingdom so to speak and it's cheaper than Cloudflare at the basic level.
Let me see if I follow. Auto-provisioned TLS (misleadingly termed "fake SSL") on the front-end for delivering static contents and caching. A private subdomain with a pinned cert not managed by the CDN to deliver static contents to the CDN. And a third subdomain for the application itself that's not going through the CDN.
I was under the impression that the same result could be achieved with Cloudflare, or indeed nearly any CDN. Was I mistaken? Though you may not actually need a secret, private subdomain for static files with all CDNs.
Again, please let me know if I've made a mistake somewhere. I'd love to learn something this morning.
You're right about that. So it might look like this
static.domain.com (CDN subdomain with auto provisioned TLS)
static-uncached.domain.com (private pass-through subdomain when CDN is missing a file)
www.domain.com (app server hosted wherever)
You're right that you could do something similar with other CDNs including Cloudflare (you can just set the www subdomain to "bypass Cloudflare" to accomplish a similar result), but I'm not aware of any way to use Cloudflare on a domain without forwarding your nameservers to them, effectively giving them complete control over the domain. At least with Stackpath I can host DNS wherever and simply point the subdomains I want at them.
Also, by the time you do the work to split static files into separate subdomains you might as well go with a dedicated CDN. One of the selling points of Cloudflare is for sites serving everything on one subdomain that they can forward to Cloudflare and get caching without any work.
> My understanding of TLS is that it's exceptionally difficult to cache content if you cannot see into the requests.
Not even "exceptionally difficult", but flat-out impossible. From the perspective of an observer, TLS sessions are random data. The protocol is specifically designed to defeat attempts to replay data -- a CDN is indistinguishable from an attacker in that sense.
Nope, but I'm sure lots of companies are in this situation, where they just barely need the features of the lowest paid plans, end up scaling across several domains and Cloudflare makes a ton of money off a vast majority of customers that will never need their more advanced features.
Plus that means hundreds of unique domains with unique content. If you have hundreds of domains all pointed back to one site with a single forwarding rule, 99/100 could be free tier accounts.
Unless you're sending all your traffic back to physical machines that you own locked into a cage in a datacenter, you are probably letting someone MITM your SSL traffic. For example if you are hosting on AWS, Amazon has access to your keys. If you are hosting on a hardware server leased from Hetzner, Hetzner has access to your keys.
When a 3rd party has access to your keys, their responsibilities to you are spelled out in your contract with them. That's true for CDNs as well as hosting companies.
For most websites today if someone can intercept traffic somewhere close to the server they don't even need the keys, they can just fake responses to pass CA validation and issue valid certificates with their own keys and MITM like there is no encryption.
And coldboot attacks performed by a hosting provider staff of dumping memory and finding keys isn't that realistic of a threat, just like putting servers into a locked cage on someone else's property isn't much of a protection.
There’s a difference between a VM host with the technical ability to carry out a targeted MITM attack against its customers using hardware-level access, and a provider that sells MITM as a service.
You don't need to use your SSL certificate for the CDN. Most websites that use CDNs only do so for static assets such as images, fonts, CSS, etc... the dynamic content comes from the origin server, encrypted with a private key that is not handed out.
If you press F12 in your browser and navigate to some major sites, you'll notice anywhere between 1-5 different CDN domain names that aren't directly related to the original host in any way.
As far as I understand it it's a freemium B2B model: If you're small you probably can get away using the free tier. Then when you get bigger you outgrow the free plan, either because you want specific features or because your bandwidth becomes too high.
That said bandwidth really isn't that expensive, at least if you're buying it at the scale that Cloudflare does. Many people seem to be used to the bandwidth prices of the large cloud hosters, which are really insane and have been marked up by a large multiplier to disincentivize people to transfer their data elsewhere for processing.
The NSA/CIA have a huge hidden budget. They are known to invest in startups. Why try to break crypto when you can just pay some tax payer money to large corps to get them to convince sites to mitm themselves.
Worth noting that cloudflare does a poor job of identifying and allowing crawlers specially Google. I have a site with 1 mil URLs , nothing big and every time you turn on cloudflare it reduces the crawler to 1k visits or less per day. If you compare your logs for visits for Google and Bing with CF turned on/off there is a huge difference.
Suffice to say, that in over 5-6 projects I have added CF to, it's never worked out in my favour.
Now that Cloudflare is also a registrar, they could pretty easily implement a nameserver failover like EasyDNS. I hope this event underscores the importance of that to them.
Reparations would show an actual sense of responsibility. Firing someone would be appropriate if they were negligent. Other measures might be more appropriate. Is it enough that any time there's a Cloudflare incident, all we get are lengthy blog posts and sorries from Cloudflare?
Firing people for making mistakes is just going to foster a culture of secrecy and shame. Being so quick to fire is not how you retain talent and it isn't how you foster a healthy, blameless development culture in your workplace.
I understand the point being made here, but what are those affected supposed to take away? Cloudflare made a mistake that caused (x millions of dollars of lost online commerce revenues, y number of missed telehealth sessions, etc.) and since we do not punish mistakes, nothing was done. Sorry everyone!
Understood. Typically, as a company, you write -
1) what went wrong
2) how did it go wrong
and
3) what you have or will put in place to prevent it from ever happening again
It's a learning process for all involved, really.
From the affected parties point of view, well, they should diversify their network a bit better. End users should hold those companies feet to the fire, not Cloudflare's.
> Firing someone would be appropriate if they were negligent
Maybe, but making mistakes is far from negligence. Besides which, if a single person can accidentally break your system, at least at Cloudflare's scale, that's an organizational failure, not a personal failure.
Question: is it even possible to have DDoS protection without using a provider of it which becomes a single point of failure? Or is it maybe possible to decouple this single feature from everything else that Cloudflare provides that could take out all the sites in the future from an unrelated misconfiguration?
I don't see the centralization as a positive, but I'm wondering what percentage of the websites that were taken offline see themselves as having no choice but to use Cloudflare in order to prevent themselves from being taken down anyway from malicious actors instead of by accident.
I think you could use Cloudflare as your primary DNS provider and benefit from their DDoS protection, but also specify backup DNS name servers with a different DNS provider in case Cloudflare fails.
Sure, Arbour and others sell devices to deal with ddos and many isps have clusters of these which you can use. Of course this is a service that costs money.
Whether this is an ad piece by a competitor or not, the problem with monopolies is that "the market" (if there is one) gets skewed incentives. Cloudflare has received heavy investment by FAANG (+MS) [1] before their IPO, so rather than eg Google or others with a vested interest and capability stepping up the game and invest into new IP control plane-level DDOS protection standards or similar, the situation smells more like a backdoor deal, such as an agreement to not go after a particular market segment.
Let's also not forget Cloudflare in particular have been accused to host/hide the very bad boys that make protection from DDOS necessary in the first place. Whether or not that is the case, a quasi-monopoly leaves customers with no choice.
Militaries. The internet was created so the US President could command the US military even in the case of nuclear war knocking out many of the internet's nodes.
Maybe it would make sense to have multiple independent sections of backbone at the BGP level. Instead of having one public AS/backbone, break it down into regions at least so that it is more confederated.
I love all the comments about fail-over for DDOS/DNS protection. What is your budget? Well we are looking at around $0.00 for our maximum allowance. Ok so single point of failure it is I guess we are done here. Companies only say they care when there is a problem, the reality is that they dont.
It obviously takes more than $0.00 but not that much more. It’s a matter of adding a second DNS provider and making sure you replicate DNS records manually or with AFXR.
What’s really puzzling is if the same companies spend money on active/passive failover for application and database servers while overlooking DNS single point of failure.
Precisely. I remember when the CEO of my old company came to me and said (with respect to moving from a on-prem model to SaaS), "What's our SLA going to be?"
"Well, what do you want it to be?"
Give me a number and I'll tell you how much it will cost and how long it will take to get there.
413 comments
[ 2.6 ms ] story [ 463 ms ] thread> We must be your domain registrar for this to work...
IIRC, Netflix OSS published some tools quite some time ago to support multiple DNS providers but I don't know/remember if they tackled the availability problem. The question comes down to build vs buy and whether the solution is general enough to warrant an Open Source Software solution.
[1] https://easydns.com/dns/proactive-nameservers/
> The only requirement to use Proactive Nameservers is that we have to be your registrar, because we need to connect to the registry to update your nameserver delegation.
So I guess technically this could be achieved with an API for your domain settings.
The classic way of doing this is AXFR (your own DNS server is a "hidden master" and the DNS providers are the slaves).
The problem is you won't be able to have redundancy at the registrar level, but that has historically at least been less of an issue.
The magic happens at BGP level.
I considered CF as a domain registrar, but they don't allow setting the NS records. So you must use them. (They basically use sane no-nonsense domain registration as a way to gain leads for their main product. Pretty smart actually, because it's a great high-level add-on for their main product, but they just went ahead and made that the bait for everyone.)
Anyway, ideally, if you add 2 separate sets of NS servers to your NS records then you eliminated this SPoF, great. Sure, it's your job to keep them updated, and in sync (preferably, to avoid problems like half of your users landing on a different CNAME/IP/etc).
And recursive nameservers will handle the failover.
What are the brands offering DNS at a flat rate over anycasted IP over a few continents?
Most of the offers I see are per query at high rates. Setting up my own ASN to do this would be too expansive in IPv4
There's HE.net's free DNS, and though they don't explicitly advertise as, it's anycasted. (Check via https://tools.keycdn.com/ping , try 216.66.80.18 [ns5.he.net].)
https://www.cloudns.net/premium/ seems to be quite affordable with no query limits :o
You could do it with BGP, but it is non-trivial and you need your own ASN to do that.
In theory they could simply create a few subsidiaries, let's call them saferDNS1,2,3 and have them build completely different redundant DNS architectures, and add then add the resulting nameservers.
That said, it'd be good to see an actual domain that uses this "proactive" feature to see what easyDNS is doing.
So if you have Cloudflare + (ex:) NS1, and you're using Cloudflare for caching, you need your NS1 records to return Cloudflare proxied IPs normally, but origin IPs under failure conditions. That's a lot of infrastructure.
It also fails completely if you're relying on Cloudflare for DDoS protection and IP obfuscation because a failure means your origin IPs get exposed. That's assuming Cloudflare DNS being down means Cloudflare proxying is down too. It might not be the case, but I think you'd have to plan for it.
Then there's also Cloudflare's detection of nameservers. I haven't tried it with more than Cloudflare's nameservers set for a domain, but if your domain doesn't actively use their nameserver they'll drop your site from their system. So, at the very least, you can't use Cloudflare as a secondary DNS provider (at least the last time I checked).
Of course CF doesn't support anything like this, but it works well (because they use quasi fixed, static anycasted IPs for the HTTP(S and TCP?) proxying/load-balancing too), even if it's hacky as hell.
If they have any active verification of nameservers, and if they disable the proxies if they detect something bad, then ... it won't work obviously :)
But technically there's nothing amazing in being a registrar of a domain. So both CF and easyDNS are just stubborn in the name of user experience (consistency).
... all in all, working around any SPoF (reliably) will usually require exponentially more resources/engineering/care.
That's a huge negative and I can't believe they think it's a good marketing point. I don't want a patent encumbered, non-standard solution for critical infrastructure.
> We must be your domain registrar for this to work.
So they're updating the domain record at the registry level to facilitate failover? That's the only scenario I can think of where they _need_ to be your registrar. Assuming that's the case...
I've always seen 24-48 hours quoted as the worst case wait when updating nameservers at the registry. I've never seen an explanation of how it works, what's allowed to be cached, how long it actually takes to update, etc.. How do they do it in a way that's suitable for failover? Do they have a special SLA with registries?
How would the registries handle a deluge of nameserver updates? Imagine a Cloudflare scale failure and corresponding registry updates. Would the registry servers be able to handle it?
I'd love to see a technical explanation of how their proactive nameservers system works.
Another good DNS option is dnsimple.com or, indeed, EasyDNS. For even more redundancy, use one provider as your domain registrar and another for your nameservers (and set short TTLs for your zones so you can re-point IPs quickly if you need to).
For the other things Cloudflare offers on their free tier, I'm not sure what good alternatives exist (there must be some, I'm just not familiar with them outside of the obvious AWS alternatives).
Edit: one caveat with above advice, I have no idea if netlify use cloudflare behind the scenes...
Edit 2: For other options, checkout https://www.cdnperf.com/ and https://www.dnsperf.com/
Full disclosure, I work at Google on the SRE team for Cloud CDN. If you want a credible alternative for the CDN part of Cloudflare, our product is extremely fast. We were all very sad for Cloudflare and watched the whole affair closely, as we've got a few customers that use our services alongside Cloudflare's.
Yes: as the weblog post points out, you can have EasyDNS as your master with their multiple DNS servers, and then also have (e.g.) Route53 slaved to EasyDNS and have those in addition to EasyDNS in your records.
DNS servers have had replication for decades.
First of all "I use easyDNS so I didn't notice it at all tbh" is not only a childish assertion, it's borderline a falsehood. You DO NOT offer the same services, nor the same scale. (No, VOD would not work if your VOD provider used Cloudflare's offering.)
Second of all, as some have noted in other comments, you are very welcome to get just as big as them if you can offer similar (excellent) service and similar extremely competitive pricing. Otherwise, keep working on your offer and stop going for low hanging fruit like bashing the competitor for an outage when they literally might handle 1000x your traffic, and perhaps offer 20x the services your offer.
Just a little rant ...
This is obviously subjective, but to me it didn't come across as "they suck use us" but rather pointing out the inherent flaws in this quite popular SPOF and cautioning to avoid it.
otoh it’s just marketing, and CF is no stranger to it, so i think it’s fair.
For whatever reason there's this modern idea that if a company A is paying money to company B for a service, that company B will handle all the 'hard stuff' for them.
The end result is we have a lot of applications/infra built with SPOFs, in some cases known, but in many, swept under the rug and abstracted away to passing the buck in case of a large failure (i.e. major AWS/Cloudflare/Azure outages).
You also see this at times when vendors pitch internal software solutions. I've been at more than one shop where a vendor's 'silver bullet' turned into a SPOF time-bomb because nobody considered this company's solution could fail. After all, the sales presentation said it had %nines%!
1. no one's going to blame me if my app goes down when half the Internet is also down but they are going to blame me if my custom solution to the same thing causes an outage,
2. there's no way my custom solution is going to achieve the same uptime. AWS/Cloudflare/Azure are not perfect, but whatever I roll for myself is almost certainly going to be much less perfect.
The thing that they're bashing is not the mistake, which happens to everyone. They're bashing SPOF:
> EasyDNS was unaffected because while we do use Cloudflare to soak up large DDoS attacks against our nameservers, we don’t use them across all of our nameservers. I think somewhere in my book I wrote “DNS providers have a near-pathological aversion to SPOFs” (Single Point of Failures). Maybe only we do.
You indicate they said " "I use easyDNS so I didn't notice it at all tbh" but NO WHERE IN THAT ARTICLE was that statement
The actual quote is
"We’re familiar with Cloudflare’s DDoS service for DNS providers, because we use it ourselves. Fortunately easyDNS was not impacted by the outage (I didn’t even notice it, tbh),"
This is a MUCH different statement than you attempt to cast out as call "childish". He is stating that the services they use of CloudFlare was not impacted by the outage
The problem with near market monopolization, oligopoly, even the singularity, the fail-case is catastrophic and may even wipe out decentralized, diffused, dispersed, decoupled system solutions that can't make it due to so much relative size from the big fish that it squashes them along the way. The bigger the ship the longer it takes to turn.
This Cloudflare issue is like the recent Facebook SDK startup crashes where everyone has a single point of failure on Facebook SDK where people should be using or able to use the OpenGraph API directly as they need which is more robust to the app that uses it, it won't crash on startup.
In business it is a goal to centralize to grow, in nature and robust systems it is more differentiation and decentralization to survive. There will always be a push and pull between these two forces.
Systems and markets are like gardens. The garden must be maintained, new seeds planted and helped to grow from small to mid-sized, mid-sized plants the bulk of the garden, and then the larger plants need to be culled back when they get too big to not take the mid-sized and then all the resources from the new seeds/small plants. The problem is we have allowed the top end to take over the garden and when they fail they fail spectacularly. The bigger the scale the bigger they can fail.
How would you even set such a thing up? I fear that you might get a couple of collusionary companies that bail each other out and smaller providers might just be left out to dry…
I do love using CloudFlare for DNS, lots of great features and generally works well, but I wish they would support AXFR for the lower tiers. I've been working on a solution for this using the CloudFlare API, but we'll see how well it works out.
However, cloudflare decision turns cloudflare "free" offering into a free SPOF. They should extend this offer to their free users, who could then use the secondary DNS that most hosts/domain name sellers provide for free to the IP that is proxified by cloudflare. It could even be limited to the case of proxy or cloudflare DNS failure, so that cloudflare could still price discriminate (make AXFR fail, unless cloudflare is down, like a dead man's switch)
That's all they did. They weren't a mail host or provider, there was no UI, nothing. All they did was allow you to specify them as, well, a secondary MX, so if you were offline they'd cache your inbound email until you were back. Simple. Efficient.
> At easyDNS we experienced so much pain from this reality that we created a system to automate flipping DNS providers at the first sign of trouble.
> We call it Proactive Nameservers, and we’re the only company in the world doing it for some reason. Maybe this is because in order to provide a service like nameserver failover, it means a company has to admit to its customers the reality that their own nameservers may at some point, fail.
Yes there are a lot of smart people at FAANG and Cloudflare corp.
There are a lot of just as capable folks not driven by job addicted meme.
Technically there’s no reason the web couldn’t be replaced with 1:many via Wireshark key sharing based access control to local content.
But via Wall Street, along these very particular rules, is how we are told to trade information. How is that not a planned economy?
Not just by doing what we’re clearly interested in doing naturally.
Make no mistake: big corp isn’t making us login at gunpoint. “They” didn’t do this. “We” did this.
You needn't use your real name, of course, but for HN to be a community, users need some identity for other users to relate to. Otherwise we may as well have no usernames and no community, and that would be a different kind of forum. https://hn.algolia.com/?sort=byDate&dateRange=all&type=comme...
How so?
On a century scale individual murderers aren't a huge concern to society because they are either dead or infirm by then.
I think for the sake of a diverse community and economy, monopolies should be difficult to achieve even naturally.
But for this specific example, a robust technical redundancy doesn't stop CloudFlare from going out of business. A technology company going out of business is pretty much the norm. Incumbents are a relatively new phenomenon for technology (sans some key exceptions), and I don't think CloudFlare is an incumbent. They are an accessory, and your business would probably run without them.
Cloudflare has had network affects from integration with wordpress and cpanel too I believe for some time now.
without cloudflare your site can be taken offline by any random person willing to spend $20 for ddos sellers.
The free plan is a pretty big moat imho, especially if 'half the internet can fail..' - I doubt 10% of those sites would be using cloudflare if there was a monthly fee pushed on them.
Admittedly, my cloudflare usage is only about 30 sites, so my data point is small. The few hosting and design clients I have and their budget constraints are not indicative of 'half the internet' - but I don't believe fortune 500 and SV sites are either.
There is no cloud monopoly either. Customers can choose from AWS, GCP, Azure, and several others.
The problem is not market concentration. There are plenty of options. The problem is customers choosing to put all their eggs in one basket.
The DNS is part of the load balancing, they serve different IPs based on location of the DNS query.
Edit: Apparently they do support a CNAME configuration if you pay for one of their business plans. That gives you the option to quickly switch away (if your TTL is low enough) but will impact performance by having to fetch the CNAME every 60 seconds.
But I'm pretty sure they use DNS do other loadbalancing and DDoS mitigations.
For example, if a site is under attack, they can send it to different IP addresses to keep it away from other sites. Or if someone is directly targeting a cloudlflare IP with a DDoS, they can redirect all sites to other IPs and just blackhole that IP.
Shouldn't be too hard to build an abstraction layer that updates them all when you need to make changes.
It makes sense that cloud companies are inclined to keep customers from giving money to competitors, but they way they sell it and structure services, reserved instances, and enterprise discounts is such that you basically are putting all of your eggs in one basket.
Nobody got fired for buying IBM, until they did.
One of the selling points of cloud providers is managed services like SQS. If you run a multi-cloud architecture, you either can't use managed services, or have to build abstraction layers on top of them (and only use the features that exist in both cloud providers' versions of the managed service).
If you want to use a managed service that only exists on AWS, then that's obviously incompatible with a fully multi-cloud architecture.
And this is, of course, why they do everything they can do discourage it. Because if you do that, not only are you not reliant on them for availability, you can switch more of your business to the other provider(s) based on current pricing, and they do not want that big time.
If you make the engineering decision to go multicloud for whatever reason those are inherent trade offs you need to be aware of. They have their own agenda of course in addition to any actual fundamental "real" in bulk efficiencies that price reflects.
Until relatively recently absolutely none, and now almost none of the tooling allows effective multi-cloud or hybrid cloud/private.
Basically the cloud providers work very hard to prevent the commodification of their services with special incompatible service offerings, lock-in, interdependency, deep and opaque APIs for integration, and networks of training and certification that position change as a direct threat to people's job security.
Cloud providers today are basically Microsoft in the 90s.
Much as open source challenged Microsoft, I would say that the world now needs open infrastructure tooling that positions hybrid and multi-cloud as first class infrastructure architectural cases in order to displace established cloud provider hegemony. Even then we will have to fight the hardware and real estate economies of scale available to large established cloud providers.
I wrote some observations about this space based significantly on HN community comments prior to the rise of Docker a few years ago: http://stani.sh/walter/pfcts/ ... click 'original' ... the conclusions still seem timely.
Until relatively recently, the cloud didn't exist.
Depends:
> IBM and other mainframe providers conducted this kind of business in the following two decades [after 1961], often referred to as time-sharing, offering computing power and database storage to banks and other large organizations from their worldwide data centers. To facilitate this business model, mainframe operating systems evolved to include process control facilities, security, and user metering.
[…]
> In 1998, HP set up the Utility Computing Division in Mountain View, CA, assigning former Bell Labs computer scientists to begin work on a computing power plant, incorporating multiple utilities to form a software stack. Services such as "IP billing-on-tap" were marketed. HP introduced the Utility Data Center in 2001. Sun announced the Sun Cloud service to consumers in 2000.
[…]
> In spring 2006 3tera announced its AppLogic service and later that summer Amazon launched Amazon EC2 (Elastic Compute Cloud).
* https://en.wikipedia.org/wiki/Utility_computing
Compared to the endless content marketing Cloudflare posts [1]? It's an ad, but they're still right. That's just good content marketing (informative, relevant, and perhaps you buy something because of it).
[1] https://news.ycombinator.com/from?site=cloudflare.com
It won't just be one single website that goes shitty with blockages and manipulation and censorship. It won't even be just the web. When Cloudflare achieves their goal of deep packet inspection at every peering and transit point it'll be the end of the internet as we knew it and the slow transition to just another cut apart "China-net (tm)".
You can add another system in parallel as the vendor of the product suggests, or you can improve the resilience in the redundant system.
To make a hyperbole: my galera cluster is failing, its a single point of failure, so I setup a cockroach cluster in parallel.
In a way, it is right, as there are failure modes specific to the individual systems, but I think, it is incorrect, to label that a SpoF.
Wait, why? [0]:
> Proactive Nameservers is a patent-pending system that optimizes the nameserver delegation for your mission critical domain names.
Oh.
[0] https://easydns.com/dns/proactive-nameservers/
It's absolutely true that if us-east-1 in AWS has a bad day, a significant fraction of the American digital economy will shut down. For some companies, the same is true of Azure and Google's various comparable offerings.
I read your post as skeptical. Why would you be skeptical? If you care about keeping your product up, you absolutely should have a fallback for cloudflare if you're a customer of theirs. Now, you might not care (and actually, for most folks I submit you need not care), but the folks making sure Ambulances get timely push notifications and realtime driving instructions probably care quite a bit.
A steel chain made of links has as many SPOF as links.
The Physalia paper is a wonderful read that explains both the justification and high-level approach to implementing said isolation [1].
Another great example is shuffle sharding [2], notably used in Route53.
The end goal of this is that aws outages would be isolated to subsets of aws customers where possible. So instead of half the internet failing it's only 10% (obviously the exact numbers depend heavily on implementation).
[1] https://assets.amazon.science/c4/11/de2606884b63bf4d95190a3c... [2] https://aws.amazon.com/builders-library/workload-isolation-u...
I went ahead and bought calls ahead of NET earnings next month. Cloudflare is becoming an increasingly bigger part of the internet backbone. Purely speculating here, but I wouldn't put it past AWS or another large player acquiring them soon.
I really hope it doesn't come to this sadly. I'd be okay with DO or somebody who isn't as massive doing a merger. Maybe they can pull resources to make each other even successful and maintaining reasonable independence.
Careful about this methodology. Some services at my org were impacted despite not being direct CloudFlare customers. They had external dependencies that used CloudFlare.
e.g. NPM.js uses CloudFlare DNS, so services which needed to talk to NPM.js weren't able to do so.
Internet™ by Amazon.
I really hope it doesn't come to this. I assume such a move would create a vacuum for a competitor. Not everyone wants to be completely owned by AWS.
My sincere thanks to John and all the PortableApps.com contributors.
Did they have some crazy in to get cheap bandwidth? Did they bet big on bandwidth prices falling? Did they figure something else out that nobody saw? Do they just to a tremendous job of migrating sites from free to paid plans?
https://www.sec.gov/Archives/edgar/data/1477333/000119312519...
“We had to have the conversation now because at some point we’ll be a public company. We had to prompt that discussion,” said Prince, who added “we want to be ready internally by July 2018,” for a possible stock offering.
CNBC Link: https://www.cnbc.com/2017/09/24/cloudflare-ceo-matthew-princ...
Editorials discussing the event: https://www.washingtonpost.com/opinions/where-to-draw-the-li...
https://www.nytimes.com/2017/09/13/opinion/cloudflare-daily-...
In that case, they still would have been taken down because of the illegal content, even if at an (extremely simplistic) surface level the timing seems to indicate otherwise.
This is where you've gone wrong.
There isn't always a line, some businesses are in the business of selling products and services and not in the business of policing content. And if some of their customers are pedophiles or the like then have the police arrest the pedophiles and leave the fry cooks and gas station attendants out of it, even if the pedophiles eat food and drive cars.
Yes there is. The line is Nazis now. There was a war, they lost. They're not welcome anymore.
Might makes right and censorship of political opponents was Nazi canon. I thought you were saying their ideology was defeated.
Nazism isn't a point of contention anymore. We know what their rhetoric leads to (because they were allowed to rise and showed the world what the consequences are). That's why I said they lost the war. It's imho a pretty strong argument to draw the line right there, to ensure the sacrifices were not made for nothing.
An employer expected to uphold freedom of speech must then require their employees to work with or for people who believe they are lesser people, not even people, or should be the victims of abuse, violence, and genocide. That is an untenable position from a human rights perspective. I as a private individual do not want to do business or associate with people who use freedom of speech as a platform to preach hate or ignorance, and that is my choice.
I am happy to see companies kick bad actors to the curb, whether they are fashionable nazis being deplatformed, gamergaters harassing women and minorities, and I am frustrated when I see legitimate journalists being censored by those same platforms. It's not cognitive dissonance to support stopping bad actors while rallying to support good actors, it's a recognition of the fact that our rights are open to abuse and that the current system isn't great at coping with abuse.
I certainly don't think that requiring businesses and platforms to provide guarantees around freedom of speech is a good idea, and most platforms that have attempted to or succeeded in doing so have turned into the cesspools of the internet.
This view is (IMO) far too narrow. Freedom of expression, particularly political expression, is absolutely essential to the functioning of western society as it currently exists. Personally, I also see it as an ideal to be pursued in and of itself regardless of any functional need for it.
Government regulation has a high potential for abuse for a number of reasons (the monopoly on violence is merely one) so it makes sense to take steps to constrain it in certain critical cases. Note that this doesn't imply anything about private entities; it is simply a logical consequence of a functional need or ideal in the context of our current system.
As to private entities (businesses, etc) things vary based on context. Certainly I don't have any wish (for example) to force YouTube to host pornographic content. However there may well exist cases where broader freedom of expression (either the functional need or the ideal) requires protection against private entities.
California actually has such a law - an employer is not permitted to take actions that would "influence or tend to influence" their employees political activities outside of the workplace. This can get very complicated (as you might imagine) when an employer takes an official political stance on an issue.
US telecoms are also subject to regulation of this sort (ie common carrier laws). Personally I think that infrastructure companies (Cloudflare and other CDNs, as well as those providing the physical layer) ought to be subject to something broadly similar. It would help protect usage of and access to the underlying infrastructure for everyone by shielding the relevant companies from negative public opinion (public outrage campaigns accomplish nothing if the company is legally required to serve all customers).
> ... have turned into the cesspools of the internet
Broadly, I think you are tending to conflate infrastructure providers with social networks (and similar end user sites). There are important differences (for example) between FedEx and an online retailer, even though they might both be involved in getting a physical product to you.
Can you identify any government regulation that is enforceable where that enforcement isn't rooted in the monopoly on violence? Note that violence doesn't just mean capital or corporal punishment, it also depriving individuals of their possessions (for example, fines, asset forfeiture, etc), or their freedom. Each case where government can impose regulation carries the full weight of the threat of violence. If not for that threat of violence, the regulation would have no meaningful impact.
> However there may well exist cases where broader freedom of expression requires (...) protection against private entities.
I agree with you, but I don't think the answer here is to give government a mandate to dictate a policy of freedom of expression, I think the correct step is to evaluate whether a business has grown sufficiently large to constitute a threat to expression in society, and I think the model of being a monopoly is close to that. Disney is a stellar example; they don't own all of the movie studios, but their enormous size and ownership of the intellectual property rights to such a huge swath of cultural history for the last century, and the way they use those rights is growing increasingly harmful. I don't think it's right for government to say that Disney should start distributing adult films, but I certainly think it's appropriate to introduce regulations governing how a company like Disney can negotiate for, or more practically, dictate terms to movie theatres and retail outfits to carry or distribute their products.
> Broadly, I think you are tending to conflate infrastructure providers with social networks (and similar end user sites)
I do, but that is a separate discussion around the way that major platforms, cloud service providers, content studios, transit providers, and online retailers have become interconnected through acquisitions and competition that it has gotten to the point where I treat any online service as a "temporarily disenfranchised critical infrastructure". This is largely due to the idea that for many online services the end goal is to either achieve enough market share to become a competitor to another internet company, or to be acquired trying.
Suppose a guy walks into a grocery store with a shaved head and swastika tattoos. He wants to buy a bag of apples. The store owner doesn't like the cut of his jib and asks him to leave. That's fine, the guy goes to the grocery store across the street and buys the apples there.
The store owner across the street doesn't care what kind of tattoos a man has and is willing to sell apples to anybody. That's fine too, right? But then an angry mob descends on the store, filling the entrances and harassing the customers, accusing the store owner of being a complicit racist for doing business with Nazis. The store owner does some simple math and realizes that selling a bag of apples a month isn't worth this kind of trouble, so they start to turn the man away too.
Then the angry mob does the same thing to anyone who will do business with the man at all. He can't buy food, electricity, clothes, medicine, nothing, or the merchant will be the subject of a harassment campaign which will cost them more than they could get from his business.
This is clearly just as coercive as anything the government could do, if not more. Even they're not allowed to deprive you of food and medicine.
And as much as we might like the idea of making life impossible for Nazis, the target is only the current whim of the mob. Nothing guarantees that a given target is actually evil.
Or that they're paying enough attention to notice that the guy with the shaved head and swastika tattoos is a Buddhist monk and not a Nazi at all.
Which is why, when the government does it, they have an obligation to give you a fair trial and access to an attorney and respect your constitutional rights. None of which the mob gives you.
But yep, this is pretty much my thinking as well - that being ruled by the whims of the mob is hardly desirable from a societal standpoint because it ultimately results in many of the same issues that government restrictions on expression do.
Funny you bring up Fedex: https://www.cnn.com/2020/07/02/business/fedex-washington-red...
No they should not.
And the analogy works: if you're getting filtered on the basis of your content - at the packet level or not - then it's fundamentally against Net Neutrality.
Wait until the PR team at Verizon decides they don't want to publish your content because you're too vocal about BLM. Or, they will only support you if you do support BLM, or something rubbish. And now your VPN, your server host, caching technology provider, Telco, wireless provider, Visa/Amex, Video-conf provider - it's completely absurd.
FedEx won't ship 'PlanB' because it's a 'controversial' medicine? But they will in 3 states?
USPS will ship condoms everywhere but not in Utah where the local Union forbids it?
Alaska Big Oil gets their local VPN owners to ban Greentech related sites?
Trump's buddies on the Board of AT&T get them to threaten anyone hosting 'fake news' about Trump?
California Teachers Union Pension Fund presses Cloudflare to ban all hosting of anything related to law enforcement?
And FYI nobody is acting 'morally' - they're scared executives just trying to do whatever to hush people up and continue making money - a system which hands arbitrary power to arbitrary groups. This is not what anyone wants.
For services that are inherently 'content neutral' - the content should not be allowed to be a basis of discrimination.
For Social Media it's different, as there is an inherent association between the platform and it's users, but not for Cloudflare, or AWS or Verizon, Gmail for example.
There is no end to the insanity otherwise; we need basic, smart and clear regulation.
Edit: I should add 'and that's just the US'. Imagine when a very vocal, organised group wants to ban Arabs living in what is commonly referred to as 'Palestine' from using the term 'Palestine'. Or Serbian authorities from hosting content using the term 'Kosovo' in any way that reflects its supposed 'autonomy'. Or Greek companies ganging up on Macedonia's usage of the term 'Macedonia'. Or Greens in Germany from banning pro-Nuclear energy content. There are at least a handful of Tweeters who would want those things. It gets infinitely messy, very quickly.
Your panic is unwarranted anyways, because many of those countries do allow free speech restrictions, yet no runaway crazy banning has happened yet. Its just a fictional slippery slope made up to protect hate speech. Words don't protect against tyrany anyways; actually fighting against tyranny does.
Ah, and there is the ugly, authoritarian, Stalinist argument right there.
"We only ban, denounce, destroy those who are GUILTY of Hate Crimes, so how could you have any worry about that?"
People are banned all day long on Facebook, Twitter, YouTube, Stripe, comments sections etc. for any and all sorts of made up reasons that have nothing to do with hate speech. And FYI you can say things that are essentially hate speech, but in a political context may not be, so it's fine i.e. "White People Are a Disease" is a common thing on Twitter. No problems there apparently.
And speaking as a former German resident, they have some real Nazis there, none of this kind of 'made up suff' like in the US.
This is a problem that needs to be addressed.
'Freedom of Expression' is the #1 Amendment to the US Constitution and the #1 thing in the relatively new Canadian constitution (#11 in EU) for a very good reason: it took thousands of years for 'every day people' to be able to speak their minds without being shot.
We can't leave this issue up to corporate CEO's, mostly trying to read the Twitter tea leaves.
I agree completely. I think democratizing speech so that it cannot be controlled by any source of power, government or financial, does have real importance. But its not the most important right, compared to life or wellbeing: instead, its important because it supports what's actually important.
But I have a message of good news: it turns out speech can be lightly restricted while leading to less authoritarianism or cronyism, as can be seen from numerous examples, by simply restricting speech in a way that is kept controlled by the specific people who are subject to it.
I would argue that people should agree on what the right solution is before we enshrine it in law.
Also, when you think of the US Congress, do you think of "basic, smart and clear" regulations?
The positions are ideological and naive - they are obviously contradictory - it's only that when they are framed in certain ways, often 'straw-man-ish', do people think they have clarity on them.
For example, it's easy to understand why we don't want Verizon charging customers different prices based on the nature of the content. It's easy for us to want to 'ban Nazis' from Reddit. But both of those are unrealistic ideals (i.e. straw men) because neither contemplate the broader application. It's a very slippery slope.
It's not a big deal that Nazis get banned from somewhere, and maybe it's not a big deal that Verizon wants to design services such that your profitable business costs 2x on their network - but those bounds expand very rapidly.
On the front pages of CNN and Fox right now we have a completely ridiculous war over 'Goya' canned foods because of some arbitrary comments the founder made about Trump.
This issue is only going to get much worse and more complicated over time.
We have basically no choice but to define what kinds of businesses can use what thresholds, and try to infer what those thresholds are.
> This issue is only going to get much worse and more complicated over time.
Slippery slope is a fallacy. Humans aren't required to follow the precedent of previous decisions like SCOTUS justices are. People eventually see the problems with previous choices once the consequences hit a tipping point.
We are having multiple moral panics right now. They will pass.
People will eventually get more tolerant of others making mistakes on social media; it will accelerate once more of us know someone who directly loses their job/status/etc to a moral panic. Also, there are examples of people who were targeted by moral panic mobs (eg. Colin Kaepernick) who survived the outrage and overcame it.
Once enough people get {"cancelled", fired, boycotted, excommunicated, etc}, these calls carry less and less effect over time and social media mobs lose their power.
> On the front pages of CNN and Fox right now we have a completely ridiculous war over 'Goya' canned foods because of some arbitrary comments the founder made about Trump.
Those are bad gauges of anything except outrage-for-clicks. Those two webpages rile up their core viewership to generate eyeballs and collect web-ad revenue. Trump only got to the front of the 2016 Republican primary because the rest of the field was boring and he riled up moral panics and owned most of the news cycles since.
> but those bounds expand very rapidly
Not always. And this ignores the observable fact that there is almost always a regression to the mean. There are social frictions which prevent these moral panics from burning too long.
The opposite of 'fallacy' - it's literally happening all around you:
1) We now have major corporate support for 'social claims' with brands backing (and cashing in) on celebrities for their politics, not their skill at 'whatever', and supporting some causes surrounding a lot of public disruption. These have led to some groups trying to 'ban' others.
2) The attempted influencing of electoral outcomes via 'bad information' from outside agents, agitators, along with a plethora of information - with no guidance other than the whims of CEOs. There's all sorts of 'banning' on FB and YouTube and it's all pretty grey, often it can have major consequences.
3) CEO's brought to congressional hearings for these issues - this is a sign that 'ban' culture is now a top concern.
4) Weekly national 'pop culture wars' over which brand has transgressed which group, promoted widely in the press, resulting in calls for bans of some products or services.
5) 'Social concern' as the primary legitimacy for populism, even among those who are supposed to be popular for their acting, music, or athletics. In the last 10 years - ever 'celeb' has embarked upon a 'personal branding campaign' to imbue themselves with moral authority. There was always a little bit of this - but now it's 'the thing'. Go and have a look at social media, try 'Kristen Bell'. Every few posts are about some kind of moral concern. Otherwise: they could get the ban. A host of globally respected individuals had to sign an piece in Harper's. This is basically 'shocking' to anyone of a certain age, because intellectuals and their ilk used to be the one's fighting for freedom of expression against the authorities - now, 'ban' culture has infiltrated their organisations and become a populist issue.
6) There are disruptions and protests around what would be normal, common, mundane events: 'The Joker' review by the Daily Beast was entitled: "Everything About the Joker was Absolutely Infuriating". The 'infuriating' parts had nothing to do with the film, but rather the supposed politics of the film. The DB and others wanted the film to not be released is it were. There are weekly arguments over who can play who in film, supposed issues of representation, wars over the composition of selection committees - often resulting in the 'cancelling' of individuals.
Twitter and social media have created the ability for agitators to move giant waves of people in an emotional, populist direction, resulting in quite a lot of effort to ban, cancel, disassociate.
It's far worse than ever before, and it's going to be worse before it gets better.
I see no pathway for it 'getting better' anytime soon, because the pathways to outrage have formed, hardened and are now part of our culture.
Individually - yes - most of us are getting sick of it - but that won't change the fact of the activity in the news, on Twitter, and journalists 'calling for the resignation' of so and so for this and that with the basic moral impetus of 'someone on Twitter said it, therefore, that's how America feels'.
CloudFlare, Facebook - your startup - does not want to deal with this, there needs to be some kind of collective clarity.
[1] https://www.thedailybeast.com/everything-about-joker-is-abso...
2, 3) No guidance other than the whims of CEOs... oh and congress. How does that add up?
4) On a slippery slope from yellow journalism to...
I'm sorry for the low-effort post, but something like this is quite tricky to respond to. Much like your view of the thing you're railing against, any legit points you may have drown in a sea of outrage completely lacking perspective and full of non sequiturs.
The question isn't whether they should be allowed to do that, it's whether they should be forced to do that by other people.
Cloudflare was purely acting out of market based fear, there wasn't a hint of moral impetus. Literally he said: "I don't want people saying they won't work with us" - which is giving into the mob.
Where is the ACLU on this?
We were all screaming for Net Neutrality just a couple years ago.
It's up to communities and governments to make decisions on content, it would actually help if the government made it illegal for CloudFlare to refuse service to someone so long as they were within certain guidelines, thereby absolving businesses of this issue.
Imagine literally the marketing and PR teams of Verizon, Facebook, Cloudflare, AWS, Google, your rando VPN provider, getting to decide if they 'think they might not like you' or not, it's just too much.
For marketplaces like AppStore, it's fine. But for other services, this is not going to work. It's not the job of your Telco or Garbage Pickup do decide if your public statements are cool/uncool enough for their Instagram.
IIRC, the reason DailyStormer pissed off CloudFlare is because the users claimed (lied) about CloudFlare was somehow participating / sponsoring the site/activities. Sports teams have non-disparagement clauses; I don't see this as much different.
The only thing I'm not clear about is if it was just a rando user on DS that said the thing or some moderator/admin who can reasonably be said to represent the organization that runs the site.
Also, Macy's and Starbucks are allowed to deny their business to individuals at their discretion (subject to Equal Rights and Americans with Disabilities laws). Last I checked, being a Nazi is not a protected status (perhaps I'm wrong).
> We were all screaming for Net Neutrality just a couple years ago.
Different concept. This is about who a vendor chooses to allow as a customer. Common Carrier status is perhaps a closer comparison, but I think AT&T is allowed to drop a customer if they violate the AT&T ToS / contract.
Also, the CEO of CloudFlare went out of his way to publicize that this was a problem for the health of the internet and to start a conversation. Society didn't walk away from CloudFlare (eg. "vote with your feet/wallet") and Congress didn't choose to create any laws.
That would make net neutrality meaningless, surely. Just put something in the ToS that says you agree not to actually use your internet service.
> Also, the CEO of CloudFlare went out of his way to publicize that this was a problem for the health of the internet and to start a conversation. Society didn't walk away from CloudFlare (eg. "vote with your feet/wallet") and Congress didn't choose to create any laws.
Which is exactly the problem. America has given up on free speech, and CloudFlare was a prime mover in that shift.
Neither of those things mean you did the right thing. It’s really easy to pick on a widely unpopular minority group and not get laws passed against you or lose a noticeable amount of customers.
I find it hard to believe the right thing would involve either CloudFlare tolerating libel (my interpretation) about them by a customer or that we should always tolerate an unmitigated amount of free speech (at least the obviously political/religious speech originally envisioned) no matter the cost to {business, society, decency, morals, etc}.
What is "the right thing" to you in this situation?
> The tipping point for us making this decision was that the team behind Daily Stormer made the claim that we were secretly supporters of their ideology.
This is more about enforcing ToS and maintaining reputation than "an internet infrastructure company cancels DailyStormer because of their ideology".
The complicating factor is that CloudFlare was not the only internet infra company to drop them. DS were rapidly dropped or denied accounts from other companies during this news cycle, so they were effectively kept offline because none of the large infra companies they approached wanted to deal with the issue in a "free speech over all other concerns" kind of way.
[1] https://blog.cloudflare.com/why-we-terminated-daily-stormer/
The argument is that we've come to depend on privately owned backbone infrastructure in much the same way we depend on publicly owned roads. Furthermore, the operators of that infrastructure have shown themselves to be vulnerable to public outrage. Therefore, it's reasonable to ask if additional regulations might be a good idea.
A good analogy here might be to privately operated toll roads. Surely such a system shouldn't be allowed to discriminate against you based on (for example) your bumper stickers?
No, you can use any other service you want. CF has no private toll roads, in fact you can make your own 'toll' road right know. But if you have your private toll road you can forbid any bumper sticker you don't want on YOUR road.
If this existed in reality somewhere in the US (or wherever you happen to live) are you seriously saying that you would be ok with it?! Do you really not see the necessity of having neutral infrastructure?
> CF has no private toll roads
They are largely analogous. They transport your traffic from point A to point B for a fee. Yes, competitors exist. No, you can't just go build your own any more than you can simply whip up a viable Twitter or Facebook clone in a weekend. Claiming otherwise is either to be profoundly misinformed or to advance an argument in bad faith.
> > we've come to depend on privately owned backbone
> No, you can use any other service you want.
... which is also private and subject to the same pressures due to public outrage. Disputing that we have come to depend on privately owned basic infrastructure in general because we could switch to depending on a different piece of privately owned infrastructure is to again miss the point entirely. There is not (to the best of my knowledge) a government operated CDN or network backbone which you could make use of in order to avoid such concerns.
The line between private and public gets a bit blurry when talking about edge routing and CDNs. Especially edge routing.
Then go to your Government and ask them to make such a Service, look there is just no point arguing about that, as a private firm you don't have to serve everyone, especially groups that are potentially bad for your brand.
Dosninja.com for example (there are dozens, though not all Cloudflare protected).
Unlike "free speech" sites, the entire point of a DDoS-for-hire site is to suppress speech.
Nice to be able to get it both ways!
Nor was/is there any debate about the legality of doing so,
So the only reason for continuing to work for/with violent anti-semites was that they wanted to. Until, at some point, they changed their mind.
HN has a strange infatuation with this idea of avoiding responsibility by pretending to be powerless. But it's neither morally sound nor logically or legally coherent to pretend to be bound by some principles that are entirely of one's own making.
Precedents aren't limited to the legal system. Their actions changed the expectation (ie established a new precedent) for their behavior going forward.
> So the only reason for continuing to work for/with violent anti-semites was that they wanted to.
Or entertaining broader ideals surrounding freedom of expression? Or perhaps seeing the inherent dangers of living in a society dominated by public outrage? Or something else entirely?
> Until, at some point, they changed their mind.
Due, by their own admission, to financial concerns. Concerns which wouldn't exist if they and all their competitors were legally required not to discriminate based on otherwise legal content.
> HN has a strange infatuation with this idea of avoiding responsibility by pretending to be powerless.
I don't see that anywhere in this comment section? Rather, I see people objecting to actions which are legal but which they feel should not be.
I was worried a bit when I saw the initial IPO that the free tier would leave (despite promises it wouldn't) but that doesn't seem to be the case. Literally the only bad thing I've seen on the site is recently you switched from recaptcha to a new one that I had a real tough time with logging in today - it was a bit glitchy on my pc. The only suggestion I thought of as well would be a simple "maintenance mode" similar to the "I'm under attack mode" which would allow those of us without super-ha to quickly toggle on something to pop up a "sorry server"/site is down form maintenance page without having to mess with our proxies/web servers.
Anyway I know this comes across as totally kissing-ass but I just wanted to say thanks to someone who actually works there. Everyone fat fingers stuff every now and again,don't sweat it.
I really liked the stories of his skill when he was in his prime. Very inspiring.
I'm not sure what to take from this. But thank you for sharing it
I wonder if there are any relationship between FTD and how clever a person or how much brain a person uses.
Robert Sapolsky - Phineas Gage and frontal damage https://www.youtube.com/watch?v=5wKDXzk8Wm4
In all, it was a captivating read.
To be sure (and for the sake of internet rando completenessism), it does look like CF waited until the original SGI patent on the technique ran out. :)
https://patents.google.com/patent/US5732138
As ever, innovation was brick-walled until IP got out of the way.
That's right. See https://news.ycombinator.com/item?id=23860658
Right now: there is an issue with Safari users on the most recent iOS and OS X, where 3rd party cookies have now been disabled by default. We're working on a solution.
If that's your issue, you can fix on your side in the short-term by not using Safari, or by enabling 3rd party cookies.
(posted in case others are hit by this) using `requestStorageAccess()` on a user click event seems to work and that's what we're rolling out :/ https://gist.github.com/iansltx/18caf551baaa60b79206
I mean CDNs that will let you override the origin's cache instructions and do a decent job of DDOS protection, and whose feature list otherwise looks a little like Cloudflare's.
There are smaller CDNs out there. You can find them readily enough.
Which suggests to me that CDNs are already a commodity in some ways.
A second answer - there are a bunch of bottom-barrel commoditized pipe services. You likely haven't heard of them because they're so generic. They've existed before Cloudflare, and more will be created in the future https://www.citrix.com/products/citrix-intelligent-traffic-m...
Respectfully, the info in the S1 (flywheels, etc.) seem to be what sustains you _now_.
Maybe a better question is "how did you identify and kick off that flywheel?"
Market Opportunity
We believe our platform disrupts several large and well-established IT markets. The key markets that are addressed by our platform include VPN, internal and external firewalls, web security (including web application firewalls and content filtering), distributed denial of service (DDoS) prevention, intrusion detection and prevention, application delivery controls, content delivery networks, domain name systems, advanced threat prevention (ATP), and wide area network (WAN) technology. From our analysis based on IDC data, $31.6 billion was spent on those products in 2018, which is expected to grow to $47.1 billion in 2022, representing a compound annual growth rate of 10.5%. We also are actively developing new products to address adjacent markets including compute, storage, 5G, and Internet of Things (IoT) that are not included in the estimate of our addressable market.
Bandwidth isn't expensive compared to what most people are paying for it. Cloudflare is paying for the infrastructure as it is to handle attacks, so why not use it? As long as it doesn't affect paying customers then it's great marketing.
I may not have a full scope of the history, but my own experience with DDoS protection was quite different. Whilst providers offered anti ddos protection through GRE tunnels and dedicated machines behind DDoS appliances and a heavy null route hand, Cloudflare had a simple few-click solution that worked at the web application level making things a lot easier and, also, allowing for features like caching, and thus, CDN benefit from a global network. Further, they've maximized performance on their machines, and as a result, Cloudflare is wicked fast.
Cloudflare does what it does really well and has built additional services on their global network that make a lot of a sense and provide a lot of value.
Hats off to CF.
If AWS was able to offer a single click "DDOS protection and CDN" feature with similar pricing and features as Cloudflare then I'd consider it since most of our infrastructure is on AWS but at the moment they don't offer anything nearly as competitive. Just the Cloudfront bandwidth costs alone would dwarf our total infrastructure costs.
My reservation with Cloudflare is the concept of letting a third party MitM my SSL traffic. That and it's more expensive than a cheapo CDN like Stackpath if all you really care about is CDN (and Cloudflare isn't even really a good CDN, just a quick hack to speed up small static files).
Perhaps I have overlooked something?
So in this way it's possible to setup CDN with shared SSL for purely static files but not the app server itself; you don't have to give the keys to the whole kingdom so to speak and it's cheaper than Cloudflare at the basic level.
I was under the impression that the same result could be achieved with Cloudflare, or indeed nearly any CDN. Was I mistaken? Though you may not actually need a secret, private subdomain for static files with all CDNs.
Again, please let me know if I've made a mistake somewhere. I'd love to learn something this morning.
static.domain.com (CDN subdomain with auto provisioned TLS)
static-uncached.domain.com (private pass-through subdomain when CDN is missing a file)
www.domain.com (app server hosted wherever)
You're right that you could do something similar with other CDNs including Cloudflare (you can just set the www subdomain to "bypass Cloudflare" to accomplish a similar result), but I'm not aware of any way to use Cloudflare on a domain without forwarding your nameservers to them, effectively giving them complete control over the domain. At least with Stackpath I can host DNS wherever and simply point the subdomains I want at them.
Also, by the time you do the work to split static files into separate subdomains you might as well go with a dedicated CDN. One of the selling points of Cloudflare is for sites serving everything on one subdomain that they can forward to Cloudflare and get caching without any work.
It requires at least the Business level plan, though.
Not even "exceptionally difficult", but flat-out impossible. From the perspective of an observer, TLS sessions are random data. The protocol is specifically designed to defeat attempts to replay data -- a CDN is indistinguishable from an attacker in that sense.
I wonder what the Venn diagram of people who insist every website must use HTTPS for privacy reasons and Cloudflare users looks like.
When a 3rd party has access to your keys, their responsibilities to you are spelled out in your contract with them. That's true for CDNs as well as hosting companies.
For most websites today if someone can intercept traffic somewhere close to the server they don't even need the keys, they can just fake responses to pass CA validation and issue valid certificates with their own keys and MITM like there is no encryption.
And coldboot attacks performed by a hosting provider staff of dumping memory and finding keys isn't that realistic of a threat, just like putting servers into a locked cage on someone else's property isn't much of a protection.
If you press F12 in your browser and navigate to some major sites, you'll notice anywhere between 1-5 different CDN domain names that aren't directly related to the original host in any way.
That said bandwidth really isn't that expensive, at least if you're buying it at the scale that Cloudflare does. Many people seem to be used to the bandwidth prices of the large cloud hosters, which are really insane and have been marked up by a large multiplier to disincentivize people to transfer their data elsewhere for processing.
Suffice to say, that in over 5-6 projects I have added CF to, it's never worked out in my favour.
A million URLs isn't big?
It's worth noting Cloudflare also supports secondary DNS, but only for enterprise customers: https://blog.cloudflare.com/secondary-dns-a-faster-more-resi...
It's a learning process for all involved, really.
From the affected parties point of view, well, they should diversify their network a bit better. End users should hold those companies feet to the fire, not Cloudflare's.
Maybe, but making mistakes is far from negligence. Besides which, if a single person can accidentally break your system, at least at Cloudflare's scale, that's an organizational failure, not a personal failure.
I don't see the centralization as a positive, but I'm wondering what percentage of the websites that were taken offline see themselves as having no choice but to use Cloudflare in order to prevent themselves from being taken down anyway from malicious actors instead of by accident.
Let's also not forget Cloudflare in particular have been accused to host/hide the very bad boys that make protection from DDOS necessary in the first place. Whether or not that is the case, a quasi-monopoly leaves customers with no choice.
[1]: https://petri.com/microsoft-google-and-others-invest-in-clou...
The other one is called AWS.
What’s really puzzling is if the same companies spend money on active/passive failover for application and database servers while overlooking DNS single point of failure.
"Well, what do you want it to be?"
Give me a number and I'll tell you how much it will cost and how long it will take to get there.