Even if it is true, getting an enterprise account, which is still very reasonably priced, gives you dedicated IP addresses to your site, so this shouldn't be a huge concern. Generally when under fire, Cloudflare is also happy to get you up and running and talk finer details on billing for the long term later.
The bigger problem is their stance on Latern mentioned above.
Honestly a bit confused on the downvotes, was just trying to provide additional information for people who aren't familiar with working with CF. Anyone care to enlighten me?
This is the correct answer. Given that this website is part of the 'civil society' sphere, it likely already gets taxpayer money from the State Department. They should go to their funders - be they public or private - and ask for help with their mission - I'm not sure it's appropriate to ask individuals for help.
But the funders won't solve this problem though, only mitigate it. (though I agree it's something they definitely should do ASAP) Putting it out there like this, seems more likely to overcome the technical difficulties of it.
Yeah, but my understanding is that it helps a lot. CloudFlare masks the actual IP address of the web site, and distributes the load through the CloudFlare platform.
It's still possible to overload that capacity, but it's a lot more than what any standalone web server can take. Furthermore, the web server itself will stay online, as it isn't actually getting hit by the flood of requests.
You don't really understand the distributed part of DDoS. I've had services taken down my attacks and when doing a post mortem we could see the increase in traffic, but when accounting for frequency, our office was still the main user.
Sure, the target will not be affected, but CoralCDN network will, and it will not be able to handle the load of the attack. They don't have enough resources.
Could you define buckets of cash? On the low end, one gets 30TB/month (1gbps uplink, 200mbs guaranteed) included for ~60 Euro a month, or 100TB for ~160 Euro (for a single server, outbound) at Hetzner:
Such a single server, isn't the same as a CDN of course, just curious what you consider "buckets of cash". I see cachefly asks for 400 USD for 2/TB a month, for comparison.
From my experience, this is confusing the issue - the OP was talking about using their DDoS filtering service, not their CDN. This is done by Prolexic - who despite being acquired by Akamai, still seems to function fairly independently.
I've no idea what the CDN team are like, but it's not really fair to say don't use product X from company Y, because product Z sucks.
We've tried a number of DDoS filtering services, and Proxleic/Akamai has been the most professional and effective so far. They're also the most expensive, by a fair bit, but you get what you pay for I guess.
Actually the Prolexic product is one of the most innovative and effective one we've seen to date. DDoS attacks are not a commodity issue, you have to pay to play..Not sure how that makes Akamai horrible..
I can confirm that Akamai is a pain to deal with. Defense.Net as well. Cloudflare is what I would chose but they are siding with the Chinese gov't at this point.
Not sure what you're basing that statement on. Cloudflare very recently gave support to various parties involed in the pro-democracy movement in Hong Kong. They claimed to have weathered the largest DDoS in history in the process.
Agreed. Can't understand why I never see DosArrest on these lists though. They've been doing DDOS protection for a while and are good at it AFAIK. That said, I have no idea if they are interested in taking on the Chinese government.
isit possible to combine them? like, would my domain actually resolve if i change my name servers to cloudflare's, then cname it to incapsula? i'm just curious...
CloudFlare is probably not a good choice. They recently blocked access to a similar service, Lantern, per the linked WSJ article.
"CloudFlare, which offers content-delivery network services, said last week it cut off Lantern’s use of the service, saying it was unauthorized. “We don’t do anything to thwart the content restrictions in China or other countries,” said Matthew Prince, chief executive of CloudFlare. “We’re a tech company and we comply with the law.”"
LOL. Booters (that carry out DDoS attacks which are illegal in CloudFlare's own country)? No problem for CloudFlare. Trying to circumvent Chinese content restrictions? Nope, that's unauthorized!
Yep, it's hilarious. Cue Mr. Prince to come in here and give a half-assed explanation as to why it's not actually a contradiction, even though it clearly is.
Forgive my outburst, and maybe this sentiment won't be well received given the context, but I just find it to be downright unpatriotic for a US company like CloudFlare to stand there saying things like what Matt Prince says in your quote, when someone comes under attack by an opposing nation state.
Again, I realize this place isn't exactly a bastion for this kind of sentiment, but have some thought for freedom here, CloudFlare. The US may suck at helping a lot of the time, but if you've got a group of folks trying to deliver some good ol' freedom to a country like this, and that country is trying to shut them up, maybe put out a helping hand, or at least don't shut off service.
Not everyone desires to take part in geopolitics and become a tool of diplomacy. Some people just want to do their business and it's perfectly fine in my opinion. You can't force people to be patriotic or to feel a patriotic call.
I worked with a DDoS protection provider briefly. Suffice to say, it's quite possible that being public with identity can bring a significant chance of physical harm. Dunno about this particular case, or China, but for other people offering services to that continent-area, they had real concerns.
I got confused, are you talking about bringing freedom to the US ? :)
Kidding aside, not saying you're wrong, but companies that want to maximize profit take a too big of a risk alienating a possible big market...
In the case of Lantern, they were taking advantage of a bug in our system. Specifically, they were setting the SNI field (outside the encrypted packet) of a request to look like it was going to an actual CloudFlare customer (e.g., news.ycombinator.com) and then setting the host header inside the encrypted request to point to some restricted site. The bug was that we did not check that the SNI field matched the host header, which allowed Lantern to do what they were doing.
Lantern was not a customer of ours, instead they were exploiting this bug to essentially disguise traffic to look as if it was coming from one of our actual customers. One of our biggest concerns was that this would put CloudFlare's actual customers at risk of being blocked. And, beyond that, even if it weren't being used to avoid Internet restrictions, that someone could effectively impersonate the identity of a customer on our network is, per se, a flaw that we should patch. As soon as we became aware of the issue, we began matching the SNI header to the host header and, effectively, patched the bug.
We've always been very supportive of a free and open Internet. However, even if we support what someone is doing, we can't put our current customers at risk of collateral damage or keep open bugs that allow our network to be exploited.
Matthew Prince
Co-founder & CEO, CloudFlare
@eastdakota
> Lantern was not a customer of ours, instead they were exploiting this bug to essentially disguise traffic to look as if it was coming from one of our actual customers.
This makes a world of difference.
Just to confirm, does this mean that if the exact same attack had happened, but Lantern had been a CloudFlare customer, you wouldn't have shut them down?
Still curious about this quote:
“We don’t do anything to thwart the content restrictions in China or other countries,” said Matthew Prince, chief executive of CloudFlare. “We’re a tech company and we comply with the law.”"
So if Lantern were a customer, would the outcome still have been the same?
Well, if Lantern were a customer, then China could just block them like they do for any CF customer they want to block. The reason the bug was allowing people to get around the firewall was because they were pretending to access a site that wasn't blocked, but actually receiving content that was blocked.
Patriotism is not a justification for violating the law. Granted, modern politicians and civilians use patriotism to justify literally anything they want to do as long as it's in the name of the Homeland (similar to religious martyrs justifying anything they do as in the name of their God).
Usually patriotism is the last justification used by those who have nothing else to stand on, like the KKK trying to oppress African-Americans, or the Nativists trying to oppress Irish immigrants, or modern-day politicians who decry all Islamists as terrorists, or the border states trying to oppress migrant workers, etc. Each time they've exhausted all other excuses, Patriotism is the last justification for their actions. (I won't touch on Mao, Stalin, Hitler, etc because they're too tied to specific nationalist policies)
Personally, I wouldn't want to identify myself as a Patriot, because usually they're the ones standing on the wrong side of history.
> Patriotism is not a justification for violating the law.
Actually, it is. Patriotism, in being a Patriot, is a loaded word in the American (USA) context. Specifically, it is about doing what is good/right for the country and her citizens regardless of the law (i.e. British rule.) Or so says my recollection of American History. I mean... just look at the Patriots (rebels, in the british colloquialism) in the image on the wikipedia page for Patriot_(American_Revolution).
"The Oxford English Dictionary third definition of "Patriot" is "A person actively opposing enemy forces occupying his or her country; a member of a resistance movement, a freedom fighter."[1]. In this definition, if the alleged DDoSers are Chines, attempting to block the actions of a foreigner imposing influence in their own land, they are the more Patriotic? Which is why the term is utterly useless in this argument; Dare, any other.
> Usually patriotism is the last justification used by those who have nothing else to stand on[sic]
Patriotism isn't a word really, it's a neologism invented in the 18th century, probably attached to by the founders because the British hated the term. And while Patriotism's historical (and more ethical) definition might have been to defend the principles of one's country and the constitution given to the people, the modern definition is waaaaay different. At this point we should bring back the word Loyalist for the people who use Patriot to mean someone who blindly follows their government.
> "We don’t do anything to thwart the content restrictions in China or other countries," said Matthew Prince, chief executive of CloudFlare. "We’re a tech company and we comply with the law."
There's a popular idea that businesses (and people) have no responsibilities to anyone but themselves, because what they have is theirs; they built it themselves. But if you think about it a little, it's obviously false. Here's a more accurate statement:
We're a tech company whose success is completely dependent on the freedoms in our nation and many other nations around the world, and on the political and economic systems, infrastructure, and enormous wealth that blossomed from them. Without the sacrifices of blood and treasure by our predecessors of hundreds of years, and of many people today, we would not have these resources or opportunities today. There are many talented people born in many countries who, without these benefits, have no opportunity for success.
They can't sacrifice their company for every principle, every time, but there's a middle ground between that and 'we're just a tech company so we have no responsibilities'.
What counts as human rights is subjective. The UN says that it is a human right to receive and express opinions through any medium. Does that mean that we should hold "human rights" to be more important than revenue and forbid service providers from charging for access to information? Like the WSJ who wrote the article that's supposedly to blame here?
Actually, there is some evidence that slavery was very economically efficient. 'The Half Has Never Been Told' by E Baptist lays out how enslaved labour picked much more cotton than the then free labour.
That was then, though. Now, when skilled labor is of greater importance and unskilled labor of comparatively little value, I submit that things may well be very different.
I'm sure whatever country you do live and pay taxes in has at some point in the recent past violated someone's human rights, or at least they have in someone's opinion.
Given that the country you live in has violated human rights to some extent, and that you could reduce your contribution to that by not earning taxable income or purchasing taxable goods, is it not also your defacto position that you value revenue over human rights?
(My apologies in advance if you've ceased paying taxes, or buying anything taxable, or if somehow no one in the world believes your country has violated human rights, or might do so in the future, or your position is that revenue > human rights)
My point is that the world is a lot more grey than you make it out to be, and that you are also in some way likely valuing revenue over some human rights abuses.
I don't agree with a lot of your other posts, but I think we're on the same page here. When I watch the youtube video where Boston Dynamics demonstrates the stability of Spot by kicking the robotic dog all I think is, "Don't kick the dog bro". It's machine intelligence descendants are going to judge us, or maybe they won't care and will kill us all of anyway.
> There's a popular idea that businesses (and people) have no responsibilities to anyone but themselves.
It's not just a popular idea, it's why they are created as firms instead of philanthropies. There is a difference and it does matter what the expectations of the donors/investors are.
> We're a tech company whose success is completely dependent on the freedoms in our nation...
This sounds great but how is it reflected in company policies?
> They can't sacrifice their company for every principle, every time, but there's a middle ground between that 'we're just a tech company so we have no responsibilities'.
A company could easily make a statement to its investors about its moral stance on issues that it expects might harm the bottom line.
The company does have responsibility to its investors not to go rogue and burn cash just because it feels good. Most of the time the kind of corporate behavior that you praise is actually clever PR that costs the companies little.
It seems like you're saying that private enterprises should either completely divest themselves from any commitment to social responsibility (defined broadly as: "doing the right thing because it's the right thing, even when it may seem to go against the bottom line") -- or they might as well thrown in the towel and become philanthropies. Yes?
In other words -- on the blackhat-whitehat scale, it's either black- (or at least very charcoal-y grey-), or whitehat. But I just don't see modern, large companies generally acting that way -- not because they're led by altruists (they're certainly not); but because that's just not human nature (across the board). Most of us are greyhats (somewhere on the scale); and the behavior most business leadership I've either read about, or seen directly (behind closed doors) seems to fall somewhere on the greyhat scale, also.
That is: large business definitely aren't philanthropies -- but in general, most of them (even many of the traditional "bad boy" players like banks, big pharma, etc) -- aren't straight-up moral nihilists, either.
At least that's the way I observe these things. I could be wrong.
I don't disagree, but in terms of a framework for evaluating the behavior of businesses I think the following is reasonable:
Businesses should act within the law, and lawmakers and the public determine what legal safeguards are necessary. For example, if you start a restaurant you must comply with health code, fire code, etc. If you start a bank you need to keep a certain amount of risk capital, etc., etc.
One could argue that all dishes used by a restaurant should go through hospital level sterilization, or that banks should contain more risk capital than they are required to by law. Such arguments would be in the name of safety or quality.
One could similarly argue that restaurants should use at least 20% locally grown produce or that banks should lend 20% of capital to underprivileged groups. Such arguments are in the name of moral responsibility, etc., and lawmakers have actually implemented many such laws for banks.
For an investor who wishes to invest in a bank or a restaurant, there are many options. Being able to compare financials and other metrics will help the investor figure out which is the smartest investment (based on her risk appetite, etc., etc.)
Why might a restaurant decide to focus on locally grown produce or a bank decide to focus on its ethical treatment of subprime borrowers? Largely for PR/marketing reasons. If such marketing campaigns are successful, customers will flock to the bank or restaurant in question and (assuming they are still able to be profitable) make the bank or restaurant a more desirable investment.
One can pick any business and any metric that he thinks has moral significance and claim either "regulators should require x, y, or z" or that "that practice is horrible". One might be right... essentially ahead of the game morally from society's average.
The perception of moral progressiveness, like the font chosen for a brand, is one factor that helps determine a business's success. It may be the case that most of the meat we eat was raised in unconscionable conditions, or that 30% of imported electronics were assembled by modern serfs in near-slavery. The more we are aware of such things, the more likely firms are to make the most progressive choices.
Replying to my own comment (I'm too late to edit it).
My comment is about that concept in general, not about Cloudflare in particular. I don't pretend to know and won't judge Cloudflare based on one sentence taken out of context. For all I know they are excellent members of the community; in fact they could do be doing good things behind the scenes without publicizing it, which might be wise if they are as exposed to China as other commenters say.
Moreover, I'm not sure their argument makes sense even on its face. When they say they "comply with the law", which law do they mean? There are many thousands of lawmaking bodies. What if a small-town mayor passes a law outlawing the word "webinar"? What if China passes a law saying that DDOS protection is illegal worldwide? Or websites not properly registered with the Central Propaganda Department may not be carried by any network provider?
Cloudflare, I'm sure, will happily ignore any laws like that. The question is: why not ignore this too?
The wonderful thing about sovereignty is that a country's law's jurisdiction is whatever the country decides they it should be.
The degree to which they can practically enforce that jurisdiction becomes a game of relative power and how willing others are to constrain it, of course.
You forgot that their success is also built on the oppression of the Chinese, and low salaries etc that gives the rest of the world affordable hardware (and rare-earth minerals, metals, etc).
Not commenting on what CloudFare should or should not do, just indicating that your high horse actually has longer legs than you gave it credit for.
Historically Cloudflare have been quite strong in their support for free speech. For example, they run Project Galileo to protect public-interest sites against DDOS attacks: https://www.cloudflare.com/galileo
I'm guessing in this case it's simply a case of them choosing which battles to fight. They probably don't want to commit to run an open proxy for everyone in China to access banned websites. That would likely get them banned outright in China, which, for a CDN like Cloudflare, would really hurt their core business.
That's funny, cloudflare has a project to "Protect Free Expression Online"[1].
It even states:
"Often these attacks appear politically motivated — going after, for instance, citizen journalists reporting on government corruption. The promise of the Internet is that it is a great leveler — that anyone with an idea can reach a global audience. These attacks threaten that promise."
Excuse my ignorance.. is it really the case that website a gets ddosed, website a gets charged by amazon for the ddos traffic... and amazon isn't inclined to mitigate the attack? Will wonders never cease......?
For the record WSJ does use Akamai already for many of it's static resources. These are mostly requests for things that can not be cached already in some way. (I'm sure the crew is looking for some additional ways now =) Source: I used to work on WSJ and now within a different division of Dow Jones.
First a 2.6bn request/hour DDoS and then making the front page of HN... talk about getting flooded with requests.
Hopefully making the front page will at least get them the attention of Amazon or enough donations to cover the temporarily (absurdly) high operating cost.
Move to OVH -- they offer free DDoS protection as standard, and unlimited bandwidth. I just moved to OVH after getting DDoSed. I'm paying $109/month for a quad core 3.7Ghz Xeon, 64GB RAM, dual 2TB software RAID. It's a pretty sweet deal, and I haven't had any problems so far.
No, GB is perfectly correct when referring to memory. Just because some people standardized on Gibibyte and redefined gigabyte doesn't invalidate what memory makers have been doing for ever. JEDEC still uses GB, as they should.
I was unaware they had a monopoly on language usage. A byte is not an SI unit. Base2 is vastly more defensible and natural than base10. The real issue is that everyone in networking likes round base10 numbers divided over some arbitrary cesium fluctuations. This leads to 1GB / 1Gbps not being 8 seconds, which is confusing. But in JEDEC's and others defense: "why should I have to change, he's the one that sucks."
Trick question: How many bytes are in a 1.44MB floppy? In a 700MB CD? A 4GB USB stick? And a 480GB SSD? How many bytes/s bandwidth for 10Gb Ethernet?
I can't understand how people can be so delusional to think that randomly redefining prefixes is a good idea. It was never, ever used consistently, and JEDEC just should get rid of this idiocy.
Trick question in return: I'm selling a 256GB SSD. How many bytes am I legally required to deliver?
This is why there is a need for the two different prefixes. If I am selling a 256GiB SSD and you are selling a 256GB SSD, those are now differentiable on their face.
The IEEE and/or SI are in charge of language? The same IEEE that sends car insurance offers to its members, just so we're clear. JEDEC is also a standards group, and they disagree. What now, a Wikipedia editing war to determine the victor?
Standards bodies aren't anything magical, and I don't get the slavish following they seem to get. So an RFC says something, or another group mandates something. BFD. Unless you're expecting interop to work, use standards as you see fit. They aren't an ends unto themselves.
In this case "GB", when referring to RAM is unambiguous. Only disingenuous cloud providers or petty editors would use a base 10 interpretation.
Unrelated to the story at hand. Have you been recently DDoSed on OVH? I know that (at least some time ago) they just null-route/deactivate your account on spot with no notification on anything that looks like a DDoS.
I moved from iweb because they null-routed my server during a DDoS attack that happened for no apparent reason. (My guess is that it was a user upset at some user-generated content, so they decided to DDoS my server rather than notifying me).
This may be true. However, they are FAR more inclined to work with a customer before just flipping a switch like a lot of (for example) American hosting companies will. I have nothing but good things to say about OVH. They were helpful during a DDoS I had about a year ago throughout the incident, but I'm sure they would have nullrouted me had I not been so responsive to their support guys when things started going badly. "I'm sure" here is pure anecdote and based on zero evidence, but it's the feeling I got.
Been running a game server on OVH that has been repeatedly DDOS'd by random people. It may take a moment or two to kick in and cause some wonky connectivity around the times of the attacks, but their service has been able to take some attacks in the several GBps range without a problem while we have used them.
Yes, and the service has remained available to most users. It's just the initial flood knocked stuff out for about 2 minutes as the worker threads dropped their request queues.
They are using Arbor in their core network, and auto mitigate attacks on customers now. I am working with Arbor systems as well as others for DDoS protections where I am working now, and I can confirm that it works quite well.
Takes about 5 mins to kick in once an attack has started, which is very good.
100% worth it. I have monitoring/notification set up on my infrastructure, and have had two hardware failures since becoming an OVH customer a few years ago. OVH techs in the datacenter were on the scene before I got downtime notifications. Their techs must deal with this kind of thing all night, every night, because they seem to have this process of replacing hardware down to a science. This as opposed to my other dedicated providers, who basically don't give a shit, for the same hardware at the same price. OVH is an excellent budget investment!
I think the reason they can offer it so cheap is a combination of scale, cheap electricity, and tax breaks. iWeb is also in Quebec, so I figured there must be some tax break. Lo and behold there is one heck of a tax break:
It's actually even better than that because there's also a national program that pays up to 35% of R&D salaries: http://www.cameronhuff.com/blog/ontario-tax-credits-for-soft.... There are hundreds of smaller special programs that can pay even more (and they often stack).
If you don't need the latest hardware and enterprisey features, give soyoustart.com a look. It's an OVH company on same network with same DDOS protection that uses the 'OVH' brand recycled servers. About half the cost.
And if you don't need any support or server hardware, give kimsufi a look.
I do not work for OVH, but am so impressed by their automation and value in this field I refuse to use anyone else.
I don't think it's too good to be true. They have over 2Tbps of unused inbound capacity (or at least they did in 2013, and probably more now), and their routers mitigate attacks at the boundary of their network.
They wouldn't just be giving in, it would take less than a day to get up and running on CloudFlare. That is probably worth avoiding $30k bill, especially for a non-profit.
As for PR, there absolutely is good will towards organizations that take on censorship. This is the #1 story on HN, so a company stepping in and saying "we got this covered, free of charge" not only shows they are good people, but also gives would be customers an idea of the quick implementation cycle for their CDN + the load it can handle.
Real world use case + helping to stop censorship = great PR
This is exactly what I was thinking. I read the blog post, then came to the comments. I completely expected something along the lines of ... "Hey, I'm the VP @ SomethingTech, we like what you're doing and will help you mitigate the attacks for free if you get in touch with us." ... to be the top comment here.
My reaction was based on the interview where they say that the 30k$ charge was harming, not killing them. You're more knowledgeable than me on CloudFlare and it may very well be a strong option.
Now in a way their $ penalty with this attack is not unlike growth. To me it sounds like they invested in their branding and will have a name as a company that can sustain damage and keep its ground - important because their goal is to be credible in bringing content in/out China.
What if they had given in? They'd be yet another victim of an attack sort of linked with China.
I do see a business sense behind what I assumed was their moral stance and I definitely stand by it.
My reaction was based on the interview where they say that the 30k$ charge was harming, not killing them. You're more knowledgeable than me on CloudFlare and it may very well be a strong option.
Now in a way their $ penalty with this attack is not unlike growth. To me it sounds like they invested in their branding and will have a name as a company that can sustain damage and keep its ground - important because their goal is to be credible in bringing content in/out China.
What if they had given in? They'd be yet another victim of an attack sort of linked with China.
I do see a business sense behind what I assumed was their moral stance and I definitely stand by it.
Seems brilliant to me if you are trying to battle online censorship in China and get your message out. I agree that one of the mitigation services could benefit from some pro-bono services.
I agree with this. Silently allowing a bully/attacker do as they want is never acceptable. I find GreatFire.org's work commendable and hope they get the help they need.
For them, using a DDoS mitigation service might be harder than it seems. Their project, "Collateral Freedom", depends on having a set of web frontends with some special properties. Namely, 1) the provider of the said frontends must be willing to ignore takedown / access restriction requests from (Chinese, Russian, etc) authorities, and 2) the access to those frontends can't be easily blocked by the Great Firewall of China / Russia / etc without causing massive problems to some unrelated widely used services.
(full disclosure: I do devops for one of the Russian sites that is currently mirrored by Collateral Freedom)
If they're a non profit, then maybe this is exactly what their mission is. They can sit back and rack up a huge aws bill, then Amazon has to decide whether to collect. If they do collect, then that was what the money was for anyway. I'm guessing that giving in to bullying from the Chinese government is against their core principles here.
As xnull6guest already noted, they are most probably already financed by the US government, that's why they can be laid back: they are exactly doing what they are paid for.
It would be interesting to see which IP space the bulk of the traffic is coming from. Seems like it would be trivial for the Chinese government to spoof traffic from any IP within China...
I would appreciate it if people who down voted my comment explained to me if it's because i'm wrong or because they don't understand what i am trying to say or just for the heck of it :).
The issue is your comment doesn't really add any value. Anyone can paste a random link saying "you should use this" but it takes effort to explain why it would be useful to them. HN comments are about fostering discussion, so say something to be discussed :-)
Thank you for your input, but 1. this is not a random link, 2. the explanation is all inside the link i posted, so why to be redundant? anyways, if i didn't care for a discussion i wouldn't post it at the first place and i wouldn't later ask why was i downvoted, thank you again for your opinion lucaspiller.
This is interesting. Though Hacker News appears to not be blocked, it has been flagged as "Contradictory" on certain days. Is the Chinese government blocking certain news items?
If you click on the day in the calendar to look at the details, the "Contradictory" status is when some of their test servers work and others don't. For this site in particular, there are several servers showing a timeout and no data received. So it's possible that HN is being partially blocked.
Censorship in China isn't just an all or nothing thing. There are content filters at the city, providence, and country wide level. Separate filters for traffic leaving the country vs internal. etc.
It looks specifically like cURL's exit value and the downloaded page size varied on some requests. It would be interesting to know what the contradictory download size was, and what the curl exit value was.
All that is there when you click on the date, if you scroll rightwards. The exit values in the blocked sites are timeouts (CURLE_OPERATION_TIMEDOUT), and the broken download sizes are 0 bytes.
Isn't this title a little sensationalist without specifying who's under attack? I assumed it was a royal we and after clicking the link realized it was just this one site.
It's the title of the blog post that's being linked to. With that title being transplanted to HN, you need to consciously think about what the context is supposed to be, which is why HN lists the source domain.
That's not how international internet peering works though; it's entirely a set of private agreements. It's not even a given that the traffic has either Chinese IP source addresses or is actually from China.
The requests can vary. Sometimes they are simple GET requests, sometimes they're exploiting a cpu|memory|io-intensive process in the application and sometimes they can be reflected DNS attacks.
The problem is separating the legitimate requests from the bad requests. Sure I can see there are 5 Million requests to the main page of the app. But which are from poeple legitimately trying to use the application and which are from the botnet? You can't just do it by IP without running the chance you're going to cause problems for legitimate users.
There are ways to mitigate this but it requires being able to analyse current traffic and past traffic quickly, and at scale while having the expertise to set up firewalls and other filtering correctly
These days, DDoSes are not just lots of GET requests, because, as you said, they're fairly easy to mitigate. These days, the most common attacks are various UDP-based attacks, like NTP reflection [1]. You send a spoofed header to a server that speaks over UDP, and they send a huge amount of traffic to the victim.
There are things like amplification attacks (DNS or NTP) where a small amount of attack traffic generates a huge amount of target traffic. Even if the traffic is symmetric (1 byte to target = 1 byte from attacker) the bad guys tend to have botnets / malware-infected systems so they don't pay the cost of the attack side.
If the attackers are just sending GETs you might be able to filter them out, but if they're sending random packets you usually need upstream help to keep them from getting to you to begin with.
There are also things like Slowloris that just use up resources vs using bandwidth, those are harder to identify but easier to deal with on a server-by-server basis.
Honestly most companies I've worked for have blocked the entire Chinese IP block. Obviously Chinese hackers can use proxies but it honestly does cut out a TON of problems. The downside of course is you will never get a Chinese customer/viewer. In preparing to respond to this post I googled "China IP block" and pretty much every result was about how to configure .htaccess or iptables to block the entire country.
You should trace the attackers by tracing back. Work with your upstream providers and mailing lists (NANOG) and publicly shame these attackers. Likely, they are spoofing addresses - validate that and make sure you let the network know where the spoofed traffic is sourcing from to follow BCP38 and BCP84, defined by RFCs 2827 and 3704.
Assuming it is direct spoofed traffic and not a reflection, naming and shaming will accomplish nothing. Names of the big ISPs allowing this are not a secret.
Transit providers do not care. They make money on it, some people are using it legitimately, and they just don't care, for the most part. It's a well known problem. It might not hurt to mention it, but they know what they're doing.
State sponsored cyber warfare is something that happens at a scale that normal private companies and organizations are not well equipped to deal with. I wonder if there's someone you can alert in the government who can consider coordinating a counterattack or pass a backchannel note to the right people to cut it out.
Calling a simple DDoS cyber warfare is a bit hyperbolic when we know there are actual sophisticated attacks being carried out against far more important targets.
282 comments
[ 4.1 ms ] story [ 275 ms ] threadEdit: I don't actually know if this is still true and on what scale, I just know that's it's true for a website I use according to chinese users.
The bigger problem is their stance on Latern mentioned above.
Perhaps the US State Department might be inclined to help?
It's still possible to overload that capacity, but it's a lot more than what any standalone web server can take. Furthermore, the web server itself will stay online, as it isn't actually getting hit by the flood of requests.
Alternatively, call CloudFlare.
Don't just absorb this through Amazon.
Try Incapsula, GigENet, Blacklotus, Cloudflare first.
They're the most abhorrent combination of incompetence and arrogance that I've ever met in the tech industry.
They're the Oracle of the network world. Just don't waste your time on them. There's plenty better and cheaper CDNs nowadays.
Leave Akamai to the Governments and MegaCorps, they deserve each other.
https://www.hetzner.de/gb/hosting/produkte_rootserver/ex60
Such a single server, isn't the same as a CDN of course, just curious what you consider "buckets of cash". I see cachefly asks for 400 USD for 2/TB a month, for comparison.
We've tried a number of DDoS filtering services, and Proxleic/Akamai has been the most professional and effective so far. They're also the most expensive, by a fair bit, but you get what you pay for I guess.
"CloudFlare, which offers content-delivery network services, said last week it cut off Lantern’s use of the service, saying it was unauthorized. “We don’t do anything to thwart the content restrictions in China or other countries,” said Matthew Prince, chief executive of CloudFlare. “We’re a tech company and we comply with the law.”"
http://www.wsj.com/articles/u-s-cloud-providers-face-backlas...
I'm not very impressed. Maybe someone from CloudFlare is around to defend that position further.
Again, I realize this place isn't exactly a bastion for this kind of sentiment, but have some thought for freedom here, CloudFlare. The US may suck at helping a lot of the time, but if you've got a group of folks trying to deliver some good ol' freedom to a country like this, and that country is trying to shut them up, maybe put out a helping hand, or at least don't shut off service.
Come on...
> Due to the sensitive nature of the content on our web sites we prefer to remain anonymous at this point
If they want help they need to be transparent about who they are and what their objective is. One man's tool of diplomacy is anothet man's... etc.
In the case of Lantern, they were taking advantage of a bug in our system. Specifically, they were setting the SNI field (outside the encrypted packet) of a request to look like it was going to an actual CloudFlare customer (e.g., news.ycombinator.com) and then setting the host header inside the encrypted request to point to some restricted site. The bug was that we did not check that the SNI field matched the host header, which allowed Lantern to do what they were doing.
Lantern was not a customer of ours, instead they were exploiting this bug to essentially disguise traffic to look as if it was coming from one of our actual customers. One of our biggest concerns was that this would put CloudFlare's actual customers at risk of being blocked. And, beyond that, even if it weren't being used to avoid Internet restrictions, that someone could effectively impersonate the identity of a customer on our network is, per se, a flaw that we should patch. As soon as we became aware of the issue, we began matching the SNI header to the host header and, effectively, patched the bug.
We've always been very supportive of a free and open Internet. However, even if we support what someone is doing, we can't put our current customers at risk of collateral damage or keep open bugs that allow our network to be exploited.
Matthew Prince Co-founder & CEO, CloudFlare @eastdakota
This makes a world of difference.
Just to confirm, does this mean that if the exact same attack had happened, but Lantern had been a CloudFlare customer, you wouldn't have shut them down?
Still curious about this quote: “We don’t do anything to thwart the content restrictions in China or other countries,” said Matthew Prince, chief executive of CloudFlare. “We’re a tech company and we comply with the law.”"
So if Lantern were a customer, would the outcome still have been the same?
Usually patriotism is the last justification used by those who have nothing else to stand on, like the KKK trying to oppress African-Americans, or the Nativists trying to oppress Irish immigrants, or modern-day politicians who decry all Islamists as terrorists, or the border states trying to oppress migrant workers, etc. Each time they've exhausted all other excuses, Patriotism is the last justification for their actions. (I won't touch on Mao, Stalin, Hitler, etc because they're too tied to specific nationalist policies)
Personally, I wouldn't want to identify myself as a Patriot, because usually they're the ones standing on the wrong side of history.
Unless you were just trolling.... ;-)
Actually, it is. Patriotism, in being a Patriot, is a loaded word in the American (USA) context. Specifically, it is about doing what is good/right for the country and her citizens regardless of the law (i.e. British rule.) Or so says my recollection of American History. I mean... just look at the Patriots (rebels, in the british colloquialism) in the image on the wikipedia page for Patriot_(American_Revolution).
"The Oxford English Dictionary third definition of "Patriot" is "A person actively opposing enemy forces occupying his or her country; a member of a resistance movement, a freedom fighter."[1]. In this definition, if the alleged DDoSers are Chines, attempting to block the actions of a foreigner imposing influence in their own land, they are the more Patriotic? Which is why the term is utterly useless in this argument; Dare, any other.
> Usually patriotism is the last justification used by those who have nothing else to stand on[sic]
Thus was it written.
[1]http://en.wikipedia.org/wiki/Patriot_(American_Revolution)
edit: add ambiguous ?
I might have to reconsider mine and my clients choice of providers for this very purpose.
There's a popular idea that businesses (and people) have no responsibilities to anyone but themselves, because what they have is theirs; they built it themselves. But if you think about it a little, it's obviously false. Here's a more accurate statement:
We're a tech company whose success is completely dependent on the freedoms in our nation and many other nations around the world, and on the political and economic systems, infrastructure, and enormous wealth that blossomed from them. Without the sacrifices of blood and treasure by our predecessors of hundreds of years, and of many people today, we would not have these resources or opportunities today. There are many talented people born in many countries who, without these benefits, have no opportunity for success.
They can't sacrifice their company for every principle, every time, but there's a middle ground between that and 'we're just a tech company so we have no responsibilities'.
EDIT: It turns out Lantern was using an exploit at Cloudflare [+], and wasn't a customer. My apologies /u/eastdakota.
[+] https://news.ycombinator.com/item?id=9234367
That was then, though. Now, when skilled labor is of greater importance and unskilled labor of comparatively little value, I submit that things may well be very different.
Given that the country you live in has violated human rights to some extent, and that you could reduce your contribution to that by not earning taxable income or purchasing taxable goods, is it not also your defacto position that you value revenue over human rights?
(My apologies in advance if you've ceased paying taxes, or buying anything taxable, or if somehow no one in the world believes your country has violated human rights, or might do so in the future, or your position is that revenue > human rights)
My point is that the world is a lot more grey than you make it out to be, and that you are also in some way likely valuing revenue over some human rights abuses.
It's not just a popular idea, it's why they are created as firms instead of philanthropies. There is a difference and it does matter what the expectations of the donors/investors are.
> We're a tech company whose success is completely dependent on the freedoms in our nation...
This sounds great but how is it reflected in company policies?
> They can't sacrifice their company for every principle, every time, but there's a middle ground between that 'we're just a tech company so we have no responsibilities'.
A company could easily make a statement to its investors about its moral stance on issues that it expects might harm the bottom line.
The company does have responsibility to its investors not to go rogue and burn cash just because it feels good. Most of the time the kind of corporate behavior that you praise is actually clever PR that costs the companies little.
It may not be coded into law, but it is still a true statement.
A false dichotomy.
I'm curious about your reasoning behind this statement.
In other words -- on the blackhat-whitehat scale, it's either black- (or at least very charcoal-y grey-), or whitehat. But I just don't see modern, large companies generally acting that way -- not because they're led by altruists (they're certainly not); but because that's just not human nature (across the board). Most of us are greyhats (somewhere on the scale); and the behavior most business leadership I've either read about, or seen directly (behind closed doors) seems to fall somewhere on the greyhat scale, also.
That is: large business definitely aren't philanthropies -- but in general, most of them (even many of the traditional "bad boy" players like banks, big pharma, etc) -- aren't straight-up moral nihilists, either.
At least that's the way I observe these things. I could be wrong.
Businesses should act within the law, and lawmakers and the public determine what legal safeguards are necessary. For example, if you start a restaurant you must comply with health code, fire code, etc. If you start a bank you need to keep a certain amount of risk capital, etc., etc.
One could argue that all dishes used by a restaurant should go through hospital level sterilization, or that banks should contain more risk capital than they are required to by law. Such arguments would be in the name of safety or quality.
One could similarly argue that restaurants should use at least 20% locally grown produce or that banks should lend 20% of capital to underprivileged groups. Such arguments are in the name of moral responsibility, etc., and lawmakers have actually implemented many such laws for banks.
For an investor who wishes to invest in a bank or a restaurant, there are many options. Being able to compare financials and other metrics will help the investor figure out which is the smartest investment (based on her risk appetite, etc., etc.)
Why might a restaurant decide to focus on locally grown produce or a bank decide to focus on its ethical treatment of subprime borrowers? Largely for PR/marketing reasons. If such marketing campaigns are successful, customers will flock to the bank or restaurant in question and (assuming they are still able to be profitable) make the bank or restaurant a more desirable investment.
One can pick any business and any metric that he thinks has moral significance and claim either "regulators should require x, y, or z" or that "that practice is horrible". One might be right... essentially ahead of the game morally from society's average.
The perception of moral progressiveness, like the font chosen for a brand, is one factor that helps determine a business's success. It may be the case that most of the meat we eat was raised in unconscionable conditions, or that 30% of imported electronics were assembled by modern serfs in near-slavery. The more we are aware of such things, the more likely firms are to make the most progressive choices.
My comment is about that concept in general, not about Cloudflare in particular. I don't pretend to know and won't judge Cloudflare based on one sentence taken out of context. For all I know they are excellent members of the community; in fact they could do be doing good things behind the scenes without publicizing it, which might be wise if they are as exposed to China as other commenters say.
Cloudflare, I'm sure, will happily ignore any laws like that. The question is: why not ignore this too?
Jurisdiction.
The degree to which they can practically enforce that jurisdiction becomes a game of relative power and how willing others are to constrain it, of course.
Not commenting on what CloudFare should or should not do, just indicating that your high horse actually has longer legs than you gave it credit for.
[1]: https://blog.cloudflare.com/one-more-thing-keyless-ssl-and-c...
Same thing with anyone who operates in China. The only difference is China is more transparent with their demands.
Response from CloudFlare's CEO here: https://news.ycombinator.com/item?id=9234367
That's really rich, considering CloudFlare happily takes money from booter services. These guys are scum, I have no idea why HN fawns over them.
I'm guessing in this case it's simply a case of them choosing which battles to fight. They probably don't want to commit to run an open proxy for everyone in China to access banned websites. That would likely get them banned outright in China, which, for a CDN like Cloudflare, would really hurt their core business.
"Often these attacks appear politically motivated — going after, for instance, citizen journalists reporting on government corruption. The promise of the Internet is that it is a great leveler — that anyone with an idea can reach a global audience. These attacks threaten that promise."
[1] https://blog.cloudflare.com/protecting-free-expression-onlin...
Hopefully making the front page will at least get them the attention of Amazon or enough donations to cover the temporarily (absurdly) high operating cost.
Not to mention when the people who don't like you have the resources of nation states.
――――――
¹ — http://www.wolframalpha.com/input/?i=1%20GiB%20to%20MiB%20an...
² — https://en.wikipedia.org/wiki/IEEE_1541-2002
Disk space even obeys this now even though it was only done by marketing so they could reduce the amount bits delivered while charging the same.
I can't understand how people can be so delusional to think that randomly redefining prefixes is a good idea. It was never, ever used consistently, and JEDEC just should get rid of this idiocy.
This is why there is a need for the two different prefixes. If I am selling a 256GiB SSD and you are selling a 256GB SSD, those are now differentiable on their face.
Standards bodies aren't anything magical, and I don't get the slavish following they seem to get. So an RFC says something, or another group mandates something. BFD. Unless you're expecting interop to work, use standards as you see fit. They aren't an ends unto themselves.
In this case "GB", when referring to RAM is unambiguous. Only disingenuous cloud providers or petty editors would use a base 10 interpretation.
Oh really? :-)
http://www.ovh.com/ca/en/a1171.protection-anti-ddos-service-...
I moved from iweb because they null-routed my server during a DDoS attack that happened for no apparent reason. (My guess is that it was a user upset at some user-generated content, so they decided to DDoS my server rather than notifying me).
Takes about 5 mins to kick in once an attack has started, which is very good.
https://www.ovh.com/us/dedicated-servers/enterprise/2014-MG-...
Thanks for posting :-)
I've been looking for provider possibilities for my next failed startup. I'm not sure how they can deliver for that price but who am I to complain!
http://www.investquebec.com/quebec/en/financial-products/smb...
Basically the Quebec government pays 24% of your company's wages, up to $83,333 per employee!
And if you don't need any support or server hardware, give kimsufi a look.
I do not work for OVH, but am so impressed by their automation and value in this field I refuse to use anyone else.
http://www.webhostingtalk.com/forumdisplay.php?f=2 (and most of the rest of the site).
(1gbps ~ 300 TB, 100mbit ~30TB/month)
First thing I'd have done is take the site offline and call the various DDoS mitigation services (Incapsula, Cloudflare, etc.).
Pretty surely most of them would gladly pick up the slack here, given the free PR (possibly even in mainstream press) they get in return.
And pretty certain that there's zero PR value in those stories.
As for PR, there absolutely is good will towards organizations that take on censorship. This is the #1 story on HN, so a company stepping in and saying "we got this covered, free of charge" not only shows they are good people, but also gives would be customers an idea of the quick implementation cycle for their CDN + the load it can handle.
Real world use case + helping to stop censorship = great PR
Now in a way their $ penalty with this attack is not unlike growth. To me it sounds like they invested in their branding and will have a name as a company that can sustain damage and keep its ground - important because their goal is to be credible in bringing content in/out China.
What if they had given in? They'd be yet another victim of an attack sort of linked with China.
I do see a business sense behind what I assumed was their moral stance and I definitely stand by it.
Now in a way their $ penalty with this attack is not unlike growth. To me it sounds like they invested in their branding and will have a name as a company that can sustain damage and keep its ground - important because their goal is to be credible in bringing content in/out China.
What if they had given in? They'd be yet another victim of an attack sort of linked with China.
I do see a business sense behind what I assumed was their moral stance and I definitely stand by it.
Unless your attitude is worth $20 USD per minute to you then yes, that's very much what you want to do.
And pretty certain that there's zero PR value in those stories.
Well, Greatfire seems to be betting a 6 figure stake on the opposite here...
(full disclosure: I do devops for one of the Russian sites that is currently mirrored by Collateral Freedom)
http://en.wikipedia.org/wiki/Coral_Content_Distribution_Netw...
Take a look here: https://en.greatfire.org/news.ycombinator.com
That would certainly be my guess.
Edit: never mind, figured it's obvious.
Aren't DDoS requests pretty much simple GET requests? Is it not possible to determine which requests to serve and which ones to ignore?
There are ways to mitigate this but it requires being able to analyse current traffic and past traffic quickly, and at scale while having the expertise to set up firewalls and other filtering correctly
https://blog.cloudflare.com/understanding-and-mitigating-ntp...
If the attackers are just sending GETs you might be able to filter them out, but if they're sending random packets you usually need upstream help to keep them from getting to you to begin with.
There are also things like Slowloris that just use up resources vs using bandwidth, those are harder to identify but easier to deal with on a server-by-server basis.
http://gizmodo.com/china-finally-admits-it-has-an-army-of-ha...
http://www.linuxjournal.com/content/back-dead-simple-bash-co...
from http://deflate.medialayer.com/
http://www.google.com/ideas/projects/project-shield/