It seems that CloudFlare's DNS is down, and affecting NameTerrific as we have a CNAME record pointing to them. I had to change the CNAME record to get our site working again.
EDIT: Based on Twitter search, all CloudFlare sites seem to be down.
Just discovered this problem on my company website, thanks for the update.
--
Seems to be getting better now - intermittent 502's and 504's, the occasional load (speed is choppy though)
--
Back to near-instant load times, great job guys! I look forward to the blog entry on this (P.S. It might be worth hosting your status page elsewhere in the future, whilst this might not happen very often, it's when it does that your site needs to be working, the lack of redundancy here is startling.)
Even their status page is down. And sure - all my sites too. And funny part is that for most of my sites i have stopped Clouflare features and use just their DNS. Never thought that I will fail because of DNS not being available.
btw - sites are back. I hope they will not go down.
Why I have switched off services? Because once i enabled them sites got slower, not faster. Sure - I doubt that most users noticed that, but for example, if i checked Pingdom or Google Crawl Stats - 'Time spent downloading a page' situation was very clear. With Cloudflare it took Google 2x more time to download page than without. I had no time to investigate why thats so, but after switching Clouflare off i was again back to 500ms.
Edit: Sites are not back. I guess that work the ones for which i have DNS cached. Lets wait...
When I briefly did that test, I didn't find much difference on Pingdom; but I like some of their other services and I assume they can handle burst activity at least as well as my own server. So I'm a big fan overall and keep using them.
I wish there were a business model for a third party site-monitor and site-uptime service, which let the site owner do more than just post updates, but also prevented the site owner from lying about historical data.
Basically New Relic (that actually worked) + Pingdom + Internet Archive + Twitter + status.example.com.
Doesn't really matter -- as long as the odds outages at both sites are independent, and relatively low, it won't happen at both at the same time, usually. You can make the status service pretty reliable, cheaply, compared to most other services, and if it loses the central servers, the remote monitoring nodes can still store test information, so when the service comes back up, the historical record should be accurate.
That part surprised me. I thought it was common for ISPs to not use their own services for critical web presence, i.e. places users might visit when it's down.
I have no idea what is wrong, or how long they will take to fix it, but I'd imagine that CloudFlare has significantly better network engineers than the average company, and so they will fix it in far less time than the average company would fix the same problem.
But, for the amount of money many paid customers are paying them (in essence, anyone at that $3k/mo level that includes the critical 24/7 phone support), you can actually get an account with a company like CDNetworks or Akamai (if nothing else, with a reasonable CDN like EdgeCast) and have still-better network engineers than CloudFlare.
Also, even if you are using them for free: they aren't replacing people you have in house... they are an additional component that can independently fail, in addition to any of the things that would have caused your average company's network engineers to fail. They don't promise to cache enough content to replace much of your infrastructure.
“While we have not completed our investigation, we believe this incident was triggered by a product issue that Juniper identified last October, when a patch was also made available"
Good network engineers tend to apply newly release patches. This vulnerability was documented for almost half a year...
As someone who hosts hundreds of PAID sites with CloudFlare this is pretty unacceptable. I'm giving them thousands of dollars so that this doesn't happen. Will probably be moving off unless they have some very good reasoning behind a world-wide shutdown of a geo-redundant service...
Fair enough. But then what is the point of paying for enterprise support if all you get is the same generic response a normal punter gets? If they cant offer better, then enterprise support seems poor value at best.
Are we sure that if CloudFlare wasn't so popular here, the word scam wouldn't be used?
So you're saying you want them to say the same thing but reword it since you're enterprise? If they don't know the root cause, they don't know the root cause.
My startup NameTerrific can support instantaneous DNS updates in a geo-redundant Anycast infrastructure. As long as your TTL is sufficiently low (<300), the impact is quite limited as propagation time is negligible at NameTerrific.
EDIT: Sorry guys. We got some issues with a gem after installing the recently updated ruby2.0.0p0. The unicorn workers were timing out. TerrificDNS is completely unaffected and the site is already running again.
Well, we have already soft launched our own TerrificDNS Anycast and it has replaced the Route 53 solution. TerrificDNS platform is running on Redis + PowerDNS.
Yep, this is definitely something I'll do in the future and I guess it was my fault for trusting a single service... even one that is designed around keep your site up/redundant.
Would love some details on how to do this, I've been a (business) CF customer for a while now and have never seen this service and I haven't turned up anything googling/scouring their website for it just now.
I'm so happy we didn't go on that wave. Redirecting your DNS to someone else seems like a bad idea in any case. In any case, what do they do, that I could not have done with Varnish?
They are only sort of a CDN; in addition to DNS, they specialize in site optimization via content transformation (a la mod_pagespeed in the cloud) and "DDoS protection" (which is pretty much them replacing your website for new users with some JavaScript that tries to determine if you are a legitimate client).
They don't promise to cache much, and they in fact don't: even on very simple single-page static information websites, such as evasion.com, they have an abysmally poor 81% cache hit ratio. They don't help at all with dynamic content due to having poorly located nodes and lots of heavyweight code running in their proxy. Their lack of many nodes in good positions (compared to something like CDNetworks or Akamai, they are one or two orders of magnitude smaller) also means they can't provide very good latency even for the times when they actually happen to have something in cache.
(Note: if someone is now going to say CDNs don't generally do well with dynamic content, they are wrong: normal CDNs actually improve the performance of dynamic content incredibly by maintaining large-window pre-connected HTTP sessions to customer origin servers, often over private networks that already provide better bandwidth: you can easily see 2x latency improvements with a normal CDN even for fully dynamic content).
So, they really shouldn't be compared with a "CDN": they have an interesting service that actually provides something valuable for many key use cases (4chan comes to my mind: in essence, something that is actually likely to experience a true DDoS attack sufficiently often and with sufficiently little provocation that it makes sense to add an external system to your infrastructure), but if you need a "CDN" there are many more reasonable alternatives that don't have as many moving parts and are thereby going to break much less often (and, if they actually do, should break only in localized regions).
I use a pretty major forum that has a huge amount of traffic. The owner migrated it to CloudFlare. For the past 5-6 weeks the site has 50% of its request go to a 'Sorry xyz is not available right now'
You should make sure that the site is not actually returning 500's for those request. We had some similar issues when we first started using the service
its sunday! seems like they pushed another faulty update (like last time)! yep confirmed, all is down including their own site, thats pretty fucked up, when they dont even have offsite status! good thing i dont use cloudflare in all my sites...
I'm getting alternating 504 and 502 errors for the main website. The server signature is "cloudflare-nginx", so it's definitely getting through, though.
151 comments
[ 3.1 ms ] story [ 237 ms ] threadEDIT: Based on Twitter search, all CloudFlare sites seem to be down.
Edit: 5:01 EST ..seems to be up and down according to mass pingdom messages.
- There is a global problem that affects the CloudFlare proxy and DNS services.
- The problem appears to be due to bad routing.
- We are working to restore correct routes in order to bring both DNS and proxy services back online.
- The operations and networking team are all online and treating this as an emergency.
- We do not have an ETA on the response time but will continue to post updates via Twitter as we learn more.
UPDATE. Sites are being restored now. DNS is operating.
I don't know all the details as bugging the network team while they were fixing wasn't going to help. We'll get a postmortem blog post up.
--
Seems to be getting better now - intermittent 502's and 504's, the occasional load (speed is choppy though)
--
Back to near-instant load times, great job guys! I look forward to the blog entry on this (P.S. It might be worth hosting your status page elsewhere in the future, whilst this might not happen very often, it's when it does that your site needs to be working, the lack of redundancy here is startling.)
Why I have switched off services? Because once i enabled them sites got slower, not faster. Sure - I doubt that most users noticed that, but for example, if i checked Pingdom or Google Crawl Stats - 'Time spent downloading a page' situation was very clear. With Cloudflare it took Google 2x more time to download page than without. I had no time to investigate why thats so, but after switching Clouflare off i was again back to 500ms.
Edit: Sites are not back. I guess that work the ones for which i have DNS cached. Lets wait...
They should host this page on a third party provider.
I wish there were a business model for a third party site-monitor and site-uptime service, which let the site owner do more than just post updates, but also prevented the site owner from lying about historical data.
Basically New Relic (that actually worked) + Pingdom + Internet Archive + Twitter + status.example.com.
reliable, accurate, high-frequency
I have no idea what is wrong, or how long they will take to fix it, but I'd imagine that CloudFlare has significantly better network engineers than the average company, and so they will fix it in far less time than the average company would fix the same problem.
Also, even if you are using them for free: they aren't replacing people you have in house... they are an additional component that can independently fail, in addition to any of the things that would have caused your average company's network engineers to fail. They don't promise to cache enough content to replace much of your infrastructure.
“While we have not completed our investigation, we believe this incident was triggered by a product issue that Juniper identified last October, when a patch was also made available"
Good network engineers tend to apply newly release patches. This vulnerability was documented for almost half a year...
We are in the same boat, but guess where I have the phone number for enterprise support? Webmail on our domain. Lesson learned.
Are we sure that if CloudFlare wasn't so popular here, the word scam wouldn't be used?
When disasters like this happen, a quick DNS change can be a life-saver.
EDIT: Sorry guys. We got some issues with a gem after installing the recently updated ruby2.0.0p0. The unicorn workers were timing out. TerrificDNS is completely unaffected and the site is already running again.
"We're sorry, but something went wrong."
http://www.nameterrific.com
"502 Bad Gateway
nginx/1.2.7"
Details much appreciated!
EDIT: Spoke too soon: https://support.cloudflare.com/entries/22054357-how-do-i-do-... shame it's so convoluted
They don't promise to cache much, and they in fact don't: even on very simple single-page static information websites, such as evasion.com, they have an abysmally poor 81% cache hit ratio. They don't help at all with dynamic content due to having poorly located nodes and lots of heavyweight code running in their proxy. Their lack of many nodes in good positions (compared to something like CDNetworks or Akamai, they are one or two orders of magnitude smaller) also means they can't provide very good latency even for the times when they actually happen to have something in cache.
(Note: if someone is now going to say CDNs don't generally do well with dynamic content, they are wrong: normal CDNs actually improve the performance of dynamic content incredibly by maintaining large-window pre-connected HTTP sessions to customer origin servers, often over private networks that already provide better bandwidth: you can easily see 2x latency improvements with a normal CDN even for fully dynamic content).
So, they really shouldn't be compared with a "CDN": they have an interesting service that actually provides something valuable for many key use cases (4chan comes to my mind: in essence, something that is actually likely to experience a true DDoS attack sufficiently often and with sufficiently little provocation that it makes sense to add an external system to your infrastructure), but if you need a "CDN" there are many more reasonable alternatives that don't have as many moving parts and are thereby going to break much less often (and, if they actually do, should break only in localized regions).
Also, randomly useful filters like adding the user's country to the request header, tweaking outbound images, and auto-injecting google Analytics.