Posted this because I can't get at the management page anymore and my NJ server has now gone down too. No word yet from them if it's being affected now or if it's just me.
yea didnt' last long, but was a black hole for about 10 minutes, lost all of it's connection to outside services too with no indication of anything wrong on the server itself.
Linode has been having DoS issues over the past few months. It feels like they've really been slipping over the past six. Has anyone left Linode because of the DoS issues? Are there other good alternatives that folks love out there?
I haven't but have been looking. Digital ocean is competitive on everything but storage space for me, i'd end up paying the same for 10GB less or pay for more ram/cpu than i need to get the same storage. (though this is largely because of the bulk discount from paying for things a year in advance).
I chose Digital Ocean recently because my needs are rather modest right now and they are cheaper at the low end. I imagine that Linode will get this worked out. It's probably only a matter of time until DO gets this too, so I wouldn't have picked based on this.
I've got Linodes in every data center except Atlanta. Dallas has been quite unreliable in the last six months, but the other data centers have been OK. (Fremont had some bad DoS and power issues a while back but it seems to have been OK recently.)
I wonder if Dallas is being affected disproportionately because Linode's own website and management console are hosted there, and that's what the DoSers are targeting.
Somewhat mystified what strategies a victim can use in an attack other than paying bribes or protection money. Obviously the attackers scale faster than defenders so there's no technological solution.
Do a bad job, so you don't make money, so you won't be targeted by people shaking down providers for money?
I suppose one strategy would be to cover up problems and not talk about them, so obviously a provider not reporting problems couldn't be having problems?
yeah I was a linode user for few years and am using digitalocean and I've never looked back. I think the droplet API is by far the most powerful feature, it's so simple to use and understand compared to Amazon. For example, if I need a new development box to test something out I can spin up a new droplet and shut it down when I no longer need it and save the money. It's pretty insane how much of a cost save DO offers.
That's one use case I can definitely see being for DO but doesn't make as much sense for me. I've just got the linode for handling random services (VPN, IMAP, WWW, and a few others) and need it up constantly. For dev work I've always got a local machine that has enough ram and storage to do that instantly right next to me and not have to deal payment for it.
This was my use case also until recently. Unfortunately, the dev box is behind a dynamic ip. I use a following service (dnsdynamic.org), but it's not 100% reliable, so when it became time to make things available to other players I went DO for a $5 box.
I switched from Linode to Uptano[0] almost a year ago now. I heard about them here.
They don't have many of the features Linode, DigitalOcean, etc. have. But they've been super responsive to the (two?) questions I had. Their one ~$85/mo server replaced five Linodes I had. Better performance for less money. Though, as I understand it, Linode has upgraded their offerings since then.
This is the second or third time this has happened this week. They put this status on their page right after it was back up and after I had complaint to them about the random outages and no mention of it on the page. Not claiming they put it up because of the complaint but that it coming after they whole debacle made it the least useful.
I usually stick up for Linode for their excellent CS but this was disappointing. My site was in the middle of a large traffic for an event signup when it went down. SSH failed, accessing Linode's website failed, no mention on twitter or other social media, status page only mentions regular maintenance and not even the last incident. No knowing what was going on, left me feeling helpless. :(
Excerpt from the chat:
>...The two very large attacks we experienced recently were both targeting the Dallas data center however.
When there's a metaphorical fire in the data center, a lot of companies have a tendency to focus first on putting out the fire and second on letting the customers know there's a problem.
So Linode's not alone in that, but hopefully they can get better at it anyway.
I do wish they'd take advantage of their pretty nifty infrastructure and make it so their management system isn't all housed in Dallas. Every time there's an issue in Dallas their management system becomes unusable.
>a lot of companies have a tendency to focus first on putting out the fire and second on letting the customers know there's a problem.
Having been told second would have been atleast comforting. I was disappointed because there was no mention or report of the last DoS attack or downtime, as if it never happened. The last incident lasted for about 17mins but since it was off peak hours, it din't bother me much. Then the same thing replayed today but this time at the peak of the traffic. You would atleast expect a hosting company to have their main page/portal balanced across different regions but nope. Its just sad because except their handling of situations like this, there is nothing really to complaint against them.
your best bet is to move to a smaller MSP with better customer service, one that actually knows your name and gives a shit about you. there might even be an account manager who calls you once every 3 months and asks if things are going well.
but they don't have the big name brand with geek cred, and can't do it for $19.95, so you probably won't. you'll come very close, and then you'll get cold feet because of a million reasons you come up with at the last minute.
instead, you will go to another large hosting provider, because you crave the comfort of knowing there are a million other people just like you, and you will undoubtedly experience the same problems again in 6 months.
interesting world we live in, isn't it? where imaginary problems are worse than real ones. it's truly fascinating.
There ops/sysadmins shouldn't be handling customer relations/support.
It takes about 3 seconds to stand up and yell across the room to the frontline staff, telling them that servers are down and you're working on it.
Then the frontline staff should only have to make a few clicks for announcements to go out (pre-written templates outlining what's going on and what's being done to fix it) via social media, etc.
This is what happens when you pay lip service to customer support but keep them at the bottom of the food chain. You must employ good CS managers and empower them to communicate. You must then ensure that any variance in service good or bad is communicated either by your social networking presence or to your key customers. Sadly most places I've worked are run egocentric idiots who are so insecure they couldn't possibly allow autonomy on the front lines.
Technically it has to be applied at the edge; effectively where the end users are aggregated. It's not really feasible for transport networks to do anything like this.
The second is there's little/no business benefit to applying this to your own network. The benefits are enjoyed by other networks, your competitors.
This is standard operating procedure for many companies in the VPS and server market. It's to ensure that they can keep customers updated even if all of their infrastructure goes down. Some companies take it a step further and put their status page on a completely separate domain, e.g. DreamHost's http://www.dreamhoststatus.com
I used Linode in the past for a number of clients, and had no issues. Then, all of a sudden, in the past 6 months or so my clients have been repeatedly hit with attack after attack after attack. Linode "support" blamed us, as if it were our fault and charged us hundreds of dollars in bandwidth overages.
Needless to say, all of my clients now reside on other platforms and are entirely attack-free now and have been since moving off of Linode.
Care to clarify to which platforms you have moved your clients? I've considered Linode, but haven't "pulled the trigger" just yet, and all this turmoil has me reconsidering. Thanks in advance.
I'm an ex-Linode customer also and hosted with them for years before switching (around the time that they were hacked via the Coldfusion exploit) and I agree, it's sad how far they've dropped from being one of the best in quality/support to what they are now. I can't tell if it's because they're trying to scale up too fast or if theres some other issue however.
Also, Linode is not really the "lowest-cost" option as Digital ocean and similar providers offer better deals for the low price VPS market.
When you say linode 'blamed you' do you mean that your customers were directly attacked (their ips) or that the platform was attacked? Does linode have an anti ddos product that the sell?
I'll take that as a no then. Lots of people in this thread are talking as if this is something all Linode datacentres experience regularly, which it isn't in my experience.
I've used Linode for the last 4 years, and prior to this week, I've only had 2-3 instances of downtime, usually only for a few minutes. This week, it's happened 3 times, which really sucks.
I've been using DigitalOcean for about a year, and I experience downtime with them (NYC1) about once or twice every month. But they're $5.
This sort of thing happens to every VPS provider at some point, and switching to DigitalOcean isn't going to make a huge difference in that respect.
I do wish Linode were a bit better about updating their status page. I don't like relying other peoples angry tweets to determine if it's Linode, or just me.
I've had no appreciable downtime on Digital Ocean, and have a network scattered around: NYC2, AMS1, SFO1, and now Singapore (whatever they call it). My app is low-RAM, low-CPU, but possibly high bandwidth and their stuff is ideal for that.
I'm in NYC1 and NYC2 and it seems like there is maintenance at least once a month, if not more. Luckily, my application is very fault-tolerant, but it is surprising how often they are emailing me about maintenance outages (however brief they may be).
I've got a bunch of monitoring set up on my instances there, and while it does usually result in actual downtime, it's not too long.
Even still, the downtime is frequent enough to where it'd be a bad fit for anything too serious. If our platform at work were to go down for 10 minute periods every month, our inboxes would be full of "What's going on?!?!" emails in short order.
Ohh okay, explains my inability to ssh into one of the linodes and a bunch of customer mails to me about our app being slow. This was around 1500-ish earlier today Eastern time
So, since there's really no difference between $10 for DO and $20 for Linode, what providers are out there that'll have a solid API or provisioning panel, a great network and great service? Doesn't matter if it's 50 bucks a month. Or 100. Anyone? What the heck do people use for business VPS hosting??
Most of what I've seen comparing any of these always brings up price, but that's like bickering over couch change if it matters to your business.
Giving public IP to every node is what I dislike about linode, also No security group like EC2. No VPC.. one has to manually configure firewall on each node.
'Linode managed' costs $100 per linode, insanely high.
Despite of all this I am on linode only because of CPU :)
53 comments
[ 4.7 ms ] story [ 149 ms ] threadEdit: It is a Dallas instance and we had few minutes of downtime. It is working now normally.
I wonder if Dallas is being affected disproportionately because Linode's own website and management console are hosted there, and that's what the DoSers are targeting.
They don't have many of the features Linode, DigitalOcean, etc. have. But they've been super responsive to the (two?) questions I had. Their one ~$85/mo server replaced five Linodes I had. Better performance for less money. Though, as I understand it, Linode has upgraded their offerings since then.
[0] https://uptano.com/
I usually stick up for Linode for their excellent CS but this was disappointing. My site was in the middle of a large traffic for an event signup when it went down. SSH failed, accessing Linode's website failed, no mention on twitter or other social media, status page only mentions regular maintenance and not even the last incident. No knowing what was going on, left me feeling helpless. :(
Excerpt from the chat:
>...The two very large attacks we experienced recently were both targeting the Dallas data center however.
They have been doing this for years and clearly they still haven't learn.
So Linode's not alone in that, but hopefully they can get better at it anyway.
I do wish they'd take advantage of their pretty nifty infrastructure and make it so their management system isn't all housed in Dallas. Every time there's an issue in Dallas their management system becomes unusable.
Having been told second would have been atleast comforting. I was disappointed because there was no mention or report of the last DoS attack or downtime, as if it never happened. The last incident lasted for about 17mins but since it was off peak hours, it din't bother me much. Then the same thing replayed today but this time at the peak of the traffic. You would atleast expect a hosting company to have their main page/portal balanced across different regions but nope. Its just sad because except their handling of situations like this, there is nothing really to complaint against them.
but they don't have the big name brand with geek cred, and can't do it for $19.95, so you probably won't. you'll come very close, and then you'll get cold feet because of a million reasons you come up with at the last minute.
instead, you will go to another large hosting provider, because you crave the comfort of knowing there are a million other people just like you, and you will undoubtedly experience the same problems again in 6 months.
interesting world we live in, isn't it? where imaginary problems are worse than real ones. it's truly fascinating.
There ops/sysadmins shouldn't be handling customer relations/support.
It takes about 3 seconds to stand up and yell across the room to the frontline staff, telling them that servers are down and you're working on it.
Then the frontline staff should only have to make a few clicks for announcements to go out (pre-written templates outlining what's going on and what's being done to fix it) via social media, etc.
http://www.bcp38.info/index.php/Main_Page
And also it is impossible to verify someone did it unless you are inside their network.
it's like saying, yeah it's totally ok to host a bunch of violent gang members in our town, since they only attack people in the next town over.
$ host dreamhoststatus.com
dreamhoststatus.com has address 96.126.109.82
$ nslookup 96.126.109.82
Non-authoritative answer:
82.109.126.96.in-addr.arpa name = li366-82.members.linode.com.
$ nslookup > set type=ns > dreamhoststatus.com Server: 192.168.1.1 Address: 192.168.1.1#53
Non-authoritative answer: dreamhoststatus.com nameserver = ns3.dreamhost.com. dreamhoststatus.com nameserver = ns2.dreamhost.com. dreamhoststatus.com nameserver = ns1.dreamhost.com.
Needless to say, all of my clients now reside on other platforms and are entirely attack-free now and have been since moving off of Linode.
Their service has gone so far downhill.
Linode support has been absolutely crude and even berating in their responses to me. They have lost all sense of professionalism.
It is sad to see a once promising hosting company go so far downhill so fast, but such is the nature of trying to be the lowest-cost option.
Also, Linode is not really the "lowest-cost" option as Digital ocean and similar providers offer better deals for the low price VPS market.
I've been using DigitalOcean for about a year, and I experience downtime with them (NYC1) about once or twice every month. But they're $5.
This sort of thing happens to every VPS provider at some point, and switching to DigitalOcean isn't going to make a huge difference in that respect.
I do wish Linode were a bit better about updating their status page. I don't like relying other peoples angry tweets to determine if it's Linode, or just me.
Even still, the downtime is frequent enough to where it'd be a bad fit for anything too serious. If our platform at work were to go down for 10 minute periods every month, our inboxes would be full of "What's going on?!?!" emails in short order.
Most of what I've seen comparing any of these always brings up price, but that's like bickering over couch change if it matters to your business.