That tweet would suggest to me that someone's credit card was stolen via another method and used to purchase Linode services, rather than the other way around. Ie, I don't think that was related to Linode breach.
Putting aside any opinions on his morals or maturity-level displayed in the chat, it's a fun read. I love how ryan keeps reconnecting after each attempt to kick him out. He must have a pre-verified list of proxies/compromised-boxes he can connect from so he's just burning through them as he gets banned. I found that amusing.
Not sure how useful that reset was. All I had to do was type in my old password and then choose a new one. No email verification, no reset token, nothing. So if the password was indeed compromised, couldn't the attacker do the same?
Yes, he could. But he didn't. If he did that, the page would tell you that your password is wrong, and you'd contact support. The attacker probably didn't change any password, because it would be futile.
Also, the attaker could also change the account email. Neither tokens nor email confirmation would help.
The biggest problem is for people that used the same password on Linode and any other important service. (And, of course, all the CC stuff...)
Not only that, it also makes me wonder about the free RAM upgrade from almost a week ago. Some people are reporting their Linode credit cards being used for fraudulent purchases as far as a week ago, so this might have been a move to gain some pre-emptive goodwill.
I don't know though... will wait until more details are available but will be keeping an eye on CC statements / VPS alternatives.
I tend to think the timing is just a coincidence. They also upgraded CPU and bandwidth over the last month and there were rumors of upgraded RAM for a while too.
Logically, they couldn't have planned it all in such a short period of time. However if they did, from a business perspective it is very good plan. Without the upgrade, today some customers would have had two reasons to leave Linode, now they only have one.
I believe that to be a coincidence. They have been performing upgrades for a while now. (Increased hard drive space, increased CPU, and finally increased ram.)
I think it was just an unfortunately timed third phase of their upgrade plan.
From a purported abridged chatlog with the alleged hacker:
> 05:42 < ryan||> credit cards were encrypted, sadly both the private and public keys were stored on the webserver so that provides 0 additional security
> 06:00 < ryann> They did try to encrypt them, but using public key encryption doesn't work if you have the public and private key in the same directory
Can't think of the exact word to describe that practice (what you did in response to the long list of questions which I've seen) but on the part of the company requesting you to answer the questions it's more or less a "absence of malice" type of thing that allows them to appear that they are doing the right thing while fully knowing that people are doing what you are doing. It's a "we will look the other way until we need to show that it's not our fault because we have passed the liability to you - look you acknowledge doing all the right things".
It does not mean anything until they decide you are not compliant and you need to prove you are compliant. (I've never had to but I'd appreciate insight from people who have)
Actually, if you're processing cards directly, you do in fact need to have an PCI-qualified outside firm† (a QSA) audit you for PCI compliance. But those audits are notoriously superficial; PCI audits are a race-to-the-bottom affair.
The quality of PCI security audits is a continual aggravation to everyone I routinely talk to in my industry. I've told more than one client: if you need a QSA audit, get the cheapest one you can. If you need a software security assessment, don't use a QSA firm.
Will add that just having gone through an ICANN registrar audit (which by the way were specified and supposed to be done literally 10 or 12 years ago but never requested by ICANN) with a third party company hired (accounting firm) it's total compliance theater.
Add: "hired by ICANN after a bidding process". Same happened with data escrow which was just implemented a few years ago and is operated by Iron Mountain.
Companies are split into PCI Levels based on how much money/customers they handle. Level 1 are big companies like amazon, level 2 are medium sized online retailers generally, and level 3 are smaller retailers.
The 'lower' your level, the easier the PCI audits are. If you are level 1 you have mandatory external audits. If you are level 2 you have a 'self assessment' which is basically a checklist which says "Yes, I promise I'm in compliance".
If you have a confirmed breach, you are upgraded to Level 1 merchant audit requirements. This is generally quite costly as the external audit is extensive and must be paid for.
The audits are toothless and ultimately the audit only happens once (if ever) and people keeping pub/secret key in the same place unprotected... well.... They're just unlikely to get security at all...
Am I the only one who is more confused about why there are compiled java classes and AMI BIOS updates in the www directory than about the hacking itself?
> Am I the only one who is more confused about why there are compiled java classes and AMI BIOS updates in the www directory than about the hacking itself?
Sysadmins being lazy.
Somebody needs to get a file from a workstation to a remote machine. There's a firewall in the way somewhere that prevents SSH directly between them or one of them is a Windows box that isn't running an SSH server.
The "correct" solution is complicated and takes 5 minutes to setup. So the sysadmin just copies the file to the web server and downloads it with a browser on the other end. Because port 80 is always open.
Something that took me a long time to notice was that you can actually copy files directly over a RDP session from/to the local machine or other RDP sessions using the clipboard.. Even nested RDPs. Has been an absolute timesaver to know about when doing Windows admin work.
I think it depends on how you mount your clipboard/drives on rdp connect. To get it right with windows you just check the share clipboard/share drive checkboxes and off to the races. With rdesktop you have to throw the -r flag and mount a clip board and then the -r flag and mount a drive. Not sure about other Linux clients but I'm sure there's a similar option in all of them.
Its also why we invoice and take wire payments rather than storing CC details. There's just so much to go wrong.
Also PKI is shit for this sort of thing. As demonstrated, the moment that public key is gone, then the whole system falls like a house of cards. For the non believers of this fact, why else would there be a certificate revocation list and root CA updates for windows periodically...
Storing credit card info just helps make you a bigger target. If your a small company, better let someone else store card info, let them be the target.
Also you're fined by the credit card companies if you lose card information. I believe it's a per card fine, so it get expensive really quickly.
Actually I don't get why any company would choose to store credit card information, when most payment providers will do it for you.
True, but I'd rather fix the problem that got them in, force reset of passwords, and delete all customer keys and require them to create new ones than be like "uhhhh, our data was hacked and your credit card is safely encrypted... But we had the encryption key on the server too, oops"
Following the traditional responsibility/accountability dichotomy: They are responsible for storing the cc number securely but you are accountable when something goes wrong (because you gave them that task)
Much like Linode are responsible for hosting my clients site but I sigh am accountable when something goes wrong.
Even better to use someone that isn't your payment processor, so that should you need to change payment processors you don't also have to re-acquire all the billing info from your customers. You can use Stripe today, PayPal tomorrow, and Braintree the next if that's what works best for your business.
There's always going to be single points of failure, but which is more likely: you want or have to change payment processors (you've been terminated, your fees have gone up, you want to switch to a lower cost provider) or you want to change flat-rate vaulting services? Plus, Spreedly will give you your data if you leave, whereas there is no way to get stored billing info out of most payment processors.
In what ways are wire payments better than using credit cards? In wire payments aren't you using the actual bank account numbers along with routing numbers which is also very sensitive information ?
Also I do not think, but I am not sure, that fraudulent wire payments/transfers are reversible.
Wire transfers often are not able to be undone once they happen (and are accepted by the other bank). This is the reason why there's so much verification that happens in wire transfers. (I helped develop 2nd factor authentication used for authenticating wire transfers for a financial company)
Here is what Linode replied to me when I asked them about that chat log in a support ticket:
Hello,
Thank you for reaching out. We appreciate and understand your concerns. At this time the evidence suggest that this activity was targeting a specific customer. We are unable to release any additional details regarding this incident at this time, as there is an ongoing investigation.
We have no comment regarding ryan*'s comments in #linode. You are of course free to take any steps you deem prudent or necessary to ensure the integrity of your online presence.
I am sorry that we cannot provide more information at this time. As always feel free to contact us at any time with any future concerns.
Regards,
Quintin
These guys are looking totally incompetent at this point.
If you believe this Ryan guy, credit cards stored on the same server as the key to decrypt them, Lish passwords stored in plain text, they've known for some time and lied about what actually happened and now they're saying "we won't do anything about it" via email?
"You are of course free to take any steps you deem prudent or necessary to ensure the integrity of your online presence."
Unbelievable.
Edit: not to mention they "made a deal" with the hacker not to tell anyone? What the hell?
for the record, i am the person who started the WHT thread.
there is a mixture of truth and lies on both sides, to be honest.
i am annoyed with it, because i reached out to several linode employees privately to given them an opportunity to explain what was going on -- they either said 'no comment' or said my linode was fine.
based on the irc log, that is clearly not the case. which is why i decided to raise my concerns publically.
luckily for me, my linode was not doing anything mission-critical, just some secondary monitoring and running an ircd for a network i like using, but there are others who are using linode for mission-critical work, and they deserve more transparency than this.
To be fair the hacker didn't say the keys were stored on the same server as the credit card numbers, he said they were stored on the web server. It's most likely the database containing the CC numbers resides on a separate set of boxes than the web servers.
That's a rather key assumption. If you don't believe him, then all you have is a trolling (or at least self-aggrandizing) hacker whose credentials consist solely of logging into an IRC channel, refusing to identify who he was working with, and offering no tangible proof of having compromised any CC info.
On the other hand, it's conceivable that if ryan managed to get into the files a customer was hosting on Linode, and that customer was improperly storing CC info, then their customers' info would have been vulnerable, and ryan's claims would be sort of half-true. Even so, that wouldn't directly affect other Linode customers or put liability in Linode's lap.
That is not the way to handle this issue. I've found my one problem with Linode is they are arrogant. It comes off pretty strong if you ever ask them questions in chat.
One thing to note is that the irc channel if that is what you mean by chat, is mostly populated by non linode staff. And many of us there tend to be sarcastic as we idle there and chat about all sorts of stuff while we are bored at work or whatever. Unless it's someone with ops, you aren't getting an official reply and even then for something official they usually refer to ticket system.
That is an absurd response. I don't care if they believe a specific customer was targeted, I want to know what happened and what information may have been compromised.
Despite what the other replies here are saying, this seems like a perfectly acceptable response to me. This comes off to me not as they're refusing to talk about it, but they _can't_ talk about it, presumably because of an ongoing investigation. I'm not sure what else people here are expecting them to say.
They probably can't say anything just yet. It's fair to be mad about them not realizing (or worse, covering up) the breadth of this breach, but do keep in mind that if they're working with the FBI, they've probably been asked to keep a lid on their official response for a few hours.
Found it interesting that Linode uses Coldfusion. Wonder if Adobe has anything to say about the apparent 0-day.
If the hacker's claims are true (Would appear so, the directory listing checks out) then Linode really need to address this ASAP. Passwords are one thing but to have CC details leaked is even worse. I'm not familiar with CC processing but it seems like bad practice to store the encryption keys on the web server.
It wouldn't take a zero-day flaw in the Coldfusion stack for a CF application to have an undocumented vulnerability; in fact, it's much more likely that the vulnerability is in the application code than in the stack itself.
Depending on who you're talking to, an app-level vulnerability in a Linode management console might be called a "0-day". But it's true that a CF stack flaw is not impossible.
The problem I have balancing the likelihood of CF stack bugs vs. CF app bugs is that I've had to assess a bunch of CF apps, and they're uniformly coded to mid-1990s best practices. No matter how many bugs have been announced in the CF stack, as a betting man my money would always be on CF app bugs.
A patch has recently been issued (09 APR 2013) by Adobe for the various versions of ColdFusion:
"This hotfix resolves a vulnerability that could be exploited to impersonate an authenticated user (CVE-2013-1387).
"This hotfix resolves a vulnerability that could be exploited by an unauthorized user to gain access to the ColdFusion administrator console (CVE-2013-1388)."
I wonder if Linode has a requirement to have their CF admin site accessible outside their network (assuming, of course, that the attacker didn't first gain entry into the corporate network, and then attacked the CF installation)?
As someone who still maintains a very old CF application, I am sure to lock down access to the admin site via IP restrictions.
This is not a brand new 0-day. This is a bug that Adobe communicated and patched months ago, a bug that affected a lot of folks who didn't follow the standard practice of locking down an administrative directory on the website.
The basic overview is this: CF servers have an administrative portal at /cfide/. A bug in the scheduler code (think cron) allowed remote attackers to upload arbitrary code to the server and then execute it. Savvy attackers could upload their own backdoors directly into the administrative folder on the site and then execute that code to gain additional access.
As a Linode customer (admittedly only for a small VM I play around with) I have to say I've been impressed with their service and their prices of course, and I'm waiting for further confirmation about the depth of this hack. I was unaware Linode was using ColdFusion. It should be pointed out that CF is a very mature language, akin to ASP.NET. It is actively maintained by Adobe and used by a huge number of websites globally.
My Visa card that I used with Linode was stolen and used on an Amazon order I didn't authorise last week, my bank successfully blocked the charge. Someone else reported their Visa had also been compromised in the thread 2 days ago, looks like that confirms the suspicions: https://news.ycombinator.com/item?id=5542015
Poor show Linode. (edit: worth noting I use the card with other things too, I have no confirmation it was leaked through Linode other than the compromise happening at the same time these supposed leaks happened).
Every site can be hacked. It's just a matter of time. You just have to properly react: call your CC provider, check for any charges on your bill, and move on.
I do not think anyone here is actually worried about their funds; as mentioned below, any reputable provider will have such charges promptly reversed. The problem is instead with their response to the situation.
Linode has addressed the breach, but assured customers nothing of value had been compromised. This infers two thoughts. One: they knew of the breach and lied, thereby unveiling a unforthcoming and dishonest nature. Or two: they did not properly investigative the severity of the issue, thereby suggesting incompetence. Both equally reprehensible.
There is no evidence to suggest that Linode leaked credit card data.
Personally I took precautionary measures and just called my bank to replace my credit card, which I think is the sane approach, as when it comes to hacking you have to assume the worst.
However, your statement on "has lots of small issues but fixes them the same day" is just stupidly childish. Linode's issues are bigger just because they are a bigger target.
Wait a second.. you consider a provider giving data from your VMs to another random customer a SMALL issue?
Maybe you don't have anything of importance on your VMs, but plenty of people do. That data could contain credit card data, passwords, etc, etc, etc. It is very much a large issue.
Sure. My thoughts don't apply to everyone here, and I certainly can't claim to be unbiased since I like DO so much.
According to DigitalOcean, they stated that this impacts 3% of all machines, only the largest and most expensive servers. None of the smaller plans were leaking data.
I don't know how many credit card numbers were leaked from linode, but I'd guess more than 3%.
Second, if security is important to you, you can use 'dd' to clear the machine yourself before shutting it off. (In fact, good data destruction policies mandate the use of 'shred' et al anyway). On Linode, affected users don't even have a workaround (like this) to avoid information compromise.
Great, now I am feeling paranoid although I don't see any unauthorized charges on my card. Does anyone know if debit cards are legally protected the same way as credit cards with 0% liability.
I know someone who got their debit card cloned. While the bank eventually repaid him, that did nothing to repay him the additional fees he owned his normal debtors (e.g. rent, utilities, etc).
With a credit card you aren't losing "actual" money. You are losing the bank's borrowed money which the bank pays back. With a debit card you're losing cash which you won't be able to replace yourself and which the bank might take days to weeks to replace.
Even if you NEED to borrow while your credit card is out of commission you can either use the overdraft facility on your debit card or other quick sources of credit. Hard to get quick cash without going to a pawn shop.
they typically have the same protections, the only difference is that while credit cards only check that the charge can be made during the authorization period, debit cards lock up the bank's funds immediately which may cause your checking account to be temporarily unusable even if all the charges are successfully disputed.
a good practice that bankers constantly tell me is to have a separate credit card for online purchases for the fact alone that it is one step removed from your checking account.
These are the FTC's rules [1], I'm not sure if Visa or Mastercard can make them 'better' (give you a larger window). They have an interesting tidbit below their chart -
>If someone makes unauthorized transactions with your debit card number, but your card is not lost, you are not liable for those transactions if you report them within 60 days of your statement being sent to you.
Isn't it free to get a new card? That'd be the easier way than worrying.
Generally these days they are, but the problem is the money is removed is from your account by the time the charge appears, (at least a period of time), whereas on a credit card you have 30 days to review charges. This could cause overdrafts, etc. depending on timing and amount. Those too can generally be reversed, but the whole thing becomes more of a headache. I never use debit cards for any kind of recurring charges, and I avoid using them online in general.
The same is possible with a credit card - you can have your card maxed, get nailed with overage charges, declined transactions, etc. It's still a headache, but when I've had it happen to me I think I had the money restored within a few minutes of making the call.
You should talk to your bank as it depends on the network your bank uses. Most of the time it's something like a 48 hour window to challenge charges but it's far less at some banks.
> [With a debit card,] "Until the bank provides provisional credit, you could temporarily be out of pocket for the amount in dispute," said Richard Foley, an FDIC attorney who specializes in consumer issues. "This would not typically happen with a credit card because consumers can withhold payment of the amount in dispute."
> Also, as discussed on the next page, consumers have better federal protections when they purchase faulty goods with credit cards.
They are. Protections are exactly the same (in the US at least.) You have protection from the moment that you learn of the problem, not when it happens.
Not that having your account drained doesn't suck, but your worst case scenario there isn't terrible unless you fail to check stuff and be responsible.
I would cancel that card right now. IMHO You should never ever ever ever use a debit card anywhere else other than the ATM. Credit cards give you way more financial protection.
I have two accounts with them, one for my day job and one through my LLC. Right now neither card is showing unusual charges.
My day job had some Google Apps account compromises last month, and this is making me paranoid that the database that contained hashed/salted passwords for our Intranet hosted on Linode was the culprit. The time frame doesn't seem to line up, but we didn't see any evidence of phishing or compromised desktops being involved.
(Don't want to spread fear - I haven't been able to find any evidence our Linodes were compromised either.)
I'm normally huge a Linode evangelist, but I'm severely disappointed with the lack of transparency on this. I'm debating right now whether to rebuild all our nodes from scratch as I'm not sure they can be trusted.
I had a CC problem, and I use it for a very restricted number of online services. Luckily the fraudulent attempt was blocked by the CC company. Linode was not at the top of my list of suspects (there was another company that seemed to be storing passwords in cleartext), but now I'm wondering. This was a couple of months ago though... I wonder if that's the right time frame?
If this is true then all the trust that Linode has built up over the years was just thrown out the window. According to the hacker they've known for 2 weeks and made a deal with the hackers. Ultimately, they were as far from transparent as it gets and on top of that they did a horrible job with their security.
Hopefully, they own up and start being transparent.
If this is true then what alternative hosts should I look at, besides AWS?
"credit cards were encrypted, sadly both the private and public keys were stored on the webserver so that provides 0 additional security"
That's just poor security and 100% they're own fault. I accept that there are security issues with every platform, but basic security measures and being transparent is still expected. My biggest issue with them in all of this is not being transparent.
Looks like someone who likes attention on some random IRC channel who is apparently a hacker may have hacked our system and we don't know who/when/where/why/how or what they may have got. Nor are we sure we were even hacked???
It takes time for people to investigate stuff. It's not just a couple hours. Also some random guys words on IRC (who could very well own INSERT RANDOM HOSTING COMPANY for all we know and looking to scare people off Linode) should be taken with a grain of salt.
If the information he offered is accurate (e.g. the public and private keys were stored together on the webserver), that wouldn't take a long time to confirm.
They are supposed to say that they would never ever store the CC numbers this way. Otherwise their customers (like me) have really no better option than to block their cards, which is quite an inconvenience. This is exactly the trouble I was hoping to avoid by not using a cheap VPS hosting.
"Logical" is not a fancy synonym for "severe" or "unforgiving", and is probably not the word you wanted here. There are sometimes good, logical arguments that you can make for cutting people some slack.
Two alternatives often mentioned on here are DigitalOcean and RamNode.
I've only used DigitalOcean. My anecdotal experience from running a Chef Server on a 1GB instance has been pretty mixed. The price is good, but network and CPU performance feels very variable to me. A month ago their Amsterdam servers were unable to be resized, and there was nothing about it on their status page. I tweeted and was told they'd be working "some time later today". Doesn't fill me with much confidence in general.
I'd still choose Linode for anything of importance - their long reputation is well earned in my opinion. But, if this breach is true, I hope they handle it well.
I've been using digitalocean for a hobby project, and am planning on launching a more serious project with them. For the past two months I've used them (so, not much experience, but some) I haven't had any issues, and they are very reasonably priced.
The chatlog does provide some evidence that it is indeed the hacker, but does little to convince me that he got CC info and Linode is not telling us the whole truth. The evidence he provides is just simple source code snips and the directory listing, which would be expected based on what Linode has told us.
This could very well be the hackers own submission to /. trying to get more attention for his hack by claiming he has CC numbers which I doubt he has.
That's a fair point. I'm speaking more as a judge/jury view on the situation though. I don't think Linode users should panic and run for the hills just yet based off this alone.
So you're unfortunate enough to be a customer who had their CC leaked. So you spend 5 minutes changing your password (you use unique, non-formulaic passwords, right?) and 15 minutes on the phone to CC company to ask for a new card. Then you use your backup card for 2 weeks (you have a backup card, right?)
A month later, spend 30 minutes on the phone with CC company only if strange transactions appeared.
Not the end of the world. The CC industry is set up well to handle this kind of thing.
You missed the step where you have to find all charges going to your old CC and then deal with moving every one of those accounts to your new one when it gets there. Hopefully you don't incur any late fees while you're going through the process!
Kind of sucks to have to spend hours doing that for someone else's oversight. It's not the end of the world, but it paints a clear picture about where a company's priorities are.
Maybe I'm weird, but I know exactly which binding credit agreements I'm in and how they're paid, and definitely none of them get paid using another binding credit agreement. :)
I mean, I know all that I'm in, but I can definitely see myself missing one or two if asked on the spot to recall all of them. Phone, cable, AWS, gym, power, garbage/etc, insurance,... I'm sure there are one or two others. Even if there aren't, I'd feel compelled to go through everything to make sure there aren't more.
Even just dealing with all of those is going to be a pain in the ass though. I'd probably end up spending hours all together just on hold with some of those people.
You don't pay for Netflix, Adwords, Amazon Prime, AWS, etc. using a card? If so, yes, I think you're weird. What do you do? Give them all your bank details?
I'm confused. The person you were replying to said:
> you have to find all charges going to your old CC and then deal with moving every one of those accounts to your new one when it gets there. Hopefully you don't incur any late fees while you're going through the process!
I've never been charged a late fee by a firm I didn't have a credit agreement with. Perhaps other parts of the world are more insane, but here that is definitely not commonplace.
I don't know who does and who doesn't, honestly. I try to avoid being delinquent. I know from the last time I had to change my card that my phone provider and ISP certainly do. Anybody who charges you on a recurring basis certainly can — they just add the amount to your next bill. They won't take you to court for it, but it will be added to the amount that you must pay or be disconnected.
Sorry, I should have said "fees". I've recently had to deal with this process, and I've found a few vendors who charge a penalty if the attempt to charge my card doesn't go through for any reason.
To dismiss this breach seems odd to me. The tech community in general has placed a lot of trust and faith in Linode over the years. The shareowners at Linode have surely been great beneficiaries to that. Part of that "unspoken agreement", if you will, is that Linode be competent at what they do and that means keeping your data and information secure.
If even an iota of what I read in the abridged IRC log is true, Linode doesn't seem to care much about security or protecting Linode customer data. I mean, storing "encrypted" card numbers alongside private/public keys? Really.
Sigh, really? Ok, you typed your credit card number into a web browser at some point. If your sole reason for doing so was "I absolutely trust the people on the other end of this socket not to do what 99% of all people handling credit card data do whether they pretend otherwise or not", instead of something like "hmm that reminds me, I haven't scanned last month's statement yet", then the problem lies squarely with you, the uninformed consumer.
I will happily dismiss this breach, not because they didn't make some amateur crypto mistake, or because they weren't using freaking ColdFusion, or because they were storing data in some nice compartmentalized form, I reject because this happens every single day and has done for decades, and there is an entire sub-industry built around its after-effects. If you don't understand this you shouldn't own a credit card.
If you type a credit card number in online not expecting to recuperate any damage caused from your card company, call them up now for clarification or cancel the damn card. That's equivalent to stuffing cash in an envelope and posting it to Nigeria because some prince promises he'll keep it in a safe for you. It's 90% the reason you should be using credit cards in the first place. Think.
Linode should not be rubbished here. They've got one of the largest VPS installs around, so they most likely know their shit. They make an ultra-common CC mistake that has happened daily for almost 20 years now, by companies large and small, got pwned due to a bug in someone else's software, and you think I'm going to play along with the righteous indignation bullshit here? GTFO.
Let he without sin cast the first stone. Despite 20+ years' experience I still cannot cast that first stone. I make bullshit mistakes like this every day, and despite your grandiose delusions you probably do too.
As for whiners complaining about their data suddenly being insecure, well, data security 101: you're making the same bullshit mistake Linode are making, and despite that you're complaining about it. If you care about data security in the "cloud", hosting it on a freaking VPS is not the way to do things.
So because companies A through X are irresponsible with data, customers should regard that as acceptable and give company Y a free pass to do the same? I don't understand how a reasonable analysis of the situation can come to that conclusion.
"Because Nigerian princes A through X are irresponsible with your cash, budding lottery winners should regard that as acceptable and give Nigerian prince Y a free pass to do the same?"
Of course not, and this drives to the very core of risk management. I've signed up for some very shady online services in my time, doing so in the full knowledge that should a product or service not be rendered as advertised, I am guaranteed to be able to reverse the relevant charge. Even when I the consumer am doing something shady (in a case last month, attempting to import goods I knew weren't certified for the EU), the system still works for me. This is the sole reason I use a credit card rather than, say, my current account's Visa number.
It's not even about assessing the risk of whether or not you're going to get ripped off, but whether or not a particular company will cause you the inconvenience of the aforementioned phone calls.
If you work on the assumption that you card data is safe, you quite simply aren't safe enough to be in possession of a computer or card. Credit cards aren't built on that assumption, instead their entire motivation is based on risk profiling both the consumer and merchant, and terminating agreements when various thresholds are reached. In return the industry guarantees that in the minority of cases where things go wrong for the consumer, the problem can be corrected swiftly.
It's understandably upsetting that their customer database might have leaked, and I can genuinely understand peoples' concern over that. But as 4chan has taught us, there are very few people left in the west whose address and telephone number aren't available within even an hour's Googling.
As for locating confidential data on machines shared with other customers and managed by a piece of unaudited software, I have no sympathy for that. That's the price of a VPS, and why it's so heavily discounted compared to real hardware.
You don't know the names of companies A through X, or supposedly safe Z for that matter. All you'll be doing is an enormous amount of work and bother to move from Y to, lets say, A, because you think you'll be more secure but unfortunately if anything its probably the other way around, its just that A hasn't been hacked... yet... so far as they know...
none of them get a free pass they all suck, but the one that just got busted is probably going to be a little more security focused in the near future.
Hmm stay at a place that just got burned, or expend lots of effort to move to a place that hasn't been burned yet...
> If your sole reason for doing so was "I absolutely trust the people on the other end of this socket not to do what 99% of all people handling credit card data do whether they pretend otherwise or not", instead of something like "hmm that reminds me, I haven't scanned last month's statement yet", then the problem lies squarely with you, the uninformed consumer.
If your expectation when taking credit card numbers was "I'm confident in my abilities to keep this information safe, and if I get hacked I expect my customers not to move to another service and never, ever touch mine with a ten foot pole", then the problem lies squarely with you, the uninformed business.
Its worth noting that charges can be credited to cancelled cards under certain circumstances. Happened to me and the bank said it happens regularly. True, its usually a painless process; but its not that simple.
I think that Linode did a big mistake here. Let's wait for a formal communication.
But this is the moment to support them.
Yes, maybe sounds crazy.
When you host on any third party datacenter, you take risks that something like this could happen. So, deal with it. Check your credit card, if your receive something wrong, call to your card and that's all. But we need to support also the good work, and this guys do great work in the hosting business. Just my opinion.
Well said. It's a fact of life that companies get hacked. So it's no surprise that it eventually happened to Linode. If you flee somewhere else, all you're doing is hoping that the other company you run to won't get hacked rather than using any logical thought.
I can think of two good reasons why you should flee Linode. It remains to be seen if either are actually true, and until indications say yes, then panic is unwarranted:
1. If it becomes apparent that Linode is far more vulnerable to hacking than other hosting providers. But one hack alone does not prove this.
2. If Linode grossly mishandles the situation. There have been a couple of allegations to that effect so far, but nothing substantial. I don't see any reason to claim that they've done this yet.
Linode has already grossly mishandled the situation by not coming out with a complete statement about what exactly happened. I only read this news because it was posed here -- no email notification, no update on their homepage, no twitter, no nothing.
The alleged hacker has made serious and specific claims, and Linode has done jack shit; without more information, how should I proceed? I don't want to call my bank and waste time getting a new credit card (not to mention replacing a million and two services) without a confirmation and I can't get a confirmation because Linodes people are having a circle jerk (or whatever the hell they do).
> 2. If Linode grossly mishandles the situation. There have been a couple of allegations to that effect so far, but nothing substantial. I don't see any reason to claim that they've done this yet.
Linode's handling of the Bitcoin incident last year was sub-optimal. This too has been sub-optimal, given that credit cards were exposed but all we heard on Friday was to change our passwords, and even that was claimed to just be a super-careful precaution.
Linode needs to start giving us some frank talk ASAP. They've already burned through a very generous helping of benefit-of-the-doubt.
I'm curious how he's going to backdoor the binaries running on my computer built by the launchpad servers without anyone noticing. Granted, I'm not checking the commit logs every time apt gives me a new version, but all the same.
Linode really needs to make a statement about what happened with this hack, stating if credit card information was taken. A lack of communication does not help me trust them. I'd rather have them speak up as to what happened and know if I need to have my CC reissued.
It seems to me that the fundamental problem is not that they got hacked (although it seems that storing a decryption key in the same directory as the encrypted data is over the top careless), but their response to the disclosure that they got hacked. I realize that there may be limitations on exactly what they can say, but they should be as open as possible on what may have happened, what they are doing to protect their customers, and what their customers should do to protect themselves. Customers taking action when there wasn't a breach is less of a problem than not taking action when there was, in fact, a breach.
They made a mistake. They'll learn from it, hopefully (among other things) by ensuring no customer credit card details ever hit their servers ever again.
Seems like it usually takes an event like this for a company to really crack down on their security practices. I would wager that Linode will be one of the most secure providers in the coming months. Whether or not people will trust them is another story.
[Warning: imperfect analogy follows.] It's one thing if Linode is like someone who gets drunk and crashes their vehicle. That's 100% their fault and they've burned any goodwill. In this case, however, Linode is like someone who was carjacked. Perhaps Linode shouldn't have been driving that type of vehicle in an area known to have people attempting to carjack every single vehicle that drives by. Perhaps they should have installed thicker bullet-proof glass. Or even have taken measures not to trust any locks that the manufacturer insists are secure but have zero-day exploits. Regardless, Linode is still the victim of unscrupulous criminals. Maybe they could and should have done more but the bigger question is now that they've been carjacked, what are they doing to ensure that the carjackers haven't installed anything malicious that still remains in the vehicle?
If the allegations are true, then Linode was keeping encrypted CC numbers, with the decryption key in nearly the same place.
Trying to make the analogy more sufficient by incorporating this type of fact would only make the carjacking analogy more absurd. At the end of the day, an analogy is not needed.
I think a more accurate analogy is to say that Linode is moving your important files from one location to another in their moving van. They park at a 7-11 to run inside and grab a snack, leaving the van unlocked. An intruder comes along, opens the unlocked doors, makes a few copies of your files and leaves. Linode gets back in the van, notices the intrusion, does nothing except tell you that "you have nothing to worry about, but you may as well change your locks" and then when the truth comes to light, they basically stop returning your phone calls.
To those of you who have claimed that your CCs have been abused -- I checked mine (which I used to pay for Linode) and it hasn't been used to do anything funny.
(not knowing anything) - wouldn't it be possible to give the cc company a white list of clients that can continue to use it and block all other requests?
Really off-topic, but still sad: This is a link to slashdot, but it's on HN's frontpage before it's on slashdot's front-page (if it'll ever get there). (And IMHO that's sad, because /. used to be top notch).
I've noticed before that stuff from the HN frontpage appears on /. one to three days after, but I've never seen it for links to slashdot :-)
I rather wish this was a link instead to the original thread rather than to Slashdot. It isn't a big deal, but it certainly would have saved an extra click.
Also off-topic: I've noticed that as well with Slashdot, which is why I lurk HN pretty regularly now. Plus, some of the front page material on /. does more to insight angry discussion, and the community has become increasingly more vitriolic.
At least here, even if someone's brash, they're fairly honest about it (in general). I've even seen a number of disagreements that have been respectful and cordial. It's sad to say, but that's a rare thing these days.
I'm looking forward to seeing an official response from Linode on this. Hopefully they are fast and honest about it. I've been a happy customer for quite a while, but this is definitely a concern.
They weren't exactly fast and honest the last time a break-in happened. In fact even to this day nobody is quite sure what went down aside from tons of Bitcoins going missing!
They are already way overdue. Their support staff is pretty damn fast, so I'm sure they could have commented on this if they wanted to.
Alas, they have not.
I just blocked my credit card, and Linode will not get anymore of my money. Too bad, i enjoyed my stay very much, but no matter what their response will be, I doubt I will be able to believe them. =/
Off topic but still relevant, but doesn't it seem a bit primitive that companies have to store you CC# for recurring payments? The one number that uniquely identifies your account and everyone you want to re-use it has to keep a copy. Couldn't the credit card company issue some unique ID to each vendor for recurrent payments? Ex. the vendor issues your CC# to the CC Company for charge and recurring process. The CC Co. responds by replying back with authorization and an ID unique to that vendor that says "Use this number for charging this customer again, but it will only work coming from you, so if you lose it, it can't be used elsewhere". The vendor then discards your real CC number.
But isn't that just between Stripe and the company requesting payment?
e.g: Acme, Inc. sends Stripe your CC#, Stripe sends them some unique token, and they store that; correct?
So Stripe still has your CC#, and is at risk.
So this is really just risk mitigation; what I think TP is suggesting we need is unique authorizations at the banking level.
Something on the order of virtual credit cards, or temporary tokens, which are ultimately verified by your bank [or in other words: the lender(s) making anti-fraud guarantees, etc.]
(e.g: this token is authorized for 24 hours up to this limit; this token is authorized indefinitely up to $xx/mo.; this token is authorized for 1 year; etc.)
No. Customer sends Stripe their CC number, via AJAX in the browser. Acme, Inc. never has it even transiently. Stripe return a token to the browser, which is sent in a POST to Acme, Inc., then they verify it server side with a private API key.
Edit: yes Stripe has your number, but since their sole business is about securing that information, they probably do a better job of it than your typical online merchant.
Depends on what part of the chain you are. Most gateways/processors offer some sort of token that the end user uses.
You then put your trust in the gateway/processor to store the credit card. Which I assume is most likely behind the best possible security stuff money can afford. Since that's their entire business. One screw up and their gone.
It's amazing that it doesn't work that way. You could argue that it's because of the amount of infrastructure already in place, but why couldn't a new system be gradually and optionally rolled out?
Stripe (and probably others) has functionality like this where the seller's server never sees the CC number, and developers can store a unique token to re-charge the customer at a later date.
This is basically what Stripe does. The "CC Company" basically doesn't offer more than a simple yes/no API -- "approved w/ #" or "declined". It'd be great if they could do more, but that's where the opportunity for folks like Stripe lies.
I guess it just surprises me that the CC companies don't do it themselves. Is the additional overhead of such a feature so high such as to warrant the choice between no payment processor (x% fee per transaction) vs. third party middle guy (x% + a bit more)? Not to mention the money the CC companies would save by not having so many fraudulent transactions (assuming they come out of their pocket, I honestly don't know).
Well, you can get AVS checks back too. But in general the entire system is so far behind the times. It really needs to be overhauled with security as the focal point.
It seems there are a LOT of incompetents then. Are the payment processors you're talking about something the vendor interfaces with in the background and not a third party that is directly utilized by the customer (eg. Stripe, Paypal)?
What if you want to change processors? If you weren't storing the CC details, wouldn't you have to have customers enter all their details again? I imagine this could cause a drop in revenues due to people either forgetting, procrastinating, or just not bothering.
It's never cool to be actually- or quasi-locked into a vendor.
Some will let you export the data, but it's a hassle. They burn it to a DVD and ship it to you and you have to sign tons of stuff freeing them of any liability.
Otherwise you just ask customers to re-authenticate. Often you have a few months headway for a switch like that.
Most decent processors have processes in place for the transfer of CC numbers. I was involved with this process at a decent-sized magazine, it involved armed security, an encrypted hard drive in a locked container and millions of dollars of insurance. It's not an easy process, but is possible.
The majority of CC names have "Virtual CC numbers" which is precisely this - You generate a new number, which links to your account, but can only be used for the merchant you specify.
From: "Linode" <support@linode.com>
Date: Sat, 13 Apr 2013 00:11:09 -0000
Precedence: bulk
Return-Path: 6723614.1706014@e2ma.net
Message-ID: <knuab.c9dae.xxx@e2ma.net>
List-Unsubscribe: <http://e2.ma/optout/c9dae/xxx>
X-Test-Mailing: no
Dear Linode customer,
Linode administrators have discovered and blocked suspicious activity on th=
e Linode network.=C2=A0 This activity appears to have been a coordinated at=
tempt to access the account of one of our customers.=C2=A0 This customer is=
aware of this activity and we have determined its extent and impact.=C2=A0=
We have found no evidence that any Linode data of any other customer was a=
ccessed.=C2=A0 In addition, we have found no evidence that payment informat=
ion of any customer was accessed.
We have been advised that law enforcement officials are aware of the intrus=
ion into this customer=E2=80=99s systems. We have implemented all appropria=
te measures to provide the maximum amount of protection to our customers. O=
ut of an abundance of caution, however, we have decided to implement a Lino=
de Manager password reset. In so doing, we have immediately expired all cur=
rent passwords. You will be prompted to create a new password the next time=
that you log into the Linode Manager. We also recommend changing your LISH=
passwords and, if applicable, regenerating your API key.
The following represent best practices in creating new passwords:
-- Avoid using simple passwords based on dictionary words
-- Never use the same password on multiple sites or services
-- Never click on 'reset password' requests in unsolicited emails - instead=
go directly to the service
We apologize for the inconvenience. If you have any questions, please do no=
t hesitate to contact our support team at support@linode.com.
Ah this is so shit. I want to support Linode, I've had nothing but a good experience. But I just had to check my credit card to be sure they hadn't lost my details. I've NEVER had to do that before with anyone - they've got to respond fast here because if I don't trust them with my CC then I can't leave five-figure contracts at jeopardy hosted on their servers.
I've been living comfortably on Linode servers for over three years. This is like suddenly being evicted and having to pack my stuff up and find another apartment.
I have to wait for some sort of verification for this but if true then I have to leave Linode. I have client sites hosted here - not for cost reasons, just because I like Linode.
For the sake of $5 a month I can't even take the slightest risk of being criticised for using Linode. And this lack of transparency could be a nail in the coffin here.
I don't want to waste a couple of days on this but that's what's going to be involved if this is true.
I'd say support tickets or posting on their forum[1] may help try to get a response. But based one one of the support ticket responses posted in the comments in this HN story already, it sounds like Linode isn't allowed to release that kind of information yet. They may be waiting on the police and/or their lawyers to allow them to talk publicly about it. And if that isn't the gating factor, they are probably trying to determine what they are required and should share about the incident.
They may be waiting on the police and/or their lawyers to allow them to talk publicly about it.
Waiting for their own lawyers would be a particularly weak excuse. This is a priority, and they are responsible from conveying that urgency to their lawyers.
I've now heard from a number of people using Linode that have suspicious activities on the cc which they used with Linode.
I just called up my bank to tell them to 'block' it as a precaution (I will now have to give them a visit later today to get a new card). I encourage all other Linode customers to do the same, because it'll be easier to just spend half an hour doing this instead of spending hours upon hours disputing specific transactions.
Linode customer support keeps saying they have "no comment" on this issue (which I suppose does make sense -- I'm assuming they've been ordered by law enforcement persons to not share details), so as we're not being given much information to work with... just treat this as a worst-case scenario (all names, addresses, credit card numbers, etc. have been compromised). Do operate now with the assumption that all of this data has been compromised and may very well be public soon.
I would be utterly shocked if nobody using Linode had suspicious activity on their CC. Linode has lots of customers, and at any given time, some of them probably have suspicious activity going on.
There's baseless speculation and then there's I have some information speculation. I'm operating on heuristics which rely on information that is handily available. Yes, in the end you're right, I'm just speculating. But hey, it's better to err on the side of caution.
If it is indeed true that credit card numbers were compromised, it would behove Linode to tell their customers quickly so they can take the proper action.
With this lack of transparency, I feel like I had no choice but to block my card.
There's no lack of transparency here. Linode expressly said in their blog post that no CC details were leaked.
> In addition, we have found no evidence that payment information of any customer was accessed.
The question isn't transparency, but trustworthiness. Either Linode is telling the truth, and this anonymous IRC person with a pastebin is trolling everyone, or Linode is lying (or alternatively, Linode is incompetent and simply didn't detect the CC access). At the moment I'm going with Linode is telling the truth, because honestly, am I going to believe an anonymous person on IRC over a company I do business with?
I'm thinking here why Linode holds CC data on its servers in the first place. Anyone care to weight in here?
Secondly, if they hold that data, it's possible one day someone will find a security breach and will access that data. The best solution is to never hold that data.
Since I don't trust most of the systems I use AND Linode has not denied it holds that data... I'm more inclined to believe in this anonymous IRC guy and err on the side of caution.
If Linode had come out and said "Look, we don't hold your CC number in our database" then I think there would be very little reason to be concerned. However...
Credit card numbers are of pretty low value. Like way less than a buck in medium volume and still just a few bucks for the super premium ones. And there is way, way more inventory of them than interested buyers. The likelyhood of a coordinated break in of a large hosting service with the intention of stealing credit cards is pretty low, and the chance that they'd be exploited so quickly is even lower.
Unless the attacker dumped them all (semi) publicly, the more likely explanation is that the breakin caused people to check their accounts and a statistically normal percentage of them showed fraud from another origin. But anybody who sees it will be sure to get online and find others in a similar situation.
Everybody would be doing themselves a big favor if they stopped treating CC info as the #1 scary OMG data theft. The banks programmed you to care because congress made sure they're liable instead of you. Theoretically you might owe $50 due to fraud but practically you never pay a dime. Sure it's a bit of a pain in the ass to get resolved, but it's not worth stressing about until it happens.
I'd be way more concerned if my hoster lost my contact info, ip logs and identity challenge questions & answers.
I contacted Linode support and they've said in clear terms that they have no evidence that payment information of customers was accessed. I initially signed up for Linode because my friends spoke highly of the tech people working at Linode. Right now amidst all the commotions it's ryan's words (some anonymous dude who joined #linode/irc.oftc.net) vs. an established company's. I'm just going to now stop worrying and get back to my work.
No weird activities on mine either but I will give a call to my CC company anyway. I have had to cancel the card I use on linode twice in the past few months because of suspicious activities. I just didn't think it would be coming from Linode
What about for people who did not use a credit card to pay for linode but instead relied on PayPal. Should they follow the same steps? What about other cautious steps?
I've got a PayPal business mastercard which is connected to PayPal Smart Connect. They may not allow PayPal proper but you could pay using PayPal by way of their card. I use my PayPal card hooked into Smart Connect for a good number of recurring payments like this.
So I guess the answer would be, if you ended up hooking your PayPal account up to Linode in the way I described, yeah follow the same steps as other cards, otherwise it's not even possible to have a problem.
Yeah I did the same thing, but new card takes 5-7 days in my case. Very disappointed that I had to find out about this incident here, imagine all the customers who dont happen to look on slashdot or hacker news.
Wow, four times? You should probably be more careful about who you give your number to. Personally, I usually get a new card every 3-5 months. If someone ever sat on my card number, it's useless to them now. Never had any issues either.
I also do that, but it's because I'm scared of recurring subscriptions that I've forgotten about, especially those that decide to sneak into my pocket after I've deliberately canceled them.
He's not tearing down and setting back up the entire credit line, just the card number associated with it. It won't be reflected on any credit reports.
@kansface It's not bad for your credit rating. A number is simply a representation of the account. The account doesn't change. It's not like getting a whole new item of credit issued. Just the means to access it.
Also, great idea. But a pain, because most of my bills - cell, internet, insurance(s), etc all go through my credit cards. Is a gigantic pain to change the numbers.
I can be in some cases. I got mugged and my card was used to pay for parking garages for 1.5 years until it expired even though it was canceled and blocked by the issuing bank. They said that for some transactions, the blocking mechanisms are so expensive its more economically sane to them to refund whatever was drawn.
I've had roughly the same experience: in the last 8 years, I've had suspicious activity on my CC about 5 times. Each time, the bank caught and trapped it before I noticed and issued me a new card quickly. I've only had to fill out paperwork for a disputed charge once, and it was a 2-page, 2 question, sign-and-mail-it-in deal.
My advice is different, though. I notice that I tend to get lucky in places where people can have very aggravating experiences. I'd say that if you've had problems before, then anticipate problems this time around too. If you haven't, then don't bother.
It depends on how sophisticated the identify theft is. I had a good friend who was taken for about $9000 in credit card fraud in 1998/1999, with Well Fargo. It took him the better part of six months, and endless correspondence with WF to prove all of the purchases were not his. There are lots of stories of people who were financially wiped out, to the point of bankruptcy, because of Credit Card/Identify fraud.
With that said - almost everyone seems to feel comfortable handing out their credit card to random taxi drivers, waiters, sales staff - with no idea whether a copy of their information is being taken down. Heck - if you give them the Credit Card, they even get your CCV as well.
That makes me wonder why the credit card system is so insecure in the first place. Why are credit card systems not secured with a password that the merchant never gets to see? Yet at the same time credit card suppliers keep bragging about how "secure" their cards are.
> Why are credit card systems not secured with a password that the merchant never gets to see?
My bank in Sweden requires MasterCard SecureCode for all online transactions on their debit cards. Stores that don't support it simply won't work with the card.
So, it's up to the bank how secure they want it to be. The technology is there.
Funny that you mention that. After a few bouts with fraud (because as i mention I have my CC out to many services), I was wanting a way to track down the offending service.. I was thinking something along the lines of a vendor-specific set of numbers to be run through.
I even wrote a blog post about it. I probably don't know what I'm talking about, but these were my thoughts at the time:
Wishlist – A Method to Pre-Approve and Track Credit Card Transactions
The issue:
A business using a credit card doing business with a relatively small number of vendors wanting to first avoid credit card fraud (stolen numbers) and secondly wanting to easily track down the offending business.
The idea:
The business would like to approve particular vendors to use the credit card with number 0000-0000-0000-0000 with each individual business pre-approved to run the transaction with a 5th set of identifiable numbers, so something like 0000-0000-0000-0000-0001.
If the credit card is used to make a fraudulent transaction, then ideally, they would have had to have used the 5th set of identifying numbers. This 5th set of identifiable numbers would then allow for easy tracking of the offending vendor, which would allow the business to either re-think doing business with them, or to serve as a starting point discuss security issues with the vendor’s credit card transaction processes.
Summary:
Basically, I believe there may be a need for a new or value added credit card type service. This transaction type would require a 5th set of numbers which have been assigned to pre-approved vendors. This 5 number set (ie. 0000-0000-0000-0000-0002) credit card transaction would most likely prevent theft right off (because the vendor is pre-approved and should provide their own private key (ie. CCV) to put through the transaction). Secondly, if and when the credit card number is stolen and used to make a fraudulent purchase, then, at least with the 5th number set a vendor can be identified and security policy with them can be re-evaluated.
I asked about the security measures and they answered with:
"We appreciate the response, and we can assure you that we have implemented all appropriate measures to provide the maximum amount of protection to our customers."
If it's a debit card that can be used as a credit card (and it must be, otherwise it couldn't have been used to pay for Linode), then it enjoys the same protection as regular credit cards when it's used as one.
Very true. If your bank gives you a separate savings account you can transfer the bulk of your money there and just use the card from the current account to limit your exposure.
Generally speaking, this is one example where having a good deployment system starts to look extremely valuable (along with tested backups and restores for non-deployed data).
If your using lvm you can create a snapshot to do this while online, if not just read from your disk (assuming sda here):
1. (offline) Boot new and old VM servers from live CD
2. old server: dd if=/dev/sda bs=8M | pbzip2 -c | netcat <newhost> <random high port>
3. new server: netcat -l <same port> | pbzip2 -cd | dd of=/dev/sda bs=8M
Compression: You can use something besides pbzip2, maybe pigz of if you only have a single core use bzip2 or gzip.
Security: You probably want to add encryption to this pipeline.
Yeah, I mentioned that at the bottom of my post. Using ssh or some other inline encryption would be a good idea if it is a system you care about. If you have a site to site VPN tunnel between your systems, you can skip adding the encryption.
I thought my only edit had been to replace might with probably. If this was not the case I'm sorry. Maybe I edited it to add that right after posting and forgot-
You don't want to leave out the compression or block size however, so add those back in. Most WAN links are low bandwidth enough that compression will not slow things down (this is usually true even on 1 GBE LAN links for pigz) and in my experience the speed-up is substantial.
pigz is much faster than using ssh compression as it is multicore. apt-get it or http://zlib.net/pigz/
It's pretty necessary if you aren't moving between colocated boxes. "less than a couple hundred GB of data" is still a lot of data to be moving around, even on a 100Mb/s link, which almost no one in America has residentially, and isn't even a guarantee for colo'd machines.
In most cases, compression at the raw block level will result in HUGE size savings, especially since a lot of that may be free space. It might even make this transfer tenable.
That's a great way to screw up the filesystem on the destination. You'll get mounted fs write combined with low-level block writes, filesystem in a blender.
Your procedure already requires booting out of live cd, and then uses "unix way" that might be easy to screw up for less advanced people, so using G4U simplifies it substantially, does the same good job and by no means is a 'overkill' ;)
Is this why Linode doubled the RAM? To bribe us and make us stay. I'm pretty pissed off about this and will be exploring other options. I'm not pissed off they got hacked, I'm pissed off they are hiding and not being forth coming about it. A simple, "We fucked up, we are going to take steps 1, 2 3 to fix it and reduce the likely hood of this ever happening again" will make me happy. I understand that any server can be hacked. I'm stunned that they are storing CC details on the servers, there are ways to go about this without storing them if you want recurring billing.
I am not sure that "gospel truth" is a fair characterization.
Anonymous IRC person has provided verifiable details that strongly suggest he or she had access to Linode administrative systems. Fyodor's post to nmap-dev supports the notion that customer nodes were accessed as well.
Linode has provided no details or evidence of anything.
I don't think one has to take that IRC log as gospel truth to be reasonably concerned about the security of their data stored by Linode.
The only "verifiable detail" I saw in the chatlog was the output of `ls` in the http root. And that's only verifiable because you can try to access that weirdly-named HTML file and get a 200 back. Honestly, that doesn't tell me a whole lot.
Everything else, such as the password hashes, don't seem at all verifiable (even if someone were to crack any of the hashes, you can't verify that the password worked at the time of the hack because Linode has presumably changed them all anyway).
If that's the same one, then that link implies Linode already fixed the issue on or around the 13th. So, I'm wondering if the silence that has some folks here up in arms is indeed because they've been instructed by law enforcement to keep quiet pending the investigation.
Now if we could only stay the torches and pitchforks for a while before this gets sorted out...
417 comments
[ 3.5 ms ] story [ 304 ms ] thread[edit] Just looked at twitter, this tweet doesn't look good: https://twitter.com/Jamiesingleton/status/322730588459114500
But it may just be random coincidence.
[edit again]
Links from slashdot article:
IRC chat: http://turtle.dereferenced.org/~nenolod/linode/linode-abridg...
Link in IRC chat (i think it is of linode.com's web directory): https://bin.defuse.ca/hq0Ay8RzpKdR6vQwYxnmhc
https://news.ycombinator.com/item?id=5541915
Also, the attaker could also change the account email. Neither tokens nor email confirmation would help.
The biggest problem is for people that used the same password on Linode and any other important service. (And, of course, all the CC stuff...)
I don't know though... will wait until more details are available but will be keeping an eye on CC statements / VPS alternatives.
Doesn't sound very plausible to me.
I think it was just an unfortunately timed third phase of their upgrade plan.
> 05:42 < ryan||> credit cards were encrypted, sadly both the private and public keys were stored on the webserver so that provides 0 additional security
> 06:00 < ryann> They did try to encrypt them, but using public key encryption doesn't work if you have the public and private key in the same directory
http://turtle.dereferenced.org/~nenolod/linode/linode-abridg...
† We are not one of those.
Will add that just having gone through an ICANN registrar audit (which by the way were specified and supposed to be done literally 10 or 12 years ago but never requested by ICANN) with a third party company hired (accounting firm) it's total compliance theater.
Add: "hired by ICANN after a bidding process". Same happened with data escrow which was just implemented a few years ago and is operated by Iron Mountain.
The 'lower' your level, the easier the PCI audits are. If you are level 1 you have mandatory external audits. If you are level 2 you have a 'self assessment' which is basically a checklist which says "Yes, I promise I'm in compliance".
If you have a confirmed breach, you are upgraded to Level 1 merchant audit requirements. This is generally quite costly as the external audit is extensive and must be paid for.
But if the things he claim in there is even half way true, nobody involved with linode should ever be allowed to be in business every again.
http://www.linode.com/y_key_57284cb2de704e02.html
Sysadmins being lazy.
Somebody needs to get a file from a workstation to a remote machine. There's a firewall in the way somewhere that prevents SSH directly between them or one of them is a Windows box that isn't running an SSH server.
The "correct" solution is complicated and takes 5 minutes to setup. So the sysadmin just copies the file to the web server and downloads it with a browser on the other end. Because port 80 is always open.
Its also why we invoice and take wire payments rather than storing CC details. There's just so much to go wrong.
Also PKI is shit for this sort of thing. As demonstrated, the moment that public key is gone, then the whole system falls like a house of cards. For the non believers of this fact, why else would there be a certificate revocation list and root CA updates for windows periodically...
Also you're fined by the credit card companies if you lose card information. I believe it's a per card fine, so it get expensive really quickly.
Actually I don't get why any company would choose to store credit card information, when most payment providers will do it for you.
But yes, significantly better than other situations.
But if that happens, it's not your responsibility (at least not 100%), it's theirs
Much like Linode are responsible for hosting my clients site but I sigh am accountable when something goes wrong.
Card vaulting as a service: https://spreedly.com/ ($10/mo for up to 5000 cards)
Also I do not think, but I am not sure, that fraudulent wire payments/transfers are reversible.
Wire transfers often are not able to be undone once they happen (and are accepted by the other bank). This is the reason why there's so much verification that happens in wire transfers. (I helped develop 2nd factor authentication used for authenticating wire transfers for a financial company)
Credit card charges can be reversed.
http://www.reba.net/news/wtransfer
But I just checked and my credit card haven't been used anywhere I didn't use it.
If you believe this Ryan guy, credit cards stored on the same server as the key to decrypt them, Lish passwords stored in plain text, they've known for some time and lied about what actually happened and now they're saying "we won't do anything about it" via email?
"You are of course free to take any steps you deem prudent or necessary to ensure the integrity of your online presence."
Unbelievable.
Edit: not to mention they "made a deal" with the hacker not to tell anyone? What the hell?
there is a mixture of truth and lies on both sides, to be honest.
i am annoyed with it, because i reached out to several linode employees privately to given them an opportunity to explain what was going on -- they either said 'no comment' or said my linode was fine.
based on the irc log, that is clearly not the case. which is why i decided to raise my concerns publically.
luckily for me, my linode was not doing anything mission-critical, just some secondary monitoring and running an ircd for a network i like using, but there are others who are using linode for mission-critical work, and they deserve more transparency than this.
Sorry if you get offended that they didn't tell you much more. But seriously? you are not special. This whole thread is a lynch mob.
What Linode did may, or may not, be dumb. They are being tight-lipped so we can only guess.
That's a rather key assumption. If you don't believe him, then all you have is a trolling (or at least self-aggrandizing) hacker whose credentials consist solely of logging into an IRC channel, refusing to identify who he was working with, and offering no tangible proof of having compromised any CC info.
On the other hand, it's conceivable that if ryan managed to get into the files a customer was hosting on Linode, and that customer was improperly storing CC info, then their customers' info would have been vulnerable, and ryan's claims would be sort of half-true. Even so, that wouldn't directly affect other Linode customers or put liability in Linode's lap.
> We are unable to release any additional details regarding this incident at this time, as there is an ongoing investigation.
> We are unable to release any additional details regarding this incident at this time, as there is an ongoing investigation.
Edit: Oops, didn't refresh before replying. jmilloy got it first! Sorry about that.
"Where are the keys?"
"In the locks."
If the hacker's claims are true (Would appear so, the directory listing checks out) then Linode really need to address this ASAP. Passwords are one thing but to have CC details leaked is even worse. I'm not familiar with CC processing but it seems like bad practice to store the encryption keys on the web server.
> 05:05 < ryan_> manager.linode.com was breached with a coldfusion exploit
...
> 05:33 < Ruchira> ryan||: give us the link to cold fusion vulnerability that you are talking about
> 05:34 < ryan||> Ruchira: 0day
> 05:34 < ryan||> linode staff apparently failed to deduce it themselves and relied on chmodding CFIDE to 000
There, I fixed it for you. Working with ColdFusion is like this: http://25.media.tumblr.com/38d67be62da60b4d3aa1d0ac22e4e314/...
"This hotfix resolves a vulnerability that could be exploited to impersonate an authenticated user (CVE-2013-1387).
"This hotfix resolves a vulnerability that could be exploited by an unauthorized user to gain access to the ColdFusion administrator console (CVE-2013-1388)."
http://www.adobe.com/support/security/bulletins/apsb13-10.ht...
http://turtle.dereferenced.org/~nenolod/linode/linode-abridg...
--------- 05:43 < ryan||> Well linode also had terribly configured coldfusion
05:57 < ryann> <cfif ListLen(cgi.script_name, "/") gt 2 AND ListGetAt(cgi.script_name, 2, "/") eq "linode" AND NOT ListFind("index.cfm,linode_edit.cfm,linode_resize.cfm,label.cfm,cancel.cfm,dc_choose.cfm,su.cfm,pastdue.cfm", ListGetAt(cgi.script_name, 3, "/"))> <cfinclude template="/members/linode/common/dsp_topNav.cfm"> </cfif>
05:57 < ryann> this code
05:57 < ryann> It's so dirty I feel bad reading it
Also if you check out my reply below I've C&P'ed the chat logs where he claims it is a zero day:
https://news.ycombinator.com/item?id=5552992
Plus another commenter has linked to a security advisory for exploits in CF that was issued a few days ago.
As someone who still maintains a very old CF application, I am sure to lock down access to the admin site via IP restrictions.
http://www.carehart.org/blog/client/index.cfm/2013/1/2/serio...
The basic overview is this: CF servers have an administrative portal at /cfide/. A bug in the scheduler code (think cron) allowed remote attackers to upload arbitrary code to the server and then execute it. Savvy attackers could upload their own backdoors directly into the administrative folder on the site and then execute that code to gain additional access.
As a Linode customer (admittedly only for a small VM I play around with) I have to say I've been impressed with their service and their prices of course, and I'm waiting for further confirmation about the depth of this hack. I was unaware Linode was using ColdFusion. It should be pointed out that CF is a very mature language, akin to ASP.NET. It is actively maintained by Adobe and used by a huge number of websites globally.
Source: I'm a long time ColdFusion developer.
Poor show Linode. (edit: worth noting I use the card with other things too, I have no confirmation it was leaked through Linode other than the compromise happening at the same time these supposed leaks happened).
Linode has addressed the breach, but assured customers nothing of value had been compromised. This infers two thoughts. One: they knew of the breach and lied, thereby unveiling a unforthcoming and dishonest nature. Or two: they did not properly investigative the severity of the issue, thereby suggesting incompetence. Both equally reprehensible.
https://www.digitalocean.com/blog_posts/resolved-lvm-data-is...
If I had a choice between a VPS provider who either:
- Only has large issues (eg. leaks credit card data) and goes weeks without reporting them to customers, or
- Has lots of small issues (eg. forgetting to clean the free space of LVM volumes) but fixes them the same day,
I'd much prefer the latter.
EDIT: My bad, apparently the problem was reported on March 27 and wasn't fixed until April 2.
Personally I took precautionary measures and just called my bank to replace my credit card, which I think is the sane approach, as when it comes to hacking you have to assume the worst.
However, your statement on "has lots of small issues but fixes them the same day" is just stupidly childish. Linode's issues are bigger just because they are a bigger target.
Maybe you don't have anything of importance on your VMs, but plenty of people do. That data could contain credit card data, passwords, etc, etc, etc. It is very much a large issue.
According to DigitalOcean, they stated that this impacts 3% of all machines, only the largest and most expensive servers. None of the smaller plans were leaking data.
I don't know how many credit card numbers were leaked from linode, but I'd guess more than 3%.
Second, if security is important to you, you can use 'dd' to clear the machine yourself before shutting it off. (In fact, good data destruction policies mandate the use of 'shred' et al anyway). On Linode, affected users don't even have a workaround (like this) to avoid information compromise.
also a twitter feed for the customers page ?
Wouldn't hurt just to ask your bank to re-authorise it anyway? It will change the three digits on the back.
I know someone who got their debit card cloned. While the bank eventually repaid him, that did nothing to repay him the additional fees he owned his normal debtors (e.g. rent, utilities, etc).
With a credit card you aren't losing "actual" money. You are losing the bank's borrowed money which the bank pays back. With a debit card you're losing cash which you won't be able to replace yourself and which the bank might take days to weeks to replace.
Even if you NEED to borrow while your credit card is out of commission you can either use the overdraft facility on your debit card or other quick sources of credit. Hard to get quick cash without going to a pawn shop.
a good practice that bankers constantly tell me is to have a separate credit card for online purchases for the fact alone that it is one step removed from your checking account.
Visa has a zero liability program for debit card - http://usa.visa.com/personal/security/visa_security_program/...
These are the FTC's rules [1], I'm not sure if Visa or Mastercard can make them 'better' (give you a larger window). They have an interesting tidbit below their chart -
>If someone makes unauthorized transactions with your debit card number, but your card is not lost, you are not liable for those transactions if you report them within 60 days of your statement being sent to you.
Isn't it free to get a new card? That'd be the easier way than worrying.
[1] http://www.consumer.ftc.gov/articles/0213-lost-or-stolen-cre...
> [With a debit card,] "Until the bank provides provisional credit, you could temporarily be out of pocket for the amount in dispute," said Richard Foley, an FDIC attorney who specializes in consumer issues. "This would not typically happen with a credit card because consumers can withhold payment of the amount in dispute."
> Also, as discussed on the next page, consumers have better federal protections when they purchase faulty goods with credit cards.
Not that having your account drained doesn't suck, but your worst case scenario there isn't terrible unless you fail to check stuff and be responsible.
My day job had some Google Apps account compromises last month, and this is making me paranoid that the database that contained hashed/salted passwords for our Intranet hosted on Linode was the culprit. The time frame doesn't seem to line up, but we didn't see any evidence of phishing or compromised desktops being involved.
(Don't want to spread fear - I haven't been able to find any evidence our Linodes were compromised either.)
I'm normally huge a Linode evangelist, but I'm severely disappointed with the lack of transparency on this. I'm debating right now whether to rebuild all our nodes from scratch as I'm not sure they can be trusted.
Hopefully, they own up and start being transparent.
If this is true then what alternative hosts should I look at, besides AWS?
That's just poor security and 100% they're own fault. I accept that there are security issues with every platform, but basic security measures and being transparent is still expected. My biggest issue with them in all of this is not being transparent.
Looks like someone who likes attention on some random IRC channel who is apparently a hacker may have hacked our system and we don't know who/when/where/why/how or what they may have got. Nor are we sure we were even hacked???
It takes time for people to investigate stuff. It's not just a couple hours. Also some random guys words on IRC (who could very well own INSERT RANDOM HOSTING COMPANY for all we know and looking to scare people off Linode) should be taken with a grain of salt.
I've only used DigitalOcean. My anecdotal experience from running a Chef Server on a 1GB instance has been pretty mixed. The price is good, but network and CPU performance feels very variable to me. A month ago their Amsterdam servers were unable to be resized, and there was nothing about it on their status page. I tweeted and was told they'd be working "some time later today". Doesn't fill me with much confidence in general.
I'd still choose Linode for anything of importance - their long reputation is well earned in my opinion. But, if this breach is true, I hope they handle it well.
That is unfortunately the problem though. If this is true, they have already handled it terribly as it has already been 2 weeks since the attack.
This could very well be the hackers own submission to /. trying to get more attention for his hack by claiming he has CC numbers which I doubt he has.
A month later, spend 30 minutes on the phone with CC company only if strange transactions appeared.
Not the end of the world. The CC industry is set up well to handle this kind of thing.
Kind of sucks to have to spend hours doing that for someone else's oversight. It's not the end of the world, but it paints a clear picture about where a company's priorities are.
Even just dealing with all of those is going to be a pain in the ass though. I'd probably end up spending hours all together just on hold with some of those people.
Edit: Netflix. I forgot Netflix.
> you have to find all charges going to your old CC and then deal with moving every one of those accounts to your new one when it gets there. Hopefully you don't incur any late fees while you're going through the process!
If even an iota of what I read in the abridged IRC log is true, Linode doesn't seem to care much about security or protecting Linode customer data. I mean, storing "encrypted" card numbers alongside private/public keys? Really.
I will happily dismiss this breach, not because they didn't make some amateur crypto mistake, or because they weren't using freaking ColdFusion, or because they were storing data in some nice compartmentalized form, I reject because this happens every single day and has done for decades, and there is an entire sub-industry built around its after-effects. If you don't understand this you shouldn't own a credit card.
If you type a credit card number in online not expecting to recuperate any damage caused from your card company, call them up now for clarification or cancel the damn card. That's equivalent to stuffing cash in an envelope and posting it to Nigeria because some prince promises he'll keep it in a safe for you. It's 90% the reason you should be using credit cards in the first place. Think.
Linode should not be rubbished here. They've got one of the largest VPS installs around, so they most likely know their shit. They make an ultra-common CC mistake that has happened daily for almost 20 years now, by companies large and small, got pwned due to a bug in someone else's software, and you think I'm going to play along with the righteous indignation bullshit here? GTFO.
Let he without sin cast the first stone. Despite 20+ years' experience I still cannot cast that first stone. I make bullshit mistakes like this every day, and despite your grandiose delusions you probably do too.
As for whiners complaining about their data suddenly being insecure, well, data security 101: you're making the same bullshit mistake Linode are making, and despite that you're complaining about it. If you care about data security in the "cloud", hosting it on a freaking VPS is not the way to do things.
Of course not, and this drives to the very core of risk management. I've signed up for some very shady online services in my time, doing so in the full knowledge that should a product or service not be rendered as advertised, I am guaranteed to be able to reverse the relevant charge. Even when I the consumer am doing something shady (in a case last month, attempting to import goods I knew weren't certified for the EU), the system still works for me. This is the sole reason I use a credit card rather than, say, my current account's Visa number.
It's not even about assessing the risk of whether or not you're going to get ripped off, but whether or not a particular company will cause you the inconvenience of the aforementioned phone calls.
If you work on the assumption that you card data is safe, you quite simply aren't safe enough to be in possession of a computer or card. Credit cards aren't built on that assumption, instead their entire motivation is based on risk profiling both the consumer and merchant, and terminating agreements when various thresholds are reached. In return the industry guarantees that in the minority of cases where things go wrong for the consumer, the problem can be corrected swiftly.
It's understandably upsetting that their customer database might have leaked, and I can genuinely understand peoples' concern over that. But as 4chan has taught us, there are very few people left in the west whose address and telephone number aren't available within even an hour's Googling.
As for locating confidential data on machines shared with other customers and managed by a piece of unaudited software, I have no sympathy for that. That's the price of a VPS, and why it's so heavily discounted compared to real hardware.
none of them get a free pass they all suck, but the one that just got busted is probably going to be a little more security focused in the near future.
Hmm stay at a place that just got burned, or expend lots of effort to move to a place that hasn't been burned yet...
If your expectation when taking credit card numbers was "I'm confident in my abilities to keep this information safe, and if I get hacked I expect my customers not to move to another service and never, ever touch mine with a ten foot pole", then the problem lies squarely with you, the uninformed business.
I think that Linode did a big mistake here. Let's wait for a formal communication.
But this is the moment to support them. Yes, maybe sounds crazy.
When you host on any third party datacenter, you take risks that something like this could happen. So, deal with it. Check your credit card, if your receive something wrong, call to your card and that's all. But we need to support also the good work, and this guys do great work in the hosting business. Just my opinion.
I can think of two good reasons why you should flee Linode. It remains to be seen if either are actually true, and until indications say yes, then panic is unwarranted:
1. If it becomes apparent that Linode is far more vulnerable to hacking than other hosting providers. But one hack alone does not prove this.
2. If Linode grossly mishandles the situation. There have been a couple of allegations to that effect so far, but nothing substantial. I don't see any reason to claim that they've done this yet.
The alleged hacker has made serious and specific claims, and Linode has done jack shit; without more information, how should I proceed? I don't want to call my bank and waste time getting a new credit card (not to mention replacing a million and two services) without a confirmation and I can't get a confirmation because Linodes people are having a circle jerk (or whatever the hell they do).
There was an email notification a few days ago.
Linode's handling of the Bitcoin incident last year was sub-optimal. This too has been sub-optimal, given that credit cards were exposed but all we heard on Friday was to change our passwords, and even that was claimed to just be a super-careful precaution.
Linode needs to start giving us some frank talk ASAP. They've already burned through a very generous helping of benefit-of-the-doubt.
If that's even true.
This is not a mistake/technical issue, it's becoming an ethics/service one.
06:07 < ryannn> They say there's no 'central weak point'
06:07 < ryannn> Yeah there is, there's the developers
06:08 < ryannn> There's been bugs in the client that have allowed the blockchain to split previously
06:08 < ryannn> One could just backdoor the bitcoin client binaries, not the source.
06:08 < ryannn> Nobody would figure it out until it's too late
http://turtle.dereferenced.org/~nenolod/linode/linode-abridg...
This sounds very very bad, and as a customer it's very off-putting.
If the allegations are true, then Linode was keeping encrypted CC numbers, with the decryption key in nearly the same place.
Trying to make the analogy more sufficient by incorporating this type of fact would only make the carjacking analogy more absurd. At the end of the day, an analogy is not needed.
"No comment."
I've noticed before that stuff from the HN frontpage appears on /. one to three days after, but I've never seen it for links to slashdot :-)
Also off-topic: I've noticed that as well with Slashdot, which is why I lurk HN pretty regularly now. Plus, some of the front page material on /. does more to insight angry discussion, and the community has become increasingly more vitriolic.
At least here, even if someone's brash, they're fairly honest about it (in general). I've even seen a number of disagreements that have been respectful and cordial. It's sad to say, but that's a rare thing these days.
Alas, they have not.
I just blocked my credit card, and Linode will not get anymore of my money. Too bad, i enjoyed my stay very much, but no matter what their response will be, I doubt I will be able to believe them. =/
e.g: Acme, Inc. sends Stripe your CC#, Stripe sends them some unique token, and they store that; correct?
So Stripe still has your CC#, and is at risk.
So this is really just risk mitigation; what I think TP is suggesting we need is unique authorizations at the banking level.
Something on the order of virtual credit cards, or temporary tokens, which are ultimately verified by your bank [or in other words: the lender(s) making anti-fraud guarantees, etc.]
(e.g: this token is authorized for 24 hours up to this limit; this token is authorized indefinitely up to $xx/mo.; this token is authorized for 1 year; etc.)
Edit: yes Stripe has your number, but since their sole business is about securing that information, they probably do a better job of it than your typical online merchant.
You then put your trust in the gateway/processor to store the credit card. Which I assume is most likely behind the best possible security stuff money can afford. Since that's their entire business. One screw up and their gone.
Stripe (and probably others) has functionality like this where the seller's server never sees the CC number, and developers can store a unique token to re-charge the customer at a later date.
You really really don't have to. Any payment processor that isn't horribly incompetent does the unique token authorization scheme.
Storing CC #s for recurring payments is solely the domain of incompetents who have no business accepting payments from anyone.
It's never cool to be actually- or quasi-locked into a vendor.
Otherwise you just ask customers to re-authenticate. Often you have a few months headway for a switch like that.
I've been living comfortably on Linode servers for over three years. This is like suddenly being evicted and having to pack my stuff up and find another apartment.
I have to wait for some sort of verification for this but if true then I have to leave Linode. I have client sites hosted here - not for cost reasons, just because I like Linode.
For the sake of $5 a month I can't even take the slightest risk of being criticised for using Linode. And this lack of transparency could be a nail in the coffin here.
I don't want to waste a couple of days on this but that's what's going to be involved if this is true.
[1] http://forum.linode.com/viewtopic.php?f=20&t=9978
/end nitpick
Waiting for their own lawyers would be a particularly weak excuse. This is a priority, and they are responsible from conveying that urgency to their lawyers.
I just called up my bank to tell them to 'block' it as a precaution (I will now have to give them a visit later today to get a new card). I encourage all other Linode customers to do the same, because it'll be easier to just spend half an hour doing this instead of spending hours upon hours disputing specific transactions.
Linode customer support keeps saying they have "no comment" on this issue (which I suppose does make sense -- I'm assuming they've been ordered by law enforcement persons to not share details), so as we're not being given much information to work with... just treat this as a worst-case scenario (all names, addresses, credit card numbers, etc. have been compromised). Do operate now with the assumption that all of this data has been compromised and may very well be public soon.
With this lack of transparency, I feel like I had no choice but to block my card.
> In addition, we have found no evidence that payment information of any customer was accessed.
The question isn't transparency, but trustworthiness. Either Linode is telling the truth, and this anonymous IRC person with a pastebin is trolling everyone, or Linode is lying (or alternatively, Linode is incompetent and simply didn't detect the CC access). At the moment I'm going with Linode is telling the truth, because honestly, am I going to believe an anonymous person on IRC over a company I do business with?
Secondly, if they hold that data, it's possible one day someone will find a security breach and will access that data. The best solution is to never hold that data.
Since I don't trust most of the systems I use AND Linode has not denied it holds that data... I'm more inclined to believe in this anonymous IRC guy and err on the side of caution.
If Linode had come out and said "Look, we don't hold your CC number in our database" then I think there would be very little reason to be concerned. However...
Unless the attacker dumped them all (semi) publicly, the more likely explanation is that the breakin caused people to check their accounts and a statistically normal percentage of them showed fraud from another origin. But anybody who sees it will be sure to get online and find others in a similar situation.
Everybody would be doing themselves a big favor if they stopped treating CC info as the #1 scary OMG data theft. The banks programmed you to care because congress made sure they're liable instead of you. Theoretically you might owe $50 due to fraud but practically you never pay a dime. Sure it's a bit of a pain in the ass to get resolved, but it's not worth stressing about until it happens.
I'd be way more concerned if my hoster lost my contact info, ip logs and identity challenge questions & answers.
I contacted Linode support and they've said in clear terms that they have no evidence that payment information of customers was accessed. I initially signed up for Linode because my friends spoke highly of the tech people working at Linode. Right now amidst all the commotions it's ryan's words (some anonymous dude who joined #linode/irc.oftc.net) vs. an established company's. I'm just going to now stop worrying and get back to my work.
On an interesting note, the big target who actually incurred identifiable damage was seclists.org: http://seclists.org/nmap-dev/2013/q2/3
Well that's a first. They were very evasive about it earlier.
Mine is not a CC or a debit card, it's a 'prepayed card' so liability is limited.
Still, I'm considering calling the lost/stolen line of the card.
So I guess the answer would be, if you ended up hooking your PayPal account up to Linode in the way I described, yeah follow the same steps as other cards, otherwise it's not even possible to have a problem.
I live on the internet. Put my credit card out on many services. Over the last 5 to 8 years I've had my credit card numbers taken I believe 4 times.
Never had to dispute it once. These Credit Card companies and Banks have a stake in not allowing your account to be drained.
I think it would be a waste of time to go out and cancel our CC until hearing from Linode that yes, CC information was taken.
Also, great idea. But a pain, because most of my bills - cell, internet, insurance(s), etc all go through my credit cards. Is a gigantic pain to change the numbers.
Are you thinking of expired cards? That is different.
My advice is different, though. I notice that I tend to get lucky in places where people can have very aggravating experiences. I'd say that if you've had problems before, then anticipate problems this time around too. If you haven't, then don't bother.
With that said - almost everyone seems to feel comfortable handing out their credit card to random taxi drivers, waiters, sales staff - with no idea whether a copy of their information is being taken down. Heck - if you give them the Credit Card, they even get your CCV as well.
My bank in Sweden requires MasterCard SecureCode for all online transactions on their debit cards. Stores that don't support it simply won't work with the card.
So, it's up to the bank how secure they want it to be. The technology is there.
I even wrote a blog post about it. I probably don't know what I'm talking about, but these were my thoughts at the time:
Wishlist – A Method to Pre-Approve and Track Credit Card Transactions
The issue:
A business using a credit card doing business with a relatively small number of vendors wanting to first avoid credit card fraud (stolen numbers) and secondly wanting to easily track down the offending business.
The idea:
The business would like to approve particular vendors to use the credit card with number 0000-0000-0000-0000 with each individual business pre-approved to run the transaction with a 5th set of identifiable numbers, so something like 0000-0000-0000-0000-0001.
If the credit card is used to make a fraudulent transaction, then ideally, they would have had to have used the 5th set of identifying numbers. This 5th set of identifiable numbers would then allow for easy tracking of the offending vendor, which would allow the business to either re-think doing business with them, or to serve as a starting point discuss security issues with the vendor’s credit card transaction processes.
Summary:
Basically, I believe there may be a need for a new or value added credit card type service. This transaction type would require a 5th set of numbers which have been assigned to pre-approved vendors. This 5 number set (ie. 0000-0000-0000-0000-0002) credit card transaction would most likely prevent theft right off (because the vendor is pre-approved and should provide their own private key (ie. CCV) to put through the transaction). Secondly, if and when the credit card number is stolen and used to make a fraudulent purchase, then, at least with the 5th number set a vendor can be identified and security policy with them can be re-evaluated.
http://www.redbridgenet.com/blogging/wishlist-a-method-to-pr...
"We appreciate the response, and we can assure you that we have implemented all appropriate measures to provide the maximum amount of protection to our customers."
Yeah, I mentioned that at the bottom of my post. Using ssh or some other inline encryption would be a good idea if it is a system you care about. If you have a site to site VPN tunnel between your systems, you can skip adding the encryption.
It wasn't there when windsurfer replied to you (I was reading the thread earlier), hence his question.
let SSH handle compression for you instead.
You don't want to leave out the compression or block size however, so add those back in. Most WAN links are low bandwidth enough that compression will not slow things down (this is usually true even on 1 GBE LAN links for pigz) and in my experience the speed-up is substantial.
pigz is much faster than using ssh compression as it is multicore. apt-get it or http://zlib.net/pigz/
In most cases, compression at the raw block level will result in HUGE size savings, especially since a lot of that may be free space. It might even make this transfer tenable.
Anonymous IRC person has provided verifiable details that strongly suggest he or she had access to Linode administrative systems. Fyodor's post to nmap-dev supports the notion that customer nodes were accessed as well.
Linode has provided no details or evidence of anything.
I don't think one has to take that IRC log as gospel truth to be reasonably concerned about the security of their data stored by Linode.
Everything else, such as the password hashes, don't seem at all verifiable (even if someone were to crack any of the hashes, you can't verify that the password worked at the time of the hack because Linode has presumably changed them all anyway).
Now if we could only stay the torches and pitchforks for a while before this gets sorted out...