The most significant security problem with Namecheap is really this: It only takes a 4 digit PIN to perform any action on an account through live chat (which seems to be outsorced to Eastern Europe), even if the account is protected with a 2FA... All you need is the PIN, and an attacker can do anything to the account.
You can get AWS customer support to reset your password if you know the last 4 digits of the credit card used to pay for the account. This is the same info that's printed on any credit card receipt.
If they have your bank account number for whatever reason you can also use last 4 of the bank account number. Your bank account number is not secret by design and most people only have one.
That's not good verification. It takes a couple of minutes to produce convincing fake ID scans, and they aren't going to have anything to verify them against.
And presumably they wanted you to send those photos to them as an unencrypted email attachment, right?
Most will require a password reset email, I'd say that's significantly better than asking for ID scans.
Edit: Since I'm getting some downvotes I'd really like to know how one could possibly argue that asking for ID scans is better than email resets. You can't really forge the ability to receive email at an address, but you can very easily replace the name on an ID scan.
Even if the email account is compromised it's still stronger proof of identity than ID scans.
An attacker can't just pretend to be able to read your email, such ability is too easy to conclusively prove. To be able to read your email they need to hack you somehow.
But for a fake ID the attacker only needs to throw your name in a PSD and they're good to go.
As someone mentioned above, they publish a gpg key for sending this data.
As for fake IDs, yes, it is certainly possible to create them. But when it is so much easier to socially engineer your way into another service like Namecheap, it creates a disincentive for going after Gandi (and other similar hosts).
No security measures can be foolproof, possibly short of sending someone to your home to take a DNS sample, but at least they're trying for a better solution.
Other comments in this thread have indicated that Gandi will take GPG-encrypted emails have have published their public key for this purpose: https://wiki.gandi.net/en/gandi/documents
Faking ID scans adds a whole layer of law enforcement on top. I'm uncertain about the situation in the US, but in germany the fake itself is punishable by law (up 10 ten years). It also creates more traces to look at and creates work. You'd also need much more information to create a convincing fake id scan of your intended victim. It's all about increasing the amount of work for the would be attacker.
But producing fake scans isn't covered by this law, scans aren't even an official document. In fact, it is illegal for a german company to ask you to send them scans of official documents.
> You'd also need much more information to create a convincing fake id scan of your intended victim
To fake a good enough passport scan you'd need your victims name. That's all the rep is going to have.
That was a few years back when another well-known registrar only required the last 4 digit of the customer's credit card, and would even help them guess if they didn't remember.
I sent the ids by fax (yeah a few years back, I still had a fax machine).
I thought asking for id's + phoning on the number listed in the whois database was a good cross check, especially back then.
FWIW I painlessly transferred five domains away from Gandi (ironically, to Namecheap), and I was never prompted for anything like a scan of my passport.
>Without context, anybody could say the same thing about anything. Care to share more?
Sure! Gandi received an abuse report regarding someone using one of my domains to scan for open dns resolvers. I informed Gandi that there was nothing I could do about this and expected that to be the end of it. Instead, they suspended my domain and started demanding that I send them ID proof.
As I needed the domain back I sent them a redacted photo of my id card, after which they demanded to see the full id. I decided to terminate my relationship with them.
I feel that this was absolutely unacceptable and likely unlawful behaviour from them as they had absolutely no need for that information. This wasn't a whois dispute.
They publish a GPG key to use for this purpose, which puts them leaps and bounds ahead of most other hosting/domain providers who do identify verification.
I've had process issues like this with them; their CEO is responsive on email/Twitter and the email alias on this page: http://www.gandi.net/no-bullshit gets things fixed. They're not perfect, but they are quite human.
Didn't Cloudflare just launch a domain registration service for its high profile clients which can't afford downtime due to a support worker making a mistake like this?
If it's "Whats your mother's maiden name?" and they let you reset it in the browser, it's a bug.
But if they send you an email (in my case to Gmail, that has 2FA turned on), then it is a feature, because then you'd be required to either 1) intercept the recovery email (and get the password reset URL) or 2) know the format of the password reset URL and just happen to guess mine after brute-forcing every possible link (assuming there is no timeout for the URL or anything else like that).
I had an interesting thought (literally as I was reading your comment) about improving "forgot password" emails, albeit only likely useful for the technically minded:
Have the customer provide an SSH/GPG public key, and store it with the account.
When a password reset is requested, encrypt a random string using said public key, and email it to the email for the account.
An attacker who may have breached your webmail is then reasonably unlikely to also have your private key to decrypt the string.
Follow the link (which didn't necessarily need to be encrypted) and enter the string you decrypted to reset the password.
On a related note: do any/many sites with 2FA, require the 2FA code to do a password reset?
That's basically what TOTP/HOTP authentication tokens are, which many sites (including Google, AWS, Github) etc use for 2FA - https://en.wikipedia.org/wiki/Google_Authenticator.
When you set it up, the service provider creates an 80 bit secret key, which you enter into your local device (or some implementations create a QR code) and then whenever you log in you need to provide a 1-time password from the app.
I'm aware of 2FA using (T|H)OTP, my thought was that a GPG/SSH key can be stored in a secure and yet reasonably easy to use way, effectively offline (i.e. add a passphrase and store it on a USB key or similar).
With a 2FA code, you either a) use the same code they use for regular logins, or b) require them to find a way to securely store the (T|H)OTP secret and then add that information to a 2FA app when they want to do a password reset.
I realise the pubkey concept is more than most people would bother with (or even be able to get through on their own), and I think the first 2FA option is definitely better than no extra security at all on password resets, but my thought was about increased security for those who are particularly paranoid/security conscious.
If you enable 2FA on an account with a service, and subsequently lose access to 2FA otp's (e.g. phone lost/wiped/etc) and lose access to (or never kept) the recovery codes, you generally lose access to the account.
This is similar, but with the benefit that you can keep the key secure by default - i.e. put a passphrase on the key and then store it wherever you like.
In reality, if you fail this "recovery" method, my next suggestion would be a billing based one (i.e. talk to a human, get confirmation of previous invoice details, what is being billed for, how it's paid for, etc)
That's bad, really bad. No 2auth can save you from humans who do support.
I also had one of my VPS attacked recently, and I feel for you.
But the name namecheap says "cheap". Maybe they are indeed cheap? I'm not sure the same would have happened with say HE. You pay, but you know what you pay for and get in return.
Personally, I am thinking about moving from a "manually setup" distribution to a "no ssh but deploy", so as to ease reimaging in the future. This way, if a server is compromised, all I have to do it to start the install of a new one.
Any suggestion for tools to do that with Debian distro? (yeah I could write a shell script, but I think there must be better tools out there)
> No 2auth can save you from humans who do support.
Well, conceivably the second factor could be used to generate crypto key material which is used to decrypt/unlock one's record, so without the second factor even support couldn't read & edit one's record.
Nothing can stop support from deleting & recreating a record though.
> Any suggestion for tools to do that with Debian distort?
If you write apps, package them as Debs.
If you need to configure other Debs, make config packages with config-package-dev [1] from the DebAthena project.
Create a metapackage that depends on your software + config packages, and your setup process just needs to be "add private apt repo, apt update, apt install <metapackage>".
On another note, there 2FA seems to broken as well. I once received a text about resetting my Instagram account from the same number that they send me to authenticate my login session. I've never had an Instagram account.
It's true :) We hear about it on Twitter all the time that the same number we use for 2FA is also texting 2FA codes for the other services I mentioned.
Does Namecheap claim to take backups? Even if they do you should be taking backups as well if you care about your data.
I do agree with more login forms needing to support 2FA. At this point I wish almost everything did. It is a bit more hassle but is easy to manage for me at least.
the first error was buying server space from namecheap. good domain server but they get hit frequently being a midsize provider of services, they have enough bait and not enough people to protect it.
The domain registration process isn't automated from my experience but it works well. However, I think the meta is (I think many people will agree) to not mix domain and hosting with the same provider. For example, if you get your domain from namecheap, you should not do hosting at namecheap. Therefore, the argument is that if you want to buy a domain name from Namecheap (as they're pretty decent), you shouldn't do hosting there.
I don't use their hosting, but I have a lot of domains with them. Anytime I've had an issue with the settings on a domain, they've been quickly resolved.
Just a few weeks ago I was getting a domain set up with Amazon SES, and one Daria P. helped interpret Amazon's docs to get it verified with them, and explained how I was using the dig command incorrectly to inspect the domain's settings. It's rare you see a support person do that at any company.
I think this is a very good point that I also overlooked when I first read the article. If someone hacked my gmail account, I honestly am not sure if there would be any account of mine that would be safe. Anyone using the Internet today has to put utmost care into protecting their email address and most email providers enable you to do that fairly easily.
There was an article here a few months ago promoting logging in via email token as the only way to log in instead using passwords. Because as you said, 99% of websites allow you to reset the password anyway if you control email, so why bother having insecure passwords? If I remember correctly then that article was fairly well received.
And this is an argument for making sure your email provider has two factor authentication to avoid having an external breach that could give someone access to accounts that do not support 2FA.
I'd love to have an option on services where I define a X-hour wait period for manual password resets. That is, "oh, I've lost my email account and I need to reset a password so I have to access my account through pleading over Live Chat... they can do that but there's an X-hour wait period before you will gain access to the account."
Which is great until the customer actually needs to access the account.
$CUSTOMER calls in, their nameservers are down and nobody has the account password. Do you think the management at $CUSTOMER is going to accept "hey we need to wait 6 hours to get our site back up because namecheap wont allow us in"?
Let the customer set it during sign-up as part of the password reset process. You set up your email address and password, let them choose $HOURS for last-line-of-defense password reset.
I don't see any perfect answer here. Ultimately you need a way to recover your account when you've lost all of the "somethings you have" and you've lost your "something you know", but then that allows a social engineer access to do the same. So let the user decide during sign-up.
What's this crowd think of this idea for solving this problem?
1) Offer an option to opt-out of all automated account recovery. If set, no more email resets, support PINs, or similar. This would be targeted at people truly care about security and have no issue with "forgetting passwords" (i.e. you use a password manager and you're not an idiot about backups).
2) Offer in-person, manual recovery. To participate in this you'd need to pre-register with full contact details (name/address/etc) of the valid people who could use this feature. The person would have to physically come to the office of the company, present two (or more) forms of identification. To add further security, you could add a mandatory wait period between initiating a reset and it taking effect (ex: min 7 days). That way a combination of fake ids and social engineering could (in theory) be stopped by getting an alert that "You initiated a manual reset of your XYZ account. Did you actually do this??"
EDIT: For #2 you could also add a non-trivial fee (say $500) that would need to be charged and cleared in advance of the person showing up.
We do something similar at Silent Circle. In your recovery options, there's a page with a high-entropy secret key and a QR code that you can print out to use if you ever forget your password.
There's also a checkbox that says "don't ever recover this account" (i.e. the "I have a password database on Dropbox") checkbox. Checking that box actually disables password resets on the admin interface, so your account is pretty much dead if you lose the password.
OT question about Silent Circle: I was just looking at your website, and I noticed that you cannot ship to PO Boxes. Is this a security feature (eg, no government knowledge of the recipient) or a logistics issue (eg, FedEx/UPS can't deliver to PO Boxes). I would imagine it is pretty hard to get service for your SIM card without revealing your identity to degree.
I'm not actually sure about that (I assume you're referring to shipping a Blackphone?). The Blackphones are a semi-separate division that I don't have much contact with, unfortunately.
weaker form of #2 would be a "send a registered letter" reset mode, in various forms (e.g. here in Germany there is a type of letter where you have to go to the post office and show matching ID to send it). Or require a (possibly named) notary to validate the request, or something along those lines.
If you announce sending the reset letter there is also time for the account owner to prevent the reset unless an attacker manages to isolate them from all notification channels.
This is more or less how NearlyFreeSpeech works. You can set what level of identity proof you need to reset your password, or disable reset entirely. They seem pretty serious about it!
Why have a procedure if your support doesn't follow it? Even if you have a procedure, everything falls apart when it isn't followed. This is the same as having no procedure at all.
People make mistakes. The customer support person was probably just trying to be helpful and not fully aware of all the ramifications. This is unfortunate but presumably there has been some retraining.
Note: I have no direct or indirect relationship with Namecheap at all.
Wouldn't it make sense that support staff can only generate and send out password reset mails if the PIN/password has been entered into a form? I don't know the term for this - like "coded procedure".
In this case, the support staff wouldn't even needed to be trusted in the first case.
This is a great point. The software should be modified to not allow the employee to even make any modifications to the account without the correct credentials.
I love namecheap but 5 sounds like victim blaming. Come on.
EDIT: My use of the term is a bit strong. I feel frustrated that company execs cannot explicitly admit a mistake or apologize. I should have worded it differently.
EDIT2: just for Tamar. By explicit I mean literally using the words "sorry", "apologize", or "mistake". What we have is the standard corporate nonapology.
EDIT3: congrats to Tamar for being promoted to a Namecheap executive!
Imagine you just lost two servers you can't replace, or you're a potential customer reading this thread, and are afraid of the same.
This is what they read as the company's response to this loss:
"Anyone with any self-managed server with ANY provider should always keep their own multiple backups. Dumbass."
Note the change I made at the end to reflect how some people [who are empathizing with someone who was attacked and lost their property] will interpret that statement. Did any of that statement help the situation at all? Did it help customers feel better? Or did it have the opposite effect? Would this be considered a good way to engender goodwill for your brand?
Now consider this reinterpretation of the statement:
"With self-managed servers, it is good best practice to keep multiple backups for yourself, no matter who your service provider is."
Agreed. However, is this messaged anywhere in your documentation or setup instructions? Do you provide instructions how how to set this up with a 3rd party or list of 3rd parties?
Although backups are #1 item on any list of best practices, making an easy, and tested, implementation method would be a good practice on your part.
I am not sure why you would respond to an accusation about victim blaming by reiterating the exact thing that caused the accusation. You might want to reconsider continuing this particular aspect of discussion for PR reasons. It's not an argument you're going to win.
Unless you sign up for a managed service that claims to include backups or whatever, you are responsible for your own backups. What's controversial about that?
The issue is that Namecheap was the one that fucked up here, and now is not the time to emphasize "you should really be prepared for us fucking up in this manner". It's victim blaming. It looks shitty. The argument I refer to isn't "you should have offsite backups". The argument is that Namecheap is implicitly victim blaming, and they're not going to convince many people that they aren't.
Eh.. I don't really agree that this is victim blaming. But then again, I find that I disagree with most uses of the phrase "victim blaming". Pointing out that somebody did something sub-optimal, while still acknowledging the mis-deeds, mistakes, etc. of other parties, is not "victim blaming" in my book. It's just pointing out the truth.
I mean, if you go for a stroll through the roughest neighborhood in town, unarmed, by yourself, at night, and you get mugged, is it wrong to point out that going for that walk was stupid? Saying so doesn't mean the the mugger isn't guilty or that what happened is right in any sense. It's just acknowledging reality.
>I mean, if you go for a stroll through the roughest neighborhood in town, unarmed, by yourself, at night, and you get mugged, is it wrong to point out that going for that walk was stupid?
Yes, this is the textbook example of victim blaming. Placing any amount of blame on the person who is the victim in this situation is saying that they don't have the right to walk down a street and not be mugged. I am admittedly not the best at describing this because up until recently I had the same thought process as you. I would encourage you to find better explanations than what I can offer and be willing to have your beliefs challenged.
Yes, this is the textbook example of victim blaming.
Then "victim blaming" is a meaningless concept and we should quit using it. Because if I choose to do something stupid, I do bear some responsibility for the outcome, even if somebody else violates my rights. That doesn't absolve the other party of course, which is my point. That is, you can blame the perpetrator of a crime while also pointing out that the victim could (possibly should) have done things differently.
Placing any amount of blame on the person who is the victim in this situation is saying that they don't have the right to walk down a street and not be mugged.
It isn't "blame" for the actions of the other person. Why wouldn't you point out the stupidity of knowingly putting yourself in a dangerous situation?
Victim blaming is used to vindicate a perpetrator of a wrong doing. That isn't being done here, nor in the mugging example.
You can say a victim is stupid without giving any vindication to the person in the wrong.
Namecheap are saying "be responsible for your backups, but yeah we screwed up on our security policy" - They are seperate things, that the victim here has conflated, but are seperate problems (in regards to namecheaps offering).
I don't know where any of you live, but saying recklessness is "victim blaming" sounds like a first world privilege. Yes, in generally in the first world, screaming for your rights can actually work.
In other worlds however, the problem is usually too widespread. You might get a lot of attention, comiseration, etc. but in the end, being reckless goes against survival. People who point this out should not be shushed for pointing out what you need to do to survive.
Its amazing to see that this "victim blaming" mentality is growing in Brazil. Violence here is out of control. You might get mugged/shot/kidnapped for no reason, or not displaying any wealth. Having been kidnapped myself, and chatted with the kidnappers, they do look for signs of wealth before pouncing. Therefore, yes, the victim does has an ounce of control over their risk and it's not wrong to point that out.
It does not solve violence, and attackers will just look for other victims regardless of their reward estimate. However, would you tell your children not to not show affluence/vulnerability in shady places just because you don't want to "victim blame"?
It's not victim blaming. It's simply a reiteration that it helps to have this in place if you are specifically opting to rent/lease a server that does not offer it.
I don't believe you properly understand what victim blaming is or the argument I am making here, hence the reason I recommended you and your CIO don't bother continuing trying to discuss this. You're giving people reason to dislike Namecheap for no gain to yourself and your brand.
"Better be safe than sorry" - namecheap for when you lose your stuff on their services.
I don't think the best way to respond to a public vent is "Here's what you should have done instead". Responses might be technically correct but they lack empathy for the customer.
That comment I made refers to data integrity across platforms. You should be smart about data, no matter where it arises, if it is important to you.
For example, let me give you a look at what my Windows hard drive looks like.
My important files are stored locally, on Dropbox, and on CrashPlan. Some is also on Google Drive. I also run an offsite backup of my own to another local Linux box.
Don't make this specific to Namecheap, @kelukelugames. It's always smart to have good recovery systems in place. If you care that much about your data, you will protect it at whatever cost.
So yeah, I repeat, better to be safe than sorry. Your mileage may vary.
Low end hosting doesn't generally have backups, because it's well, cheap. Extra overheads make the price increase, then you're not cheap and can't compete at that end.
Usually there are backup options included in the plan for upsell possibilities with these kinds of providers. Really, you should not expect a service that has 'cheap' in the name to offer any kind of backup.
Furthermore, is Namecheap authorized to copy their clients' data by their terms of service? If not, automatic backups may bypass totally-reasonable expectations that other users have. Backups can potentially be a threat vector, for example. There might be many reasons why one of Namecheap's clients might say "you copied this data?! and now I have no control of the environment the backup lives in?!"...
An example would be if some service stored credit card information temporarily while waiting for transactions etc. to process but then purged each record after two weeks later. A compromise of the backups containing, say, weekly snapshots could then contain 90% of a client's ever-stored financial information whereas a compromise of the main site might only reveal a couple percent of them.
That's true, although we should be fairly concerned about a company using very cheap hosting on VPS with no form of encryption storing anything sensitive. You may also be breaching PCI DSS (fwiw) doing that.
In reality though, more companies do this than should be allowed. I worked for an ISP in a previous life and even on the super cheap shared hosting there were companies that were making a decent turnover and then using the cheapest possible hosting for their email/site. The quantity of these companies was a significant number too.
Especially when they were kicking off on the phone due to inevitable maintenance/downtime. Trying to appease customers that turn over 10 million a year and pay £5 a month for hosting is a bit wtf. You pay for what you get... that's no different in hosting.
> Really, you should not expect a service that has 'cheap' in the name to offer any kind of backup.
They never spell this out for you though, usually they imply that their service is just as good as their pricier rivals. As a result, many people get burnt before they get savvy. Some never get savvy, they just get turned off to the industry.
Not sure what the alternative is. I suspect a company that did clearly spell out their pros and cons would risk having stunted growth or go out of business entirely.
Re: Your edit - just a note, it is very clear here that in points 1-4, Namecheap has acknowledged a mistake. That's exactly why there was a lot of training (and retraining) internally to ensure this mistake does not recur. But we do acknowledge it is an isolated incident. That doesn't mean it's not less important - we're fully aware of what happened here and it will not recur.
Let's not throw personal attacks at me (and the tongue-in-cheek "congrats for being promoted to executive!" comment). There's plenty of remorse and there's plenty of acknowledgment of mistakes here. That said, as we acknowledged elsewhere, we're responding to the matter across several different platforms and specifically say we're rushed in trying to get out some basic insights behind what happened and transpired. A more well crafted blog response for all to see (this time from the CEO) has been published to https://blog.namecheap.com/social-engineering-issue/
It's a common practice. I don't see how it is personally offensive to you. My description was for the CIO and execs in general, yet you insisted they were for you. So maybe you should stop making tongue-in-cheek comments. In fact, I am moving my domains off of Namecheap because I don't think Namecheap is very good at handling customer relations, particularly on social media.
(the reply link didn't show up before, so I don't know if you saw my response posted right after this)
To be fair, your third edit was only for me. And that was the only comment I was replying to.
As I said, we are working to respond to hundreds of comments across dozens of platforms. I certainly respect your distaste in the more rushed responses in order to address all of the deluge, and that is why we were also simultaneously working on a longer and more thoughtful response that speaks for all of us at the company via that blog post (in a far more emotional tone).
It certainly is difficult to envision the challenges of responding to dozens of responses if you're not in our shoes. But I genuinely thank you for the feedback - and we're noting this (as well as the feedback all have been sent to date; a lot of that was factored into policy adjustment and our blog response) for handling it differently next time.
I guess take Matt up on his offer where he said this: "Also let me reiterate this is an isolated event. We handle over 10,000 chat sessions every day without a glitch. I invite people to use our live chat service and see what is and what is not possible, as well as the security precautions we have in place."
With #3 - ideally your systems should not allow you to break established procedure. Mitigate the risk by not giving the support staff tools to shoot yourself in the foot so easily. This could be achieved with peer verification or some other mechanism (lots of ways if you think it through).
> 4. With thissaid, we've used this as a learning example and additional training has been provided to the individual involved
This is not the correct solution. What's to prevent the next new person from making the same mistake?
If it shouldn't happen, don't make it possible to happen. Put in place a technical solution that doesn't allow it happen. And if there is some special case where it still needs to be possible, make it that it needs a secondary signoff from a senior team member.
That's part of the whole issue. It wasn't simply an issue of retraining. The entire company is well aware of this issue and is using it to improve, not simply to reprimand a single person.
This is the kind of 'learning experience' that becomes part of corporate culture and future training. When someone says 'Why bother with all this?' the response can now be 'Read this writeup of how ONE person NOT doing this correctly cost the company a ton of marketing $$$ and STILL left us with a black eye with our more technically-savvy customers.'
And how many people at Namecheap do you think aren't aware of this by now?
But yes, technical solutions should go in but those take longer to implement. Among other things, it seems to me that re-prompting for the account password might be a good idea before any VPS reinstall/reinitialization that's going to wipe an existing VPS (not that it would've helped much here).
My hobby: role-playing how I would respond as the CEO if my company was getting skewered on HN. Here is my version!
---
Disclaimer: I'm [not] CIO @ Namecheap
We messed up, big time. While we handle 1000s of live chat sessions everyday without issue, I realize that even one breakdown in security protocol can cause huge problems and a loss of trust for our customers.
In response to this isolated case (in which our established procedure was not followed), we will be creating additional training material for all our live support staff. Additionally, we will be exploring technical solutions to try to make this kind of breakdown much harder. Mistakes happen, but if we can prevent them, it is worth doing.
We also would like to take this opportunity to remind folks that any self-managed server (regardless of provider) should always be backed up in multiple places. For information on how to do this with Namecheap, we've published a guide here: <link>
I've reached out to author of the post already by email and we are working to help them resolve any outstanding issues.
^^ we'll take it. We've been responding for the last 1+ hour to things in real time across several social networks, so we're a little rushed. But thanks for the role play :)
For a "what not to do", have a look how (the CEO of?) FTDI responded after they were caught intentionally "bricking" chips that were detected as counterfeit by the Windows drivers.
Go out right now and get the book "Crucial Conversations". It is BY FAR the best book I've ever read on this kind of thing. It is simultaneously the best relationship book I've ever read and the best business book I've ever read. It goes through the basic principles for handling these situations in an easy to understand way.
We have apologized, admitted mistakes, and made tremendous internal change to move on for the better. We would not do that or even post here if we didn't care.
The offer of a free year of hosting is nonetheless a paltry joke. The high road here is to acknowledge that customer may choose never to host with you again and still go above and beyond in attempting to make it right to them, e.g. by offering a full refund for the last year of hosting they'd paid for or the like.
Tell me one registrar where you can be sure that this will not happen and I will move my domains today. I wouldn't even care if I have to pay 100$ per year for a domain.
Putting #1 as #1 looks like bitter deflection. You do it elsewhere in the thread too, saying that lack of 2fa on the email account opened the door to this. You should be well aware both that most security issues end up being perfect storm of circumstances, and that attackers can and will target multiple points in the chain. Relying on #1 as the spearhead of your apparent defense here is tantamount to admitting that you are relying on the security of people's email accounts as part of your own security process, which is wild.
You also didn't mention all the terrible things the OP pointed out that someone can do with just your password even when 2fa is enabled.
However, it's relevant to the story because there's a huge difference between sending a password reset to the email already listed on an account vs. resetting it for any random person who starts a chat.
This doesn't excuse their other issues, but it makes the customer support rep's behavior a bit less awful, even if they still violated protocol.
Regardless, to fix this PR disaster, I suggest you add some strong and perhaps just as importantly modern security features in the future that would regain you good will with HN types (and therefore everyone else).
And although it's not you area, can I just say that Namecheap's website is just way too slow since the redesign? I appreciate that you even did a redesign, but for some reason it's one of the slowest websites around. I don't know if it's because of the large images you use on your pages or what's the problem, but I suggest you fix it. It may be losing you customers. A web services company's site should be snappy.
Not trying to be snarky, but the biggest lesson here seems to be "don't operate without off-host backups". Cheap VPS providers don't typically offer that sort of thing as a standard feature. Even when they do, the backups would be on the same infrastructure, and easily wiped from the same (compromised) console.
You could have just as easily lost all the data in an accidental way, with no malice or 3rd party involved.
That said, I do empathize, and it's disappointing that a major player like namecheap would be so easily socially engineered.
I've reviewed all 16 of your comments on the page and beyond sawing a single person at Namecheap didn't follow policy and blaming the user in question, I don't see anywhere that you've stated there's an issue with controls.
Am I missing something, or is Namecheap saying they didn't do anything wrong?
Example of on HN expecting everyone (newbies and all) to know who you are. This happens with DANG and SAMA comments as well. Back when PG used to comment also happened. Look at their profiles, really no explanation of who they are here:
Why is it so hard to put info in your profile or to put a footnote in your comments for the newbies? Would you have your business act this way? Reply to a person's inquiry and not say who you are and what you do? Of course not.
Was interesting that the victim said they used 2FA for everything they considered important, but not email. I guess their email provider doesn't provide it?
Yeah I'm pretty sure the data loss is not a namecheap specific problem. Speaking from experience I lost an entire website (zero backups including app code - it was a horrible technical debt perfect storm situation no VCS etc) in the huge TigerMate inmotionhosting hack back in 2011 so this is nothing new. It was a PITA but the site was nothing special so I wrote it off as a valuable lesson.
I don't see how that can be the biggest lesson. Someone at the service provider bypassed their own protocols in order to hand control of the system over to an unauthorized user. Even with local backups he would have needed to restore the servers because Namecheap royally screwed up.
Yes, you should always have more backups than you need. But wouldn't you be moving to a different provider after something like this anyway?
Did you make it through to the part of the article where he says "my biggest personal lesson is to make off-host backups"? Not sure why you're making this post.
And the second lesson is "avoid weak passwords". He admitted it was cracked because of a weak password.
But Backups should be everyone's first priority. I'm sure many of the ransomware victims would have simply restored their machines from backup--if only they had them!
I think another lesson in security is that 2FA are sometimes completely broken because of real use-case where the user lose his second authenticator device. This is one of the reason I'm really iffy about setting up 2FA on my own accounts.
Also let me reiterate this is an isolated event. We handle over 10,000 chat sessions every day without a glitch. I invite people to use our live chat service and see what is and what is not possible, as well as the security precautions we have in place.
Also let me reiterate this is an isolated event. We handle over 10,000 chat sessions every day without a glitch.
What do you use to tell whether a chat session is a genuine user or someone successfully using a social engineering attack against your chat operatives? If the answer is "nothing" then you can't know if this is an isolated event or how many of your chat sessions go without a glitch.
Parent's point was, how do you tell whether or not your rep was socially engineered? Only some mistakes get complained about. If you don't have such a method then your "10000 sessions a day without a problem" number is fantasy.
Have you considered making this something that can't be done manually?
First of I can social engineer one of your staff. Regardless of how much you train them. I could also bribe your staff or try to get you to hire a plant. Yea that last one is far fetched but just making a point that as long as someone can manually do these things someone will.
I had my 2FA at Singlehop bypassed by social engineering attack. They helpfully changed the entire account contact info without any notice to me, presumably from a phone call. The attacker didn't even have any information to go off other than the IP address. I only found out when I saw the server rebooting into rescue mode and luckily I still had an active management portal cookie (changing the password doesn't log you out of the portal, another big problem) so I immediately knew what was happening.
I wish there was a way to disable the "customer support backdoor" in all these kinds of services. I've started to deploy full disk encryption to all my servers now so if the attacker does manage to get into the management account the server itself is still protected from single user / rescue mode / etc.
Even better, have the option to disable tech support and get an alert if a reset is requested with the metadata related to the party making the request.
Fairly common for enterprise type apps to have a list of preapproved contact points, not on the list they won't even talk to you. Maybe other places could take this up... not foolproof, but at least adds another layer to the challenge.
I've personally gotten past lists like that a number of times simply by stating that person is not on staff any more, I'm their replacement; legitimately did replace the old point of contact.
Getting people to do stuff on the phone is easy a huge amount of the time.
We have had this at Amazon AWS. We had 2FA, one phone call was enough to disable 2FA. The only thing they asked were the last four digits of our credit card.
A reporter at Wired had his digital life destroyed (including personal mac wiped) by social engineering, using little more than info from one account (last 4 digits of credit card from Appstore account) to socially engineer Amazon customer support - http://www.wired.com/2012/08/apple-amazon-mat-honan-hacking/
Edit. Had some details wrong. It was a reporter at Gizmodo, and basically a collection of data was gathered from various online services (mailing address, last 4 digits of CC) to ultimately social engineer his Apple account and remote wipe his computer. A number of factors were involved, but ultimately he was done in by not having 2FA on google, and not backing up his machine.
I mean from a system level perspective. The trouble is that they have policies that are optimized for people shopping on a site with a money back guarantee not for hosting critical infrastructure.
Too bad you can't ask for the CVV code, and run a dummy $1 transaction. Anyone could have the last four of the card number, but the person is much more likely to have the card itself with the CVV.
Disclaimer: I use AWS extensively. Please do this.
If you're using the payment method as auth, you should be locking it out as an auth method for X days after a change has been made, and emailing/SMSing the contact regarding the change.
Yeah, that's the only thing amazon seller support asks me for verification as well. Either last 4 of credit card or last 4 of bank account - which is even stupider - anyone who you wrote a check out to has your bank account number.
Lastpass will let you remove 2FA using the attached email address. I do not believe it requires you to enter codes from the 2FA to disable. It makes me uncomfortable despite my email having two factor.
It's tricky because a lot of customers really DO lock themselves out of a service, and forget their password reset code.
Fun story time. I use to play MTGO, the online Magic the Gathering game. Played it from beta for a few years say 2002-2004. Wanted to check it out in 2014 to see how it changed. Failed password reset online, had to call in to support.
The support guy was like chortle what was your security passcode? I had no idea. He tried giving some hints. I said it has literally been 10 years, I am never going to remember. So he went ahead and rest my password. He told me for the record my security code was "I am the nacho king", but I would be prompted to change it at next login.
So could I have social engineered a 10 year old MTGO account? Yes. But without it, I would have been locked out. I was NOT going to remember some stupid passcode kid me set on a game account.
AT&T has a security code which is "What is your favorite restaurant?" that we set a decade ago when signing up for internet service.
My wife and I have made, I don't know, 10 guesses over the years and have never been able to figure out what our response was back then.
Questions with fact-based answers are much better. But...I once had a site ask me for my best man's first name (Good! This probably won't change over time!). I filled in "Dave" ... and the site gave me an error "Your answer must be at least 6 characters."
Security questions should be treated as secondary password fields, since they are that. Use Diceware for a good tradeoff between entropy and memorability/pronounceability or more complex random passwords and store them in a safe place.
This works well until you get to the "Our site is so secure that we need you to answer three security questions from our canned list, and they can't all be the same string" geniuses. Such an antipattern.
If you have a password manager that successfully tracks the questions, then there's no reason to need to recover the password, as you'll just track the password in the same system.
The catch-22 of these systems is that recovery questions need to be obvious, memorable and unchanging enough to the user that they are useful for recovery, while also being hard for a third party to guess/research. I feel like for the most part those are more often than not mutually exclusive.
I think that the drop-downs are trying to prevent people from mistyping things and locking themselves out because "Accordien" doesn't match "accordion".
I had something along those lines tryin to log in to mojang on a new computer. "We've not seen you log into this pc before (although I had on that IP), please answer these three security questions. Of course I don't remember so I just reset them. I imagine the new answers and the old answers had a lot in common - they were composed primarily of expletives.
Favorite restaurant! What percent of the full business name did you use? Did you capitalize all the letters the same way or in a consistent predictable way? Did you add a word to meet the minimum character/word count? Did you actually have a favorite when you made this? Is your favorite restaurant public information?
This is why I set all my security question answers to a single answer for low security stuff and random pronounceable strings (in case I ever need to read them to a support person) stored in KeePass for high security stuff.
I'm always amazed at how little thought seems to go into these questions.
My wife filled one out a few weeks ago where both the questions and answers were selected from popup menus. One of the questions was "What's your favorite summer activity?" Her answer was, "Swimming." Yeah, that's going to add about one bit of entropy to most people's accounts, you idiots.
Another favorite is "middle name of your youngest child." That answer can change over time!
Best way to handle these are to use a random string for all the answers if you can, and if they let you create your own questions use more random strings; same goes for login names.
What city was my dad born in? xGU,wT&Yvcn6vr?]#,mE of course.
I did that to my payroll account to try and prevent this very issue.
Little did I know that it's one of those services you need the password (I had written that down at the time as it was temporary) AND these questions that are usually used for password resets.
I don't think that will ever get fixed until I change jobs again.
Until someone says "I know my dad was born in Minneapolis, what does it say??" and the customer service representative replies "Huh, it looks like the answer is just gibberish...", "Ah! I must have just mashed on my keyboard when I made the account, sorry about that!!", "No problem, your password is now reset to foobar".
I think all account credentials related things should be handled by special support people, who were trained to understand the situation. Shouldn't be that big percentage of all requests when people need to change/remember password and can't use regular ways.
I can't reply to the sister comment for some reason, so I'll piggyback on the parent.
I always fill these with awkward or absurd questions/anwers that would be amusing if a human operator ever needs to verify them. E.g.
Would you like to go on a date with me?
What color pants am I wearing?
What is the square root of insanity?
Obviously you need to store these in a password database in order to remember them, which kind of defeats the purpose. If I have to choose from predefined questions, it goes along these lines.
I've just started filling them with randomly generated strings that my password manager helpfully creates for me. Though, apparently my bank uses those answers for phone verification also, which makes answering questions like "What's your Significant Other's nickname?" awkward when the answer is "F9-#g7a2<qj"
I imagine that may make make your significant other who was previously friendly, markedly less so, if they see your "friendly value". But that may have been your point, as it may be quite a bit easier to remember. :)
I once tried to perform an internet banking task only to find out I had to call in by phone and enable it first. So I did, and I was asked a few security questions about my data, one of them was: what's the name of your spouse?
I gave the name, was asked to repeat it, so I did, they informed me I was wrong.
I still don't know if they had a name with a typo in the records, a maidem name, or maybe they didn't even have her name (don't remember telling the bank about marital status) and it was some sort of trick question where I was supposed to answer I'm single (even though I wasn't).
Story time: I have been trying for 3 years to figure out what I wanted to hint at with "If it's not this one then it's the other one" as a secret question. I thought I was a clever boy not choosing the usual predetermined "what's your mother's name ?".
It most certainly is but... it doesn't work. At that time I had some kind of semantic combinations for passwords but it doesn't compute for that website.
Comcast's password recovery is pretty weak. I just did it last night. They ask for your zip code and your favorite sports team. If I have a Boston zip code there are likely only 4 options for favorite sports team.
I had a similar situation just recently with an old Gmail account. Despite knowing the password, Gmail wants me to answer the security question or log in from a place I logged in ten years ago or list folder names (which didn't exist the last time I used that account) or ...
The whole point of this misery was to recover my Steam account to play a few games. Fortunately, Steam lets you recover your account if you can provide proof of ownership (like CD keys of physical copies).
I don't want Google to be the safekeeper of my digital identity.
I might have the answer for you- the security question on some of my unimportant shared accounts where a question was required is "what color is my VGA cable?"
The answer to "I don't know my password and I don't know my security question/answer" is, "Sorry, for security reasons we can't help you access this account, you'll need to create a new account." This isn't a problem for banks, why is it a problem for tech companies?
There are at least two such services in Germany WebID[1] and PostID[1] (not to be confused with the older PostIdent that requires identification at a post office).
I haven't used PostID yet but with WebID you basically have a Skype video call where you show them your ID.
I have a few bank accounts with banks that don't have branches. To "verify" your Id they ask you questions from your credit report - which can be problematic. "What was the payment and term on a loan you had 5 years ago?" Fuck if I ever knew what the payment or term was, I didn't care when I took out the loan, I had my own payment schedule (I think if you can't pay back a loan [with the exception of a mortgage] in a year or two you really can't afford the loan...). Some of the stuff I just plain can't remember!
The best was when they asked me which model of car I had owned... and listed two cars that I had owned... I could only select one.
These records can be flat out wrong too. The DMV associates a car with my address that I don't own, for example. I think this happened because the owner never changed their address with the DMV. Or someone could have just fat fingered something which gets populated to other databases with data sharing.
Well you can go to the bank with your ID if everything else fails
Of course that might not be even necessary as there have been reports of people withdrawing money or wiring it somewhere with not even that (but the bank has legal responsibility)
I agree. With banks, if you need to prove identity remotely, you need to get a medallion signature and a notary. Takes time and money. I assume that web businesses would be happy to have to do that in return for robust security not easily broke via social engineering. I think it would even be a competitive advantage. If you want cheap and easy (and insecure) then you can use a competitors offering.
With MTGO above I had maybe $500 in virtual stuff on my account. As the gatekeeper I'm not sure that would go over well.
With servers a similar thing. Say my only copy of a database is on my VPS. May have a business value of $50k. Can't really just say no unconditionally. Need some process to unlock..
> This isn't a problem for banks, why is it a problem for tech companies?
Banks have the option of you physically going into the branch and identifying yourself with relevant legally backed forms of ID. That would not really work for most online companies.
Other methods which involve sending in copies of ID and/or letters signed by appropriate notaries would fail due to human engineering too because your average tech company isn't going to have people sat ready who are capable of accurately verifying this information.
The other problem is PR: due to the lack of another option the average person who is locked out of their account will instantly turn to twitter/facebook/any-where-else-they-can-post and scream as loud as they can that they've been mistreated by company X. Many other average persons will take this at face value without checking the fats and start avoiding company X, or worse bombarding them with communication in support of the inconvenienced user.
Banks don't try that hard. One of my bank is happy to resend me a password by snail mail with an account ID reset by phone.
Also, rechecking the ID of a user can be as simple as asking for a new token payment by the same credit card as used by the account. It's not infailable, the CC can be compromised as well, but it should be way better than what we have now.
> Banks have the option of you physically going into the branch and identifying yourself with relevant legally backed forms of ID.
Not always true! Less than a month ago I needed to login to my Wells Fargo account. Unbeknownst to me, they had been doing some 'upgrades' and there were some glitches. After a frustrating period, I decide I'd just go to the physical branch 1/4 mile from my house and get this fixed!
On site, the bank personnel have access to exactly the same system that I did. (At least they knew there were glitches and sorta how to work around them.) I had two accounts, one for a credit card that I rarely used and my mortgage. Turns out, if you have a credit card then the new system requires a piece of information only found on the physical card - the onsite employees couldn't get around and neither could their call-in tech support!!!!
Point is - for log-in purposes - don't assume going to a physical branch will be any more helpful!
Since I didn't have the credit card with me ('cause rarely used) I canceled the rarely used credit card and was able to login shortly thereafter.
Actually, this could be an interesting and lucrative side business for banks, identity verification. You could have varying levels of verification, requiring varying levels of authenticating documentation and numbers of employees to review and vouch that could then be used to provide a certificate of verification for a service.
E.g. namecheap.com generates verification ticket item requiring valid identification and SSN that Bank of America then uses to verify your identity for $30, and vouched for the identity. Meanwhile Goldman Sachs generates a verification ticket requiring much more strenuous authentication, and the bank charges $200 for (with increased insurance, etc), which satisfies the much higher validation standard the Goldman Sachs requires to authenticate you for your ritrement account with over $X in it, etc.
One option is to look at when the user last logged in. I would be a lot less pissed if an account that I've never touched in 10 years got compromised... I'm probably going to remember my info for recent accounts and want it to be difficult to social engineering those
This is more about preventing the social engineering attacks. The example you're replying to is where the actual user logged in 20 minutes ago, while the attacker is trying to claim to customer service that they forgot the password. If customer service were looking at login attempts, they would see that it doesn't make sense for the user to not know their password, when clearly they provided it to the site just 20 minutes ago.
People using a password manager might not ever know their password. Funny things happen with password managers where history is missing, changes don't save, keystrokes break things. We can't penalize users who use them.
That's why it should only raise a flag rather than totally stop. Perhaps the customer service rep can ask a few more questions.
It's a similar situation to someone who only ever uses their credit card to buy small amounts from their local supermarket. Then suddenly they use it to buy a flight in another country. It might be legit, but it's often not, and should suggest that customer service need to do more investigation before approving.
> I would be a lot less pissed if an account that I've never touched in 10 years got compromised...
You don't need to log into your VPS provider's account or domain name provider's account very often, compared to how often you use the machine or domain. But you don't want those getting reset more easily just because you haven't logged into them in a while.
I had a similar thing happen with my Battle.net account. I forgot to transfer over my authenticator backup code when I switched password managers last time. I had to send them a photo of my driver's license next to my face and another one of it next to a physical newspaper with the date on it. This seems like a much better process for recovering accounts that matter.
Some people would be happy to pay for a leaked copy of those photos, for use with any other company that would accept only the "face/license" photo as sufficient proof.
I think having a recent date on the newspaper prevents reuse of the images. I'm not a Photoshop expert, so maybe that's trivial to change. When I look at my support ticket history on the website, the ticket attachments are gone, so hopefully they're shredded after the support person views them.
Very easy? I'd say "possible" at best. And now you've got access to a battle.net account. Took a heck of a lot more work than asking someone for username/pass in a live chat, and what you gained access to is worth a heck of a lot less. Plus Blizzard actually does keep backups and records and will be able to fix the situation for the account owner.
I'd be surprised if ever a Blizzard account was compromised by someone sending in a false picture.
I don't know what battle.net is so I was speaking generally. Since pictures don't have security features on them it wouldn't be too difficult to photocopy your own id, change the name/address, print it out and glue it on a plastic card. You now have an Id that looks good enough for photo verification. It's a lot of work but if the steaks are high then it will be done. There's also the case when these pictures get leaked, there's a lot of people who have scans of my id. Or just photoshop the pic after its taken.
There's services on the darknet where you can buy these fake Id picture/scans as well.
It seems like a general principle that the less important something is, the better the security probably is. Steam, for example, is really paranoid, constantly asking for verification whenever it thinks I'm logging in from a new computer, bugging me nonstop to set up 2FA, e-mailing me with alerts, etc. Meanwhile my bank does straightforward username/password authentication, with the bonus that the password is case insensitive and silently truncated to eight characters.
It's not that Steam is paranoid for no reason. People keep virtual items on their accounts which attackers can sell for real money, potentially yielding hundreds to thousands of dollars from one account, and so attempted Steam account hacking is rampant. Of course, that's nothing compared to the amounts stored in bank accounts...
Not to mention an account with saved billing info could be used to sell "extra copies of games I can gift you for a few bucks" since Steam allows gifting. What, you mean you didn't mean to buy 10 copies of "DARK SOULS III Deluxe Edition + Steam Controller Bundle" for $115 each to gift to all your friends and family?
I think you are a little confused here. There's no way for Blizzard to authenticate those pictures, you can take literally anyones passport and just swap the name on it.
> Took a heck of a lot more work than asking someone for username/pass in a live chat
This might be true in a world without photoshop, but that's not the world we live in.
>I'd be surprised if ever a Blizzard account was compromised by someone sending in a false picture.
In my personal experience, they'll very rarely insist upon receiving those photos. My battle.net account isn't even under a real name and despite that I've had the authenticator added and removed several times.
Very much this. As a long-time Namecheap customer, this thread caught my attention, but my layman's conclusion is that it doesn't really sound like they did anything worse than any other host would have done.
As someone who forgets his passwords on a regular basis, I'm kind of glad that there's no such thing as perfect security...
To be fair, that's a game account. I realize some MMOs can have really real-money valuable characters/items, so this argument can break down, but the security should be different from an MMO and a VPS solution or a bank.
All of my security questions are passwords. I once had someone at a bank ask "Wait, your mother's maiden's name has a number in it?"
"Wait, you actually answer security questions that any of your friends can guess honestly?"
> To be fair, that's a game account. I realize some MMOs can have really real-money valuable characters/items, so this argument can break down
You want to know how bad it can break down? I imagine the worst case scenario, for so many reasons, actually happened and was mtgox.com. It started out as a Magic: The Gathering Online Exchange (from what I understand) before it became the now infamous Bitcoin exchange that was hacked[1] and massive amounts of money was stolen. I don't know for a fact that old accounts before the pivot to Bitcoin still existed and worked, but it's not inconceivable that they would. One hopes they adopted much better security compared to when they were a trading card exchange (if it wasn't already exceptional at that time), but there could very well have been a time when it was gaining traction for financial type services but didn't have good account safeguards.
1: Or whatever. From what I remember that's a story convoluted and with enough conspiracy theories it's worth a movie.
Are you sure? Wikipedia[1] seems to indicate it at least got into a beta, but it also implies that the domain was what was reused for the bitcoin exchange, not the code base. That said, the references on wikipedia point to the Internet Archive, which shows a placeholder page that says it's in beta, but there's little there to indicate there was ever anything working. It also says it was used to advertise another card game later, but that was just a link to another domain.
I guess that's a long-winded way of saying you are probably right.
Then again, bitcoins were worth so little in 2010 that I could imagine the account security on exchanges back then being comparable to hobbiest exchange sites, so the spirit of the comment may be valid even if the specific example falls down. :/
Magic Online simulates paper Magic. You buy packs to get cards, each card is its own individual digital object which can be traded and sold, card sets go out of print or have limited runs, and rare promos are released. It even has its own digital currency, "tickets", which are reasonably easily converted to cash.
Magic Online accounts can be worth tens of thousands of dollars.
And at least back when I played, if your account got compromised and all your cards were liquidated, the response from customer support was pretty much "that's too bad, shouldn't have let your account get compromised". I seriously doubt they ever added 2FA or anything either.
> Magic Online accounts can be worth tens of thousands of dollars.
It might cost 10k+ dollars to make a behemoth account, but I don't think they are worth that much. It's basically fake internet points, I can gain these for free in this very comment.
So could I have social engineered a 10 year old MTGO account? Yes. But without it, I would have been locked out. I was NOT going to remember some stupid passcode kid me set on a game account.
And apparently I am the nacho King.
What are the chances that you'll remember it in the future? You're already two years into remembering it for the next decade after you found it out again.
I would prefer being stuck without being able to login to my server, compared to potentially having customer support allow someone else to access my server.
I recently changed my phone support password at work to "aaah, f*, I'm not sure is it.." after listening to all my previous support calls and realising that was what I answered with 9/10 times. I suspect its only a matter of time before someone else accidentally guesses it. Its only for my regular user account, for my admin accounts I need to get another domain admin to reset the password, there is no process for anyone to exploit, just an audit every month.
I tried to call a few times and unfortunately kept getting disconnected from my VoIP line. I ended up using their live chat and the person I spoke to was very quick in recognizing it as a social engineering attack and worked with me to reset all my information.
Following up I had to reset my passwords a few more times to ensure the attacker's session was properly terminated. They identified the IPs accessing my account but they were all proxy IPs from Europe and Israel so not much could be done.
I still have the support tickets the attacker made, and I feel their support staff should have recognized it as a compromised account (they changed the name to "John Smith", bad spelling / grammar, asking to reset the root password, accidentally "lost" their private key and need password logins and root SSH enabling, etc).
Aside from this incident though I've been pretty happy there, the hardware is competitive, uptime is good and the network is solid which is all I really ask for with a server provider.
I agree on the 'customer support backdoor,' with some caveats. I worked for a small IT Services provider, and we a policy of "don't do dangerous stuff that an unknown party asks for unless a known party verifies it." But sometimes someone gets fired, and there are no known parties, especially with cloud services.
I realize there has to be a way to work around 2FA, for situations like loss of device, terminations, etc, but that should have some known policy way of verifying (say, you must send a notarized or local equivalent letter, or appear in person somehow) identity, especially for a cloud service.
(Totally incidental and you perhaps won't see this, but, r1ch, Conflict Crusher was instrumental in 15 year old me learning how to mod TA, which led me to learn how write BOS scripts, which led me to realize that I liked this programming thing, which led me to my current career/way to support my family. Thanks! :) )
Conflict Crusher, wow that takes me back! Back to that awful Visual Basic UI with random windows metafiles for backgrounds, what was I thinking.. :). Great to hear that I helped you on your path to become a programmer! Modding games inspired me a lot too, especially Quake and TA.
That's crazy that customer service was able to turn off 2 factor!
Godaddy has gotten really good at preventing social engineering stacks like this. I use 2 factor authentication for my account and customer service can't event talk to me till I give them the code. Don't have that? Need to send them my drivers license and other proof to get the account reset.
The problem I think, is that Godaddy only does hosting so has appropriate controls in place. Since Amazon does shopping and hosting and did the former first they have policies designed for the customers that are shopping.
Another vote for gandi.net here. I transferred all my domains to them during the past years and have never been happier.
Security is almost bomb proof with IP restrictions, GPG keys, 2FA and one little checkbox in their settings I like a lot: "This setting allows you to authorize or disable password resets from the login screen."
I think a good practice is to try to social engineer your own account and see if you end up getting it.
The lesson here is you should always keep your own off-site backups - especially if you don't pay for a 'managed' server.
There will always be rare occasions such as this, but considering how many customers Namecheap handle, I don't think we should be seriously concerned. I'm pretty confident lessons will be learned.
It wouldn't happen to you. As you may see by other comments here (written by @matthewdrussell), this was an isolated incident that occurred specifically to an already-compromised email account. Still, we could always do better, and there have already been many meetings and policy improvements that have resulted from this single incident. We always take these opportunities to improve.
Yes, I work for Namecheap, but that's probably implied by my comment.
I use them for domains, but I wouldn't host there. (nor any other registrar, and if Starbucks started selling burgers, I doubt I'd buy one of those either)
So he is using 2FA for all the important accounts but for the most important one (the email which he used to register an account at all these services) he's using a weak pw and no 2FA? Am i missing something here? Yes they did not follow protocol but why would one not use 2FA for such an important email addy?
I thought the same thing! Although he is definitely right to complain about Namecheap, the biggest takeaway is, your email is the most important service you have on the internet:
> I’m pretty careful to use 2FA for any service that I consider important
The email being compromised opened the door to his email being compromised. The door to his Namecheap account being compromised was apparently already wide open.
No, OP didn't have 2FA enabled on their namecheap account. It was namecheap's fault for improper handling of the social engineering attack but OP could have protected themselves by having 2FA
Its been months since I wanted to write a detailed summary, but the notion that "namecheap is hackers best domain registrar" is not valid anymore!
About year ago I noticed DNS changes on many of my there-parked domains. Upon reaching via Chat (no phone support so that angry customers cannot vent off) I was told that they cannot help me cause Im not the owner of the account! Upon full verification even with CC on file and telling them purchase history going back to 2009, I was still denied the access. As it turned out, all hacker needed to know is my public WHOIS info to take over my account!! That was insane! Only continuance of threats from my side that I will plaster it all over the net made them change their mind, which again is a breach of trust - what if I was actually the hacker??
What really made me start moving domains to NameSilo (Im not affiliated) is that upon doing a thorough research, I found many cases where Namecheap gives up on fighting for peoples domain! I seen names like nanotmz where company was building some sort of magnetic devices and TMZ came in and threat to sue Namecheap if they dont shut the domain down. That's where I found similar cases for NameSilo and learnt that they stand their ground and would not give up on your domains, even if are threatened with legal action.
I'm out of Namecheap completely as of last month with last SSL expiring.
Namecheap has had two factor authentication and was the first provider to have it. Knowing public whois would not grant someone access to anyone's account at Namecheap. They'd still need to know your Namecheap username, your password, and your PIN, and if you had 2FA, that would need to be provided as well.
If you have specific information, you are welcome to contact us with full details. Policies and procedures are in place to ensure no one falls victim to social engineering and as you call it, "intimidation."
skj, there's a lot more to policy and procedure than just human intervention. We're fully aware of this valid concern and are committed to security on both human and technical sides.
I remember when NameCheap launched the "security notifications" feature where it would email you whenever there was a login or activity on your account. I noticed that logging in on the mobile site didn't trigger any emails. When asked, they replied that the mobile site was just a beta version.
It doesn't help that the front door is securely locked when the back door is not! :-/
Crazy. I have trouble figuring out how you'd even program it that way. Is it not obvious that security notifications belong in the authentication layer that everything uses, and not in platform-specific front-end code?
I'm leaning more and more towards treating the email address that you use to register for business-critical services as secret.
It's a level of security-through-obscurity, yes, but that doesn't mean it's wrong. It means you can keep that address monitored well. You could make any activity on it send a page, for example.
> but on the way out decided to click the conveniently located “Re-install” button next to each VPS. This instantly wipes everything and installs a new OS. Again this action requires no 2FA authentication or any other form of confirmation
This is the same for DigitalOcean. I'm always amazed that clicking "Rebuild" or "Delete + Scrub Data" doesn't require _any_ confirmation at all.
425 comments
[ 3.0 ms ] story [ 264 ms ] threadSometimes you get what you pay for.
I just wish that their DNS updates were push through faster.
And presumably they wanted you to send those photos to them as an unencrypted email attachment, right?
Edit: Since I'm getting some downvotes I'd really like to know how one could possibly argue that asking for ID scans is better than email resets. You can't really forge the ability to receive email at an address, but you can very easily replace the name on an ID scan.
An attacker can't just pretend to be able to read your email, such ability is too easy to conclusively prove. To be able to read your email they need to hack you somehow.
But for a fake ID the attacker only needs to throw your name in a PSD and they're good to go.
As for fake IDs, yes, it is certainly possible to create them. But when it is so much easier to socially engineer your way into another service like Namecheap, it creates a disincentive for going after Gandi (and other similar hosts).
No security measures can be foolproof, possibly short of sending someone to your home to take a DNS sample, but at least they're trying for a better solution.
That's why nobody has ever used a fake ID at a bar!
> but in germany the fake itself is punishable by law (up 10 ten years).
https://dejure.org/gesetze/StGB/267.html 5 years.
But producing fake scans isn't covered by this law, scans aren't even an official document. In fact, it is illegal for a german company to ask you to send them scans of official documents.
> You'd also need much more information to create a convincing fake id scan of your intended victim
To fake a good enough passport scan you'd need your victims name. That's all the rep is going to have.
I sent the ids by fax (yeah a few years back, I still had a fax machine).
I thought asking for id's + phoning on the number listed in the whois database was a good cross check, especially back then.
Use 2FA on your Namecheap account
Maintain a sensible backup policy
Store your passwords in something secure like KeePass
So what expensive provider do you recommend instead?
What? Why do they do this?
And no, it wasn't related to registry rules or transfers as other users have suggested.
I personally haven't had this happen to me. I've hosted dozens of domains with gandi, under a variety of different TLDs, and can only recommend them.
Cons: Slow website. Bad UX all over the admin/purchasing interface. Feels like they're not doing anything to improve that.
Sure! Gandi received an abuse report regarding someone using one of my domains to scan for open dns resolvers. I informed Gandi that there was nothing I could do about this and expected that to be the end of it. Instead, they suspended my domain and started demanding that I send them ID proof.
As I needed the domain back I sent them a redacted photo of my id card, after which they demanded to see the full id. I decided to terminate my relationship with them.
I feel that this was absolutely unacceptable and likely unlawful behaviour from them as they had absolutely no need for that information. This wasn't a whois dispute.
If it's "Whats your mother's maiden name?" and they let you reset it in the browser, it's a bug.
But if they send you an email (in my case to Gmail, that has 2FA turned on), then it is a feature, because then you'd be required to either 1) intercept the recovery email (and get the password reset URL) or 2) know the format of the password reset URL and just happen to guess mine after brute-forcing every possible link (assuming there is no timeout for the URL or anything else like that).
Have the customer provide an SSH/GPG public key, and store it with the account.
When a password reset is requested, encrypt a random string using said public key, and email it to the email for the account.
An attacker who may have breached your webmail is then reasonably unlikely to also have your private key to decrypt the string.
Follow the link (which didn't necessarily need to be encrypted) and enter the string you decrypted to reset the password.
On a related note: do any/many sites with 2FA, require the 2FA code to do a password reset?
With a 2FA code, you either a) use the same code they use for regular logins, or b) require them to find a way to securely store the (T|H)OTP secret and then add that information to a 2FA app when they want to do a password reset.
I realise the pubkey concept is more than most people would bother with (or even be able to get through on their own), and I think the first 2FA option is definitely better than no extra security at all on password resets, but my thought was about increased security for those who are particularly paranoid/security conscious.
Doesn't this just move the problem from "I forgot my password" to "I lost my private key"?
If you enable 2FA on an account with a service, and subsequently lose access to 2FA otp's (e.g. phone lost/wiped/etc) and lose access to (or never kept) the recovery codes, you generally lose access to the account.
This is similar, but with the benefit that you can keep the key secure by default - i.e. put a passphrase on the key and then store it wherever you like.
In reality, if you fail this "recovery" method, my next suggestion would be a billing based one (i.e. talk to a human, get confirmation of previous invoice details, what is being billed for, how it's paid for, etc)
I also had one of my VPS attacked recently, and I feel for you.
But the name namecheap says "cheap". Maybe they are indeed cheap? I'm not sure the same would have happened with say HE. You pay, but you know what you pay for and get in return.
Personally, I am thinking about moving from a "manually setup" distribution to a "no ssh but deploy", so as to ease reimaging in the future. This way, if a server is compromised, all I have to do it to start the install of a new one.
Any suggestion for tools to do that with Debian distro? (yeah I could write a shell script, but I think there must be better tools out there)
Well, conceivably the second factor could be used to generate crypto key material which is used to decrypt/unlock one's record, so without the second factor even support couldn't read & edit one's record.
Nothing can stop support from deleting & recreating a record though.
If you write apps, package them as Debs. If you need to configure other Debs, make config packages with config-package-dev [1] from the DebAthena project.
Create a metapackage that depends on your software + config packages, and your setup process just needs to be "add private apt repo, apt update, apt install <metapackage>".
[1] https://packages.debian.org/jessie/config-package-dev
I find it hard to believe Microsoft of all companies has outsourced two factor authentication.
(disclosure: obviously I work for Namecheap.)
I do agree with more login forms needing to support 2FA. At this point I wish almost everything did. It is a bit more hassle but is easy to manage for me at least.
The domain registration process isn't automated from my experience but it works well. However, I think the meta is (I think many people will agree) to not mix domain and hosting with the same provider. For example, if you get your domain from namecheap, you should not do hosting at namecheap. Therefore, the argument is that if you want to buy a domain name from Namecheap (as they're pretty decent), you shouldn't do hosting there.
Sorry if I sound like a prick.
Our KB platform is getting some attention as articles are improved and then the UX will be overhauled. We have work to do here and we're doing it.
Just a few weeks ago I was getting a domain set up with Amazon SES, and one Daria P. helped interpret Amazon's docs to get it verified with them, and explained how I was using the dig command incorrectly to inspect the domain's settings. It's rare you see a support person do that at any company.
And, of course, they did ask for a support PIN.
Would anyone here seriously expect that someone in control of their email wouldn't be able to take control of associated accounts?
$CUSTOMER calls in, their nameservers are down and nobody has the account password. Do you think the management at $CUSTOMER is going to accept "hey we need to wait 6 hours to get our site back up because namecheap wont allow us in"?
I don't see any perfect answer here. Ultimately you need a way to recover your account when you've lost all of the "somethings you have" and you've lost your "something you know", but then that allows a social engineer access to do the same. So let the user decide during sign-up.
1) Offer an option to opt-out of all automated account recovery. If set, no more email resets, support PINs, or similar. This would be targeted at people truly care about security and have no issue with "forgetting passwords" (i.e. you use a password manager and you're not an idiot about backups).
2) Offer in-person, manual recovery. To participate in this you'd need to pre-register with full contact details (name/address/etc) of the valid people who could use this feature. The person would have to physically come to the office of the company, present two (or more) forms of identification. To add further security, you could add a mandatory wait period between initiating a reset and it taking effect (ex: min 7 days). That way a combination of fake ids and social engineering could (in theory) be stopped by getting an alert that "You initiated a manual reset of your XYZ account. Did you actually do this??"
EDIT: For #2 you could also add a non-trivial fee (say $500) that would need to be charged and cleared in advance of the person showing up.
There's also a checkbox that says "don't ever recover this account" (i.e. the "I have a password database on Dropbox") checkbox. Checking that box actually disables password resets on the admin interface, so your account is pretty much dead if you lose the password.
If you announce sending the reset letter there is also time for the account owner to prevent the reset unless an attacker manages to isolate them from all notification channels.
1. The credentials were resent to an already compromised email account
2. This is an isolated case
3. Established procedure was not followed
4. With thissaid, we've used this as a learning example and additional training has been provided to the individual involved
5. Anyone with any self-managed server with ANY provider should always keep their own multiple backups
Why have a procedure if your support doesn't follow it? Even if you have a procedure, everything falls apart when it isn't followed. This is the same as having no procedure at all.
Note: I have no direct or indirect relationship with Namecheap at all.
Wouldn't it make sense that support staff can only generate and send out password reset mails if the PIN/password has been entered into a form? I don't know the term for this - like "coded procedure".
In this case, the support staff wouldn't even needed to be trusted in the first case.
EDIT: My use of the term is a bit strong. I feel frustrated that company execs cannot explicitly admit a mistake or apologize. I should have worded it differently.
EDIT2: just for Tamar. By explicit I mean literally using the words "sorry", "apologize", or "mistake". What we have is the standard corporate nonapology.
EDIT3: congrats to Tamar for being promoted to a Namecheap executive!
The opposite of what I'm suggesting is that people - individuals/companies - do not look after their own backups. That's a dangerous precedent.
This is what they read as the company's response to this loss:
"Anyone with any self-managed server with ANY provider should always keep their own multiple backups. Dumbass."
Note the change I made at the end to reflect how some people [who are empathizing with someone who was attacked and lost their property] will interpret that statement. Did any of that statement help the situation at all? Did it help customers feel better? Or did it have the opposite effect? Would this be considered a good way to engender goodwill for your brand?
Now consider this reinterpretation of the statement:
"With self-managed servers, it is good best practice to keep multiple backups for yourself, no matter who your service provider is."
Although backups are #1 item on any list of best practices, making an easy, and tested, implementation method would be a good practice on your part.
Unless you sign up for a managed service that claims to include backups or whatever, you are responsible for your own backups. What's controversial about that?
I mean, if you go for a stroll through the roughest neighborhood in town, unarmed, by yourself, at night, and you get mugged, is it wrong to point out that going for that walk was stupid? Saying so doesn't mean the the mugger isn't guilty or that what happened is right in any sense. It's just acknowledging reality.
Yes, this is the textbook example of victim blaming. Placing any amount of blame on the person who is the victim in this situation is saying that they don't have the right to walk down a street and not be mugged. I am admittedly not the best at describing this because up until recently I had the same thought process as you. I would encourage you to find better explanations than what I can offer and be willing to have your beliefs challenged.
Then "victim blaming" is a meaningless concept and we should quit using it. Because if I choose to do something stupid, I do bear some responsibility for the outcome, even if somebody else violates my rights. That doesn't absolve the other party of course, which is my point. That is, you can blame the perpetrator of a crime while also pointing out that the victim could (possibly should) have done things differently.
Placing any amount of blame on the person who is the victim in this situation is saying that they don't have the right to walk down a street and not be mugged.
It isn't "blame" for the actions of the other person. Why wouldn't you point out the stupidity of knowingly putting yourself in a dangerous situation?
You can say a victim is stupid without giving any vindication to the person in the wrong.
Namecheap are saying "be responsible for your backups, but yeah we screwed up on our security policy" - They are seperate things, that the victim here has conflated, but are seperate problems (in regards to namecheaps offering).
In other worlds however, the problem is usually too widespread. You might get a lot of attention, comiseration, etc. but in the end, being reckless goes against survival. People who point this out should not be shushed for pointing out what you need to do to survive.
Its amazing to see that this "victim blaming" mentality is growing in Brazil. Violence here is out of control. You might get mugged/shot/kidnapped for no reason, or not displaying any wealth. Having been kidnapped myself, and chatted with the kidnappers, they do look for signs of wealth before pouncing. Therefore, yes, the victim does has an ounce of control over their risk and it's not wrong to point that out.
It does not solve violence, and attackers will just look for other victims regardless of their reward estimate. However, would you tell your children not to not show affluence/vulnerability in shady places just because you don't want to "victim blame"?
Also, it's stated in the knowledgebase that it is advisable to set up server backups of your own if you do not have a managed server: https://www.namecheap.com/support/knowledgebase/article.aspx...
Even my managed services have offsite backups. Better be safe than sorry, I always say.
I don't think the best way to respond to a public vent is "Here's what you should have done instead". Responses might be technically correct but they lack empathy for the customer.
For example, let me give you a look at what my Windows hard drive looks like.
My important files are stored locally, on Dropbox, and on CrashPlan. Some is also on Google Drive. I also run an offsite backup of my own to another local Linux box.
Don't make this specific to Namecheap, @kelukelugames. It's always smart to have good recovery systems in place. If you care that much about your data, you will protect it at whatever cost.
So yeah, I repeat, better to be safe than sorry. Your mileage may vary.
Usually there are backup options included in the plan for upsell possibilities with these kinds of providers. Really, you should not expect a service that has 'cheap' in the name to offer any kind of backup.
An example would be if some service stored credit card information temporarily while waiting for transactions etc. to process but then purged each record after two weeks later. A compromise of the backups containing, say, weekly snapshots could then contain 90% of a client's ever-stored financial information whereas a compromise of the main site might only reveal a couple percent of them.
In reality though, more companies do this than should be allowed. I worked for an ISP in a previous life and even on the super cheap shared hosting there were companies that were making a decent turnover and then using the cheapest possible hosting for their email/site. The quantity of these companies was a significant number too.
Especially when they were kicking off on the phone due to inevitable maintenance/downtime. Trying to appease customers that turn over 10 million a year and pay £5 a month for hosting is a bit wtf. You pay for what you get... that's no different in hosting.
They never spell this out for you though, usually they imply that their service is just as good as their pricier rivals. As a result, many people get burnt before they get savvy. Some never get savvy, they just get turned off to the industry.
Not sure what the alternative is. I suspect a company that did clearly spell out their pros and cons would risk having stunted growth or go out of business entirely.
"If it's too good to be true, it probably is"
and
"Cheap, Good, Fast - pick two"
To be fair, your third edit was only for me. And that was the only comment I was replying to.
As I said, we are working to respond to hundreds of comments across dozens of platforms. I certainly respect your distaste in the more rushed responses in order to address all of the deluge, and that is why we were also simultaneously working on a longer and more thoughtful response that speaks for all of us at the company via that blog post (in a far more emotional tone).
It certainly is difficult to envision the challenges of responding to dozens of responses if you're not in our shoes. But I genuinely thank you for the feedback - and we're noting this (as well as the feedback all have been sent to date; a lot of that was factored into policy adjustment and our blog response) for handling it differently next time.
This is not the correct solution. What's to prevent the next new person from making the same mistake?
If it shouldn't happen, don't make it possible to happen. Put in place a technical solution that doesn't allow it happen. And if there is some special case where it still needs to be possible, make it that it needs a secondary signoff from a senior team member.
People will always be fallible.
And how many people at Namecheap do you think aren't aware of this by now?
But yes, technical solutions should go in but those take longer to implement. Among other things, it seems to me that re-prompting for the account password might be a good idea before any VPS reinstall/reinitialization that's going to wipe an existing VPS (not that it would've helped much here).
---
Disclaimer: I'm [not] CIO @ Namecheap
We messed up, big time. While we handle 1000s of live chat sessions everyday without issue, I realize that even one breakdown in security protocol can cause huge problems and a loss of trust for our customers.
In response to this isolated case (in which our established procedure was not followed), we will be creating additional training material for all our live support staff. Additionally, we will be exploring technical solutions to try to make this kind of breakdown much harder. Mistakes happen, but if we can prevent them, it is worth doing.
We also would like to take this opportunity to remind folks that any self-managed server (regardless of provider) should always be backed up in multiple places. For information on how to do this with Namecheap, we've published a guide here: <link>
I've reached out to author of the post already by email and we are working to help them resolve any outstanding issues.
* Actually apologize in a human way
* Show empathy by identifying the impact of what happened to customers (not your impact internally)
* State action items that you've created, even if they are just in 'evaluation' state
* Indicate that the specific incident in question is being handled outside of this forum
* Take responsibility for things even if you shouldn't "have to"
http://blog.statuspage.io/why-public-apologies-suck
Some important bits:
>4 PARTS OF A BAD APOLOGY
- Justifying the offending actions or words.
- Blaming the victim.
- Making excuses.
- Minimizing the consequences.
>8 PARTS OF AN EFFECTIVE APOLOGY
- You actually have to use the words I’m sorry.
- Acknowledge that you messed up. (As in, “I take full responsibility for my words.”)
- Tell the person how you’ll fix the situation.
- Describe what happened, but without foisting the blame off on someone else.
- Promise to behave better next time.
- Make sure the person knows you know exactly how you hurt or inconvenienced them.
- Much like the first rule, it’s important to use some version of the phrase “I was wrong.”
- Ask for forgiveness.
Overblown/fake apology is not very informative - it's hard to say what exactly can you trust in it.
It sounds like you are confirming that this incident did happen and it was your fault for not following your procedures.
I am not a lawyer, but since there was signification loss, it would probably be in your best interest to offer better reparations.
OpenDomain has several domains that are on NameCheap - I will transfer them immediately since it appears you do not care about customers.
We have apologized, admitted mistakes, and made tremendous internal change to move on for the better. We would not do that or even post here if we didn't care.
You also didn't mention all the terrible things the OP pointed out that someone can do with just your password even when 2fa is enabled.
However, it's relevant to the story because there's a huge difference between sending a password reset to the email already listed on an account vs. resetting it for any random person who starts a chat.
This doesn't excuse their other issues, but it makes the customer support rep's behavior a bit less awful, even if they still violated protocol.
And although it's not you area, can I just say that Namecheap's website is just way too slow since the redesign? I appreciate that you even did a redesign, but for some reason it's one of the slowest websites around. I don't know if it's because of the large images you use on your pages or what's the problem, but I suggest you fix it. It may be losing you customers. A web services company's site should be snappy.
You could have just as easily lost all the data in an accidental way, with no malice or 3rd party involved.
That said, I do empathize, and it's disappointing that a major player like namecheap would be so easily socially engineered.
Self-managed a customer is responsible for their own backups. Just like with DO and that full server loss a couple of months back.
Am I missing something, or is Namecheap saying they didn't do anything wrong?
https://news.ycombinator.com/user?id=pg
https://news.ycombinator.com/user?id=dang
https://news.ycombinator.com/user?id=sama
Why is it so hard to put info in your profile or to put a footnote in your comments for the newbies? Would you have your business act this way? Reply to a person's inquiry and not say who you are and what you do? Of course not.
"we offer service X"
could be from anybody trying to say they offer the service, not obvious at all it is a representative of Namecheap
> Disclaimer: I'm CIO @ Namecheap
https://news.ycombinator.com/item?id=11479810
It's a $30 to $75 a month upcharge on top of the base VPS price. I don't think the OP was paying for a managed VPS.
Edit: Apparently, only the $75/month package includes backups. The $30/month package does not. (http://i.imgur.com/Iy7iacH.png)
Are those backups purgeable from the account control panel?
Honestly I'm not a huge fan of same-provider backup solutions. It seems like asking a fox to guard your sheep.
I think a better suggestion/wording (depending on your intent) re: backups would be "don't operate without independent backups".
If you give a shit about it, back it up.
Yes, you should always have more backups than you need. But wouldn't you be moving to a different provider after something like this anyway?
But Backups should be everyone's first priority. I'm sure many of the ransomware victims would have simply restored their machines from backup--if only they had them!
Is it? Does that mean that my ability to reset your users solusvm passwords with or without 2fa constitutes as a 1337 0day?
Hey BTW, remember that time you got hacked through your support site and didn't tell anyone?
What do you use to tell whether a chat session is a genuine user or someone successfully using a social engineering attack against your chat operatives? If the answer is "nothing" then you can't know if this is an isolated event or how many of your chat sessions go without a glitch.
First of I can social engineer one of your staff. Regardless of how much you train them. I could also bribe your staff or try to get you to hire a plant. Yea that last one is far fetched but just making a point that as long as someone can manually do these things someone will.
2. not deleting the mail with the login information
3. not having a backup
Yeah. This was just as much your fault.
I wish there was a way to disable the "customer support backdoor" in all these kinds of services. I've started to deploy full disk encryption to all my servers now so if the attacker does manage to get into the management account the server itself is still protected from single user / rescue mode / etc.
Getting people to do stuff on the phone is easy a huge amount of the time.
Edit. Had some details wrong. It was a reporter at Gizmodo, and basically a collection of data was gathered from various online services (mailing address, last 4 digits of CC) to ultimately social engineer his Apple account and remote wipe his computer. A number of factors were involved, but ultimately he was done in by not having 2FA on google, and not backing up his machine.
Disclaimer: I use AWS extensively. Please do this.
Call 2: "I just locked myself out of the account, can you reset it for me."
Fun story time. I use to play MTGO, the online Magic the Gathering game. Played it from beta for a few years say 2002-2004. Wanted to check it out in 2014 to see how it changed. Failed password reset online, had to call in to support.
The support guy was like chortle what was your security passcode? I had no idea. He tried giving some hints. I said it has literally been 10 years, I am never going to remember. So he went ahead and rest my password. He told me for the record my security code was "I am the nacho king", but I would be prompted to change it at next login.
So could I have social engineered a 10 year old MTGO account? Yes. But without it, I would have been locked out. I was NOT going to remember some stupid passcode kid me set on a game account.
And apparently I am the nacho King.
My wife and I have made, I don't know, 10 guesses over the years and have never been able to figure out what our response was back then.
Questions with fact-based answers are much better. But...I once had a site ask me for my best man's first name (Good! This probably won't change over time!). I filled in "Dave" ... and the site gave me an error "Your answer must be at least 6 characters."
Doh!
Q: what was your childhood best friend's last name? A: pathway-titian-slowly-quiver-kodiak-hue
etc., even for fact-based things like "what city were you born in?" or "what street did you live on in 1995?".
Drop-downs for answers. Just got this on United.com:
http://imgur.com/84l0CdU
The catch-22 of these systems is that recovery questions need to be obvious, memorable and unchanging enough to the user that they are useful for recovery, while also being hard for a third party to guess/research. I feel like for the most part those are more often than not mutually exclusive.
My wife filled one out a few weeks ago where both the questions and answers were selected from popup menus. One of the questions was "What's your favorite summer activity?" Her answer was, "Swimming." Yeah, that's going to add about one bit of entropy to most people's accounts, you idiots.
Another favorite is "middle name of your youngest child." That answer can change over time!
What city was my dad born in? xGU,wT&Yvcn6vr?]#,mE of course.
Little did I know that it's one of those services you need the password (I had written that down at the time as it was temporary) AND these questions that are usually used for password resets.
I don't think that will ever get fixed until I change jobs again.
I always fill these with awkward or absurd questions/anwers that would be amusing if a human operator ever needs to verify them. E.g.
Would you like to go on a date with me?
What color pants am I wearing?
What is the square root of insanity?
Obviously you need to store these in a password database in order to remember them, which kind of defeats the purpose. If I have to choose from predefined questions, it goes along these lines.
Q: What was your mothers maiden name?
A: Why, are you stalking her?
Q: Where were you born?
A: Oh I can't remember, it's been so long!
#$%&+@ is going to be hard to type or speak, just say their nickname is "the frozen one", same entropy, easier to handle
> Use a random but user friendly value
> "the frozen one"
I imagine that may make make your significant other who was previously friendly, markedly less so, if they see your "friendly value". But that may have been your point, as it may be quite a bit easier to remember. :)
But yeah, what you said might happen.
Error: Security question answer cannot be more than 10 characters long.
Error: Security question answer must be one word.
Error: Security question answer must be unique.
Error: Security question error. Please try again.
United Airlines does this.
Their reason is incredibly depressing.
https://mobile.twitter.com/evacide/status/711853927134842880
Infosec is an unsolved problem in so many ways, especially for the majority of people that are nontechnical.
I gave the name, was asked to repeat it, so I did, they informed me I was wrong.
I still don't know if they had a name with a typo in the records, a maidem name, or maybe they didn't even have her name (don't remember telling the bank about marital status) and it was some sort of trick question where I was supposed to answer I'm single (even though I wasn't).
The whole point of this misery was to recover my Steam account to play a few games. Fortunately, Steam lets you recover your account if you can provide proof of ownership (like CD keys of physical copies).
I don't want Google to be the safekeeper of my digital identity.
It's harder online when you don't have the same ability to interact face to face.
Most tech companies (especially startups) don't.
I haven't used PostID yet but with WebID you basically have a Skype video call where you show them your ID.
[1]: https://www.webid-solutions.de/en/ [2]: https://www.deutschepost.de/de/p/postid.html
So KYC as a service is certainly a thing, and I know of two: Tracesmart & Onfido.
The best was when they asked me which model of car I had owned... and listed two cars that I had owned... I could only select one.
These records can be flat out wrong too. The DMV associates a car with my address that I don't own, for example. I think this happened because the owner never changed their address with the DMV. Or someone could have just fat fingered something which gets populated to other databases with data sharing.
Of course that might not be even necessary as there have been reports of people withdrawing money or wiring it somewhere with not even that (but the bank has legal responsibility)
With servers a similar thing. Say my only copy of a database is on my VPS. May have a business value of $50k. Can't really just say no unconditionally. Need some process to unlock..
Banks have the option of you physically going into the branch and identifying yourself with relevant legally backed forms of ID. That would not really work for most online companies.
Other methods which involve sending in copies of ID and/or letters signed by appropriate notaries would fail due to human engineering too because your average tech company isn't going to have people sat ready who are capable of accurately verifying this information.
The other problem is PR: due to the lack of another option the average person who is locked out of their account will instantly turn to twitter/facebook/any-where-else-they-can-post and scream as loud as they can that they've been mistreated by company X. Many other average persons will take this at face value without checking the fats and start avoiding company X, or worse bombarding them with communication in support of the inconvenienced user.
Also, rechecking the ID of a user can be as simple as asking for a new token payment by the same credit card as used by the account. It's not infailable, the CC can be compromised as well, but it should be way better than what we have now.
This is excellent security, as long as they're not sending it to an address you provided over the phone when you requested a reset.
Not always true! Less than a month ago I needed to login to my Wells Fargo account. Unbeknownst to me, they had been doing some 'upgrades' and there were some glitches. After a frustrating period, I decide I'd just go to the physical branch 1/4 mile from my house and get this fixed!
On site, the bank personnel have access to exactly the same system that I did. (At least they knew there were glitches and sorta how to work around them.) I had two accounts, one for a credit card that I rarely used and my mortgage. Turns out, if you have a credit card then the new system requires a piece of information only found on the physical card - the onsite employees couldn't get around and neither could their call-in tech support!!!!
Point is - for log-in purposes - don't assume going to a physical branch will be any more helpful!
Since I didn't have the credit card with me ('cause rarely used) I canceled the rarely used credit card and was able to login shortly thereafter.
E.g. namecheap.com generates verification ticket item requiring valid identification and SSN that Bank of America then uses to verify your identity for $30, and vouched for the identity. Meanwhile Goldman Sachs generates a verification ticket requiring much more strenuous authentication, and the bank charges $200 for (with increased insurance, etc), which satisfies the much higher validation standard the Goldman Sachs requires to authenticate you for your ritrement account with over $X in it, etc.
> "You forgot the password that you've logged in with multiple times... including 20 minutes ago."
That should raise a flag.
People using a password manager might not ever know their password. Funny things happen with password managers where history is missing, changes don't save, keystrokes break things. We can't penalize users who use them.
It's unfortunately a really messy area.
Source: was a password manager in a past life
It's a similar situation to someone who only ever uses their credit card to buy small amounts from their local supermarket. Then suddenly they use it to buy a flight in another country. It might be legit, but it's often not, and should suggest that customer service need to do more investigation before approving.
Depends on what that account controlled.
You don't need to log into your VPS provider's account or domain name provider's account very often, compared to how often you use the machine or domain. But you don't want those getting reset more easily just because you haven't logged into them in a while.
Some people would be happy to pay for a leaked copy of those photos, for use with any other company that would accept only the "face/license" photo as sufficient proof.
I'd be surprised if ever a Blizzard account was compromised by someone sending in a false picture.
There's services on the darknet where you can buy these fake Id picture/scans as well.
I think you are a little confused here. There's no way for Blizzard to authenticate those pictures, you can take literally anyones passport and just swap the name on it.
> Took a heck of a lot more work than asking someone for username/pass in a live chat
This might be true in a world without photoshop, but that's not the world we live in.
>I'd be surprised if ever a Blizzard account was compromised by someone sending in a false picture.
In my personal experience, they'll very rarely insist upon receiving those photos. My battle.net account isn't even under a real name and despite that I've had the authenticator added and removed several times.
As someone who forgets his passwords on a regular basis, I'm kind of glad that there's no such thing as perfect security...
I backup my 2FA tokens using TiBu (encrypted locally and uploaded straight to cloud services). I'm 99.9% sure I'm never going to lose them.
I want an option in my panel that says "do not let me use support without providing a 2fa token".
All of my security questions are passwords. I once had someone at a bank ask "Wait, your mother's maiden's name has a number in it?"
"Wait, you actually answer security questions that any of your friends can guess honestly?"
You want to know how bad it can break down? I imagine the worst case scenario, for so many reasons, actually happened and was mtgox.com. It started out as a Magic: The Gathering Online Exchange (from what I understand) before it became the now infamous Bitcoin exchange that was hacked[1] and massive amounts of money was stolen. I don't know for a fact that old accounts before the pivot to Bitcoin still existed and worked, but it's not inconceivable that they would. One hopes they adopted much better security compared to when they were a trading card exchange (if it wasn't already exceptional at that time), but there could very well have been a time when it was gaining traction for financial type services but didn't have good account safeguards.
1: Or whatever. From what I remember that's a story convoluted and with enough conspiracy theories it's worth a movie.
I guess that's a long-winded way of saying you are probably right.
Then again, bitcoins were worth so little in 2010 that I could imagine the account security on exchanges back then being comparable to hobbiest exchange sites, so the spirit of the comment may be valid even if the specific example falls down. :/
1: https://en.wikipedia.org/wiki/Mt._Gox#Founding
Magic Online accounts can be worth tens of thousands of dollars.
And at least back when I played, if your account got compromised and all your cards were liquidated, the response from customer support was pretty much "that's too bad, shouldn't have let your account get compromised". I seriously doubt they ever added 2FA or anything either.
It might cost 10k+ dollars to make a behemoth account, but I don't think they are worth that much. It's basically fake internet points, I can gain these for free in this very comment.
And apparently I am the nacho King.
What are the chances that you'll remember it in the future? You're already two years into remembering it for the next decade after you found it out again.
Following up I had to reset my passwords a few more times to ensure the attacker's session was properly terminated. They identified the IPs accessing my account but they were all proxy IPs from Europe and Israel so not much could be done.
I still have the support tickets the attacker made, and I feel their support staff should have recognized it as a compromised account (they changed the name to "John Smith", bad spelling / grammar, asking to reset the root password, accidentally "lost" their private key and need password logins and root SSH enabling, etc).
Aside from this incident though I've been pretty happy there, the hardware is competitive, uptime is good and the network is solid which is all I really ask for with a server provider.
I realize there has to be a way to work around 2FA, for situations like loss of device, terminations, etc, but that should have some known policy way of verifying (say, you must send a notarized or local equivalent letter, or appear in person somehow) identity, especially for a cloud service.
(Totally incidental and you perhaps won't see this, but, r1ch, Conflict Crusher was instrumental in 15 year old me learning how to mod TA, which led me to learn how write BOS scripts, which led me to realize that I liked this programming thing, which led me to my current career/way to support my family. Thanks! :) )
Godaddy has gotten really good at preventing social engineering stacks like this. I use 2 factor authentication for my account and customer service can't event talk to me till I give them the code. Don't have that? Need to send them my drivers license and other proof to get the account reset.
Security is almost bomb proof with IP restrictions, GPG keys, 2FA and one little checkbox in their settings I like a lot: "This setting allows you to authorize or disable password resets from the login screen."
I think a good practice is to try to social engineer your own account and see if you end up getting it.
There will always be rare occasions such as this, but considering how many customers Namecheap handle, I don't think we should be seriously concerned. I'm pretty confident lessons will be learned.
Yes, I work for Namecheap, but that's probably implied by my comment.
> I’m pretty careful to use 2FA for any service that I consider important
I'm surprised no one else has mentioned no 2FA for the email. The email being compromised opened the door to this happening.
https://news.ycombinator.com/item?id=11480221
You should not be able to overcome 2FA with social engineering wtf!
Its been months since I wanted to write a detailed summary, but the notion that "namecheap is hackers best domain registrar" is not valid anymore!
About year ago I noticed DNS changes on many of my there-parked domains. Upon reaching via Chat (no phone support so that angry customers cannot vent off) I was told that they cannot help me cause Im not the owner of the account! Upon full verification even with CC on file and telling them purchase history going back to 2009, I was still denied the access. As it turned out, all hacker needed to know is my public WHOIS info to take over my account!! That was insane! Only continuance of threats from my side that I will plaster it all over the net made them change their mind, which again is a breach of trust - what if I was actually the hacker??
What really made me start moving domains to NameSilo (Im not affiliated) is that upon doing a thorough research, I found many cases where Namecheap gives up on fighting for peoples domain! I seen names like nanotmz where company was building some sort of magnetic devices and TMZ came in and threat to sue Namecheap if they dont shut the domain down. That's where I found similar cases for NameSilo and learnt that they stand their ground and would not give up on your domains, even if are threatened with legal action.
I'm out of Namecheap completely as of last month with last SSL expiring.
It doesn't help that the front door is securely locked when the back door is not! :-/
It's a level of security-through-obscurity, yes, but that doesn't mean it's wrong. It means you can keep that address monitored well. You could make any activity on it send a page, for example.
This is the same for DigitalOcean. I'm always amazed that clicking "Rebuild" or "Delete + Scrub Data" doesn't require _any_ confirmation at all.