HN Explorer

Tell HN: How to Respond to a Domain Takeover

41 points by netsectoday ↗ HN
After having my domain hijacked today I'd like to share an incident postmortem just in case you find yourself in the same situation (swearing at your terminal output at 10am on Friday morning when your website is down).

9:58am I can no longer deploy to production

10:15am Finished troubleshooting all services, no problems identified

10:16am nslookup resolves to some random IP address instead of my prod server (WTF!!!!!!)

10:20am Log into registrar and find out they replaced my custom DNS servers with their own and added records to serve a "Parked free courtesy of GoDaddy" page with ads and a button that says "Get This Domain"

10:30am Changed my domain on the registrar website back to my custom DNS servers

10:32am Changed my password on the registrar website

10:38am Got told by GoDaddy support they didn't have anything to do with this and it was my fault it happened (f-me, right?)

11:55am DNS records across the internet are still jacked

12:00pm Manually blow out the cache on cloudflare for my domain

Postmortem Suggestions:

* If your website goes down; don't blow 15+ minutes troubleshooting your app services before checking DNS

* Enable 2fa with your registrar (even though there was no alert for us)

* Set up an alert for when your domain resolves to a different IP address (make a script and host it elsewhere or pay for a service)

* Don't trust your registrar!!!!

* Take a screenshot of your registrar settings and DNS settings right now so you have a record when they disappear

* Get access to your registrar account ASAP after the attack and change your DNS records back using the screenshots you just took

* Manually purge the cache of major DNS providers (for your domain) to allow your DNS records to propagate: https://cloudflare-dns.com/purge-cache/

22 comments

[ 4.2 ms ] story [ 62.6 ms ] thread
dazc
Rule #1: Domain registrar and DNS should never be the same source.

Rule #2: See Rule #1

Rule #3: https://www.gov.uk/guidance/protect-domains-that-dont-send-e...

somehnguy
Laziness wing again! I’ve been meaning to move my domains from Namecheap to Cloudflare for no reason other than I switched to Cloudflare DNS years ago. Turns out I was just following best practices :)

Side note so this comment isn’t completely devoid of info: in the 10+ years I’ve had domains at Namecheap I’ve never had so much as a hiccup. Highly recommend them as a registrar.

netsectoday
> Rule #1: Domain registrar and DNS should never be the same source.

The domain registrar POINTS to whatever DNS servers, so in this attack the registrar just POINTED to their own DNS servers. Rule #1 and Rule #2 were followed but the domain was still taken over. The attacked domain also needs to send email so the gov.uk link is great guidance but can't be followed here.

agwa
> Rule #1: Domain registrar and DNS should never be the same source.

But why? A compromise of either service leads to a domain takeover (although recovering from a DNS compromise is easier than recovering from a registrar compromise) so you're doubling your risk exposure if you follow this rule.

dazc
Recovering one service from one source is a lot easier than recovering two at the same time from one single source.
boolean
Please don't use GoDaddy. You can search HN to see why it's a horrible company.
netsectoday
What registrar would you recommend and why?
codecutter
I would recommend Porkbun.com or Gandi.net I have personally used both, but now moved away from Gandi. (There was billing issue and I was not happy how it was handled.) Anyway, happy with Porkbun. When I tried to register my domain for 10 years, they actually called me to confirm that it was not a mistake on my part. Very pleased with Porkbun support & yes, you can talk to a human.
itake
I switched from porkbun to cloudflare recently due to lower pricing.
jacooper
But you are locked to cloudflare DNS.
itake
Not sure if that's an issue for me. I was using cf dns on porkbun anyways. not sure what is wrong with cf dns...
aspyct
I recommend OVH. My oldest domain is now 17yo, and I've never had any issue.
cpach
Dynadot has worked well for me. They have support over chat, email and telephone.
megraf
101domain has been my go to for a while now. I have a half dozen domains with them.

You missed a couple steps, I'd say:

- make sure you've enabled domain transfer / termination locking - setup uptime monitoring - get off of GoDaddy

netsectoday
The domain was never transferred, so a transfer lock wouldn't have stopped this. I need to look into the termination locking. It's not 100% clear to me if GoDaddy locks their DNS nameserver pointer to an external DNS server, or if they would just lock DNS records hosted on their own DNS servers.
the__alchemist
What would cause this? Compromised password? Hacks into GoDaddy? What is the recourse?
netsectoday
[OP] here...

There were no 2fa alerts or any record of anomalous logins, so it's unlikely the password was compromised.

Even locking my domain from transfer wouldn't have stopped this from happening, the domain wasn't transferred: only the pointers to new DNS servers were changed.

This really smells like GoDaddy has bad programmers that introduced bugs in their infrastructure, or hackers poking around their back-end triggering stored procedures.

I'm not sure if it's 100% clear from the write up, but my domain was "seized" by GoDaddy's systems that host an insecure "Hosted by GoDaddy" welcome page. I didn't click on the "Buy This Domain" button, but it could have been spoofed to look like GoDaddy and actually taken me to some command and control server if this wasn't GoDaddy that performed the domain takeover.

Recourse? I called GoDaddy support and they were 700% confident their systems are perfect. I was told by the first support rep "I'm writing this down as caused by customer", so I told him "Well I'm marking this in my system as caused by GoDaddy" as we both laughed.

I called GoDaddy support a second time after I was partially convinced this was my fault and told them "Absolutely not, you guys messed up this needs to be escalated" as I gave the support representative a primer on the basics of how DNS works and why it broke for my domain (because he was clueless). Then emailed him a screenshot of my domain with "Parked by GoDaddy" written all over it because he didn't believe me.

So my recourse here is getting called an idiot by barely conscious GoDaddy support staff, posting this on HN to share the experience and getting called out again as an idiot for using GoDaddy (this is my first issue with them in 20 years). This has been happening to random GoDaddy customers for 4+ years and GoDaddy is so full of themselves they won't even open a support ticket to consider it might be their fault. I have no recourse except to transfer my domains to a responsible registrar.

arunnanda
Rule of thumb in dealing with domain and hosting vendors - always gauge how educated their support staff is. They should be at least slightly more educated than the average developer who admins their own stuff. And do judge based on website design - if their design looks like it'd would also be appropriate for selling quick ways to lose fat, you should know to stay away.
mackatsol
I’ve been very happy with Hover.com.

… and I’ve seen the same issue happen with many many domains registered with GoDaddy and NetWork Solutions. Avoid them. Either NAmeServers get reset or DNS entries vanish even though they are all there when you login to fix it.