HN Explorer

Tell HN: Don't put your status page on the same domain as your app

98 points by IgorPartola ↗ HN
If your status page returns a 502 at the same time that your app returns a 502, you might be in trouble. https://status.discord.com/ really should be on a different domain. Of course the naked 502 from CloudFlare is enough to tell me that there is an outage, but overall this is probably not the ideal way of doing things. Domains are cheap. Get a status page with a different one (or a hosted service).

59 comments

[ 6.2 ms ] story [ 132 ms ] thread
fire
It's usually less about the subdomain and more about dns and content hosting of the status page
mmcallister
I think the _real_ takeaway here should be that your status page should be simple, with as little dependencies as possible, ideally none.

IMHO it should be a static page using Hugo, Jekyll or roll your own if you really have to.

None of this precludes you from using the same domain, though you'd want to use a subdomain, for instance status.product.com

ameliaquining
You can't avoid all dependencies. Certainly, avoiding a dependency on a complicated application server is good, but somewhere in the mix there has to be a DNS nameserver, an HTTPS server, some kind of persistent storage for the status, and some way for you to update that status. If your status page is a static page, then that just means that your persistent storage is wherever the HTML page itself is stored; an S3 bucket, say, or the filesystem of a machine that's running Apache.

The thing being pointed out in the OP is that, while you can't avoid all dependencies, you can avoid having any shared dependencies (other than core internet infrastructure) between your status page and the service whose status it's reporting on. That way, outage risk will not be correlated between the two, which is generally good enough, since almost no one cares about your status page when your actual service is not experiencing an outage. One effective way to do this is with a service like status.io that specifically hosts status pages and specializes in having very high uptime for just that one kind of page.

If you use the same domain for your status page as for your main service, then that may mean that an outage in your application server, application database, etc., won't affect your status page, but the two will still share a dependency on your load balancer (i.e., whatever your A records are pointing directly at), so if anything goes wrong there then your status page will go down with your main service. If you use a subdomain then there won't necessarily be a shared dependency on the load balancer, but there will be one on the DNS nameservers. The only way to avoid all shared dependencies is to use an entirely separate domain.

I don't know if Discord's problem that the OP is talking about had anything to do with DNS, but I think that's been a source of outages for them in the past, in which case a separate domain is the solution.

IgorPartola
The behavior I observed in this particular case is that their server would timeout on requests, especially writes, then come back with a quick 502, then time out again. Their status page displayed the same kind of behavior and the same status page. I wouldn’t be surprised if the issue with the status page is that it might also even share the webservers with their main service.
the_arun
I thought real takeaway is your status page should be up even if your application is down OR they have to be independent.
neilv
Some startup could do the `.status` Internet domain name TLD, plus offer an optional SaaS there.

(The SaaS could provide numerous different ways to push status to your branded `.status` domain page, plus optional watchdog/heartbeat. And the implementation could be be resilient against even a major cloud provider outage.)

Using dotStatus could be the no-brainer go-to for companies that know they should have an independent status page, or are required to have it, and don't want to spend a lot of engineering resources doing it right.

nexuist
Then we'd need a .status.status to tell if the SaaS is down :)
IgorPartola
Clearly we need to put this on a blockchain.
Karunamon
You meme, but having status updates on a widely distributed network already being used for other things would make it incredibly resilient. You can already sign messages without involving currency malarkey in most blockchains that matter.
cguess
Sure, but actually committing the status every few minutes (seconds?) to a blockchain would use so much resources it's be... quite impolite.

It'd be easier, and cheaper eventually, to have it on a properly sharded database, maybe with a few different DNS and server front ends.

tnzm
>committing the status every few minutes (seconds?) to a blockchain Only if the status actually changes every few minutes.
supermatt
I’d expect this would need to be like a heartbeat / dead mans switch, I.e keep flagging as up, and when you stop flagging, it’s considered down. Otherwise you would need a different status monitor to monitor the status agent…
Karunamon
Which is fine. Having a standardized .status domain for the pages, even if manually updated, is a big upgrade from the hodgepodge of what's out there now.
KingMachiavelli
You could use something like IPFS to publish a static page every X minutes. No blockchain required.
Karunamon
IPFS for pages like this is like a random webpage with extra steps - content on IPFS disappears if nobody has pinned it, so I don't think its benefits would be realized.
AceJohnny2
Status updates do not need distributed consensus or proof of work.

not the "Right tool for the job"

bob33212
Mint coins so that they can be used to check the status. Then trade the coins.
toast0
Well, it should be on a different domain, so status.com or statusstatus.com
tomlagier
This is a great idea and caused me to go down a rabbit hole of learning what it would take to actually register a new gTLD.

Short version: The ICANN would need to do a new round of gTLD registrations. This is unannounced but rumored to begin some time this year.

Then, a corporation would need to apply for the gTLD string. This application cost $185,000, non-refundable, in 2012 [1]. Smart money is a steeper fee.

Next, they would actually need to be approved for the gTLD. This requires a company with enough funding to pass the ICANN's audit, and enough technical chops to maintain name servers with enough bandwidth and availability to serve the traffic generated by the new gTLD [2]. There is also a criminal background check and anti-cybersquatting check [3].

Finally, they would have to win in a contention process that involves any other registrar applying for the same string. This may be settled by an auction [4].

The buzz around the 2012 process is that it was generally a waste of time and money - many of the gTLDs far underperformed expectations by the registrars. Further, many of the corporate-owned gTLDs are unused (such as the 76! owned by Amazon)[5].

There were a few winners - the ICANN themselves made a hefty sum of money - some $212 million left over in 2020 [6]. Some other companies figured out the right way to "game" the contention process and intentionally lose the auction for the string [7]. Finally, a company named "Donuts, Inc", won some 270 gTLDs under various subsidiary companies. With such a massive portfolio, they were able to capture a few big winners (such as .guru) [8].

[1] https://newgtlds.icann.org/en/applicants/global-support/faqs... point 2.2

[2] https://www.icann.org/en/system/files/files/gtld-drd-ui-10se...

[3] https://newgtlds.icann.org/en/applicants/agb/guidebook-full-... module 2

[4] https://newgtlds.icann.org/en/applicants/agb/guidebook-full-... module 1.1.2.10

[5] https://domainnamewire.com/2019/02/28/new-tlds-five-years-in...

[6] https://www.icann.org/en/system/files/files/annual-report-20... page 33

[7] https://www.circleid.com/posts/20130603_icann_auctions_or_pr...

[8] https://en.wikipedia.org/wiki/Donuts_(company)

neilv
Great investigating!

Given that the ICANN process is potentially long, the dotStatus software, infrastructure, and business-ing can be done in parallel.

And, have a backup/iterim plan of using a different domain name scheme. (Though `.status` would be awesome, and perhaps even -- dare I say it -- unicorn-scented?)

Whoever gets the funding for the dotStatus startup should bring on all the rest of us as advisors. :)

lhoff
While i like the idea I must say, not going to happen. Companies want to have the full control over the status page. At least those company that have service level agreements with their customers.
diamondhandle
According to https://discordstatus.com/, there isn’t an outage. Are you suggesting that there’s currently an ongoing outage, and their status page is useless? (Wouldn’t be the first time, but it’s surprising that an otherwise excellent engineering team would make such a mistake.)

Disclosure: not a Discord user

IgorPartola
I saw a consistent outage with their web servers, yes. It lasted about 10-15 minutes, just enough time to think of posting about it on HN.

And yes their status page was useless during this. Yes I have seen this happen elsewhere.

LinuxBender
Self reporting can be problematic if a company is looking to be acquired. I prefer to use 3rd party monitoring tools. The paid ones like 1k-eyes are great, but there are some free ones like DownDetector [1] that sometimes help paint a picture albeit not perfect.

[1] - https://downdetector.com/status/discord/

jeswin
> Domains are cheap.

Sorry, I didn't follow. How is a subdomain with an A record any different from a separate domain?

epoch_100
Suppose the outage was caused by an issue with the registrar, DNS, the name server, or the domain registration itself. Having a separate domain for the status page — ideally purchased through a different provider – could prevent a lot of issues.
ipaddr
What if the internet is down. You should offer a phone number to call to get the status.
necovek
What would customers read that phone number off?

You should advertise in Yellow Pages!

ipaddr
Millions of tiny Townhalls.
jrockway
DNS can be the cause of the issues. You can misconfigure the server, you can misconfigure NS glue, your registration can lapse, etc. If you have a second domain for your status page, it's less likely that a DNS problem with your primary domain will make that unavailable.

There is a tradeoff, of course. Your automated configuration management system can dutifully push the misconfiguration to both domains. You might not ever configure the second domain correctly, because it's not like customers are visiting and reporting back "hey your status page is broken".

In my mind, I'm not sure how useful this is in practice, however. I would never think to look for status updates at anything other than status.myapp.com. I'm not sure I'd make the leap to myappstatus.com, and I'm not sure it's going to rank particularly high in search engines. I'd probably look at Twitter second. (I'm not even sure if people routinely check status pages. If you visited my personal website, jrock.us, and it didn't load, would you check status.jrock.us? That actually exists, but I bet that nobody has ever visited it.)

blahyawnblah
Couldn't status.discord.com point to any server?
0172
I would also like to know the reason why a subdomain is not sufficient to host a status page as long it's pointing to an independent infrastructure.
spc476
subdomain.example.com can be hosted on completely different infrastructure, but what about DNS? Unless you have two different DNS providers for the main domain, you still have single point of failure for the entire domain (and all subdomains).
heipei
"Status Update, Monday: Our main domain was suspended / is being blocked for reasons unknown to us. Investigation is ongoing."

That happened to notion (notion.so) for example.

grensley
This assumes it's not a DNS problem, and it's ALWAYS DNS.
hinkley
Is that why consul has dns built into it?
sascha_sl
The point isn't that DNS is bad, but it is often neglected as a problem source, something taken a bit too much for granted.

The Kubernetes community loves this saying because pairing something like nodejs (which doesn't do DNS caching) with a Docker image with no local DNS cache/resolver and a Kubernetes cluster that doesn't usually come with a highly scaled DNS server out of the box often leads to DNS failing more often than anticipated. Also kube-dns, the now deprecated default DNS, was extremely shit.

I have a pcap file captured on a kubernetes node with 60000 DNS packets in a 0.2s frame. They're all queries for s3.amazonaws.com.

ShakataGaNai
So status.discord.com redirects to discordstatus.com - so it already is on a different domain. But the domain matters very little unless your DNS infrastructure is offline. However, then you have to assume your status site and domain have to have their own DNS. Unless you're really dedicated, you'll have some SPOF that can take down both (think even different domain registrars).

What most companies need is a solid status page that's hosted in very different infrastructure... that is very static or can scale very fast. Often the do the first... but not the second. If something like discord is down and 10 million daily-active-users all try and check the status page... they'll just DDOS that out of existence as well. Again, if not well built

huxflux
Robin Hobb is amazing, her books which begin with Assassin's Apprentice really helped me through extremely stressful periods of life being able from time to time to escape from reality and into another world and then return.
m12m3kg
> If your status page returns a 502 at the same time that your app returns a 502, you might be in trouble. https://status.discord.com/ really should be on a different domain

Subdomain can be hosted on a different server. You don't need another domain to do that.

sascha_sl
Nameserver outages happen.
m12m3kg
No they don't. One domain can point to multiple name servers. Also, good name servers provide great uptime. If you are not using some no name company's name servers, but something like AWS route 53.
edf13
You absolutely SHOULD use the same domain name - this gives confidence that the status page is from the right service and not a fake/scam site...

You simple point it somewhere else, a completely different infrastructure.

moviuro
What about DNS? If DNS is down for myapp.com, it will also be down for status.myapp.com
supermatt
You use multiple DNS servers for the domain - a common practice. (Its actually mandatory to have 2 which resolve to a different IP, but in reality this doesn't make them distinct servers, hence why i say "common" instead of "universal").
nix23
And don't store your backup in the same DC, and not with the same provider.
matt_s
I believe there are ways to solve this entirely inside of CloudFlare by having them host pages or code snippets that render a "Sorry we're down" page in the event of an outage.