We had a similar incident with Google's Safe Browsing list at work several weeks ago. It's the most infuriating thing to one day wake up and have all our (legitimate) sites effectively taken off the Internet for the public because some AI at Google decided our site was unsafe (in our case, presumably, because a self hosted Atlassian login page on a subdomain looked too similar to other sites).
To make matters worse getting hold of anyone at Google is next to impossible and in our case it took 48 hours before the review we requested in their console had any effect. On top of that though it didn't only affect Google Chrome, it was all major browsers _and_ services such as NextDNS, or DNS services provided by ISPs(!) which took far longer than 48h to eventually drop off.
Nice to see thorough incident reports from gov.uk and I think actually this shows the google safe browsing system working well. The domain assets-origin.production.govuk.digital "looks" very much like a phishing url, even if it's internal.
> The domain assets-origin.production.govuk.digital "looks" very much like a phishing url, even if it's internal
.digital is a real TLD, and govuk.digital is an actual domain, registered by "Government Digital Service" (created 2017-07-14T14:17:30Z) although the registrant's details are redacted "for privacy".
Is this domain really registered to some part of uk.gov? ...or is this some bad actor/squatter who found out about the internal naming and snapped up the matching public domain?
Q: Shouldn't gov.uk just stick to domains under .gov.uk?(!) Would make it easier to work out who's actually gov.uk and who's merely pretending :)
Aren’t gov.uk domains transferred internally within UK government depts at zero cost ?
I guess there’s got to be some incentive for teams to not register hundreds of subdomains, but if they are turning to 3rd party TLDs to host government assets, I’d say something has gone wrong.
If the process is complex (write a detailed ticket, wait for feedback, get some manager's nod, etc.), buying a domain and deploying it yourself is orders of magnitude cheaper and faster, and can be the only way to test something in an env that resembles prod. I have done it myself for non critical stuff, although I'm not sure I would at some government org.
> If the process is complex (write a detailed ticket, wait for feedback, get some manager's nod, etc.), buying a domain and deploying it yourself is orders of magnitude cheaper and faster [..]
Isn't this basically the very definition of "shadow IT"?
Then you find that some prod[-ish] service relies on an external provider that's not listed as a provider in any of your org's financial systems, is paid by some random internal person's credit card and expensed back ... then that internal person leaves, and N weeks/months later all hell breaks loose when your service falls over and no-one knows why.
Most non SOC certified companies have some non auditable processes, the ubiquitous one being deploying stuff manually / non reproducibly. Whether something is "shadow IT" will depend on who you ask. Personally I have never deployed something to prod that way (I did it for some lower envs), but it's something I've seen people do at every single place I've worked. And your last paragraph is 100% true in my experience.
This might sound a bit crazy, but... the UK government is in sole control of .gov.uk, right? They shouldn't have issues issuing themselves with as many subdomains as needed. For free, or for symbolic prices.
Sort of. The UK Government (specifically the Cabinet Office) manages various aspects of the *.gov.uk space, but .gov.uk domains can be registered not only by the UK Government, but also local governments and other public sector organisations.
.gov.uk domains are registered through commercial registrars (who charge a fee), much like .co.uk domain names. The main difference being that you need to be an eligible organisation to register a .gov.uk domain name.
Absolutely untrue. The main reason these domains get bought is because it’s simpler.
There are likely procedures in place and security checks and people involved in gov.uk domain control, that are a non issue if the team owns the tld, which makes internal testing easier.
So why is using .govuk.digital for internal services - while still having to purchase and renew the actual govuk.digital domain - preferable to, say, setting everything up under .internal.gov.uk?
- To reduce the attack surface
In the event of site with a *.gov.uk subdomain getting compromised at least it won't now be able to steal auth cookies for internal services
- to keep test/stage as faithful a copy of prod as possible they will have a totally separate but the same DNS set up/CDN set up/ load balancing etc. Theoretically the only difference would need to be one routing rule rather than stuff that might start creating edge case bugs with certs/cookies etc where there are different numbers of segments in domains. Also allows for more certainty/confidence when something is tested in a lower environment that it will work when promoted to prod
It's good security practice to have separate sandboxes/domains for various types of services so it's harder for people to pivot from one to another in the event of an exploit. Similar to having a different domain for internal services, if you host user content on your site you should host all that user generated content on a different domain so it can't access your cookies etc.
> Shouldn't gov.uk just stick to domains under .gov.uk?
TFA says this was the intention, and the govuk.digital reference never worked outside the internal network (the subdomain in question does resolve right now, but the pointed-to host seems to be firewalled). That alone is much better than I’ve seen banks do.
Given that *.production.govuk.digital is presumably accompanied by *.staging.govuk.digital or similar, I can see why they took out a separate domain for internal purposes, e.g. so that breaking DNS for it does not immediately put the whole of public-facing gov.uk on fire, or so that the CDN can pick up backend IP changes without modifications to the gov.uk DNS zone, or for some other infrastructural reason. It’s not like domains are expensive, and this one honestly looks like it might have been left over from the time they were rolling out the whole thing.
(TIL you can set favicon URLs for non-HTML documents: apparently putting rel="shortcut icon" in the HTTP Link header should work? Although gov.uk is not doing that right now.)
My guess is that the "UK Government Digital Service", who run most central government it services, use govuk.digital as a top level domain for all internal services.
I believe it's fairly common to have a TLD that is only used for internal addresses, it helps to ensure they are only routed internally and not accidentally exposed. Using a "real" TLD also ensures you own the namespace and it can't be accidentally used by a third party.
If they were to use an *.gov.uk domain for internal services there is a higher chance that someone would inadvertently expose it to the internet.
> Shouldn't gov.uk just stick to domains under .gov.uk?(!) Would make it easier to work out who's actually gov.uk and who's merely pretending :)
Much smaller scale example: Where I work, our production web app is at `<subdomain>.<charity>.ac.uk`.
DNS and SSL certs are managed centrally by IT. We open a support ticket, someone in IT gets it done. DNS entries are usually done within a few hours, but they like to purchase SSL certs in batches so if we need a cert for a new domain/subdomain it could take a few days or weeks.
It's useful for us to spin up multiple different dev/test/staging copies of our app, which are otherwise nearly identical to production in terms of how the infrastructure is set up. These are short-lived and used for internal user acceptance testing before we roll out changes to the live app - they're only accessible on the local network (so, via VPN or onsite), but we want to reduce friction to our users, as the ones doing the acceptance testing are domain experts (i.e. scientists, researchers, electrical engineers) but don't want to be tinkering around with software dev stuff like "editing /etc/hosts" and "accepting self signed SSL certificates".
We don't want to wait around for a whole sprint length's amount of time waiting on new temp subdomains when we've got new stuff to demo to people. It's a lot easier to just claim a domain name of our own (i.e. `.ga` and `.ml` are free) and set them up with external DNS management (have used cloudflare before, because it's easy to do letsencrypt DNS challenge method with it), and use letsencrypt for the SSL certs.
We could theoretically ask IT to let us manage the DNS ourselves for everything under `<subdomain>.<charity>.ac.uk`, i.e. they point that subdomain to an `NS` record to an authoritative DNS server that we run ourselves, and we could ask them for some wildcard certificates so we could do stuff like `staging-env-123.<subdomain>.<charity>.ac.uk`.
But our team is small, and we don't really _want_ to manage yet another service (our own DNS)* and have it adequately locked down to appease "cyber essentials" and so forth :-) and neither do IT want to defer DNS to "knowledgeable, but still users" users of the corporate network. And wildcard certificates are iffy from a security standpoint also.
*In practise this'd be letting kubernetes components manage it, but k8s is so bloody brittle and oh god I cannot keep up with all of this devops stuff there's like 5 of us stop making us run servers on top of servers.
It would not surprise me if there's similar gatekeeping of .gov.uk subdomains and SSL certs, necessary-for-security bureaucracy given their high profile, which slows down the deployment of temp. staging / "internal user acceptance testing" environments, and so the internal dev teams at some point went "fuck it, we'll buy a new domain ourselves" and someone paid for the govuk.digital domain using a Government Purchase Card and claimed it on the relevant Project Code.
EDIT: Reading comments about buy-your-own-domain being "shadow IT". Salient points about random people in the org paying for third-party services with a company credit card and then leaving. It's important to get in the "bus factor" mindset with this stuff, e.g. "if I got hit by a bus today, would anything fuck up in work if I wasn't there to remember login details?". Make sure logins for things are shared with the whole team in a secure way, make use of tools which support organisation accounts, have a shared mailbox etc.... oh and make sure the companies/services gets registered as a supplier with finance.
Yeah, it's annoying, it's all political reasons too AFAICT.
LE is also verboten because of stipulations from Certain Government Departments. There was definitely pushback years ago about just using LE because "it's bad for security" which effectively translated to "we don't trust it because it's free and we have less control over the domains issued". Hopefully that mindset changes now that LE is well-established.
Also should be noted that technically most of the domains are owned/purchased by the research councils (govt level) and they buy certificates which cover like 100 different domains (wtf, this seems ripe for domain-squatting) at a time, so at least they're getting their moneys worth out of those certificate renewals. So it takes even longer for us to get certs because the _research council(s)_ are busy batch-renewing certs for multiple institutions they're involved with.
Individual research projects also end up with their own domains that one EU country pays for, another country hosts, and a third one manages, because they all met up for beers in Brussels for a kickoff meeting and decided that was the fairest way of doing it. Each institutions IT department trying to make sense of the chaos which the principal investigators are creating, "seek forgiveness rather than permission" :-)
90s, Microsoft IE, blah blah blah — you know the history.
For my part, I'm pushing as hard as I can to replace Chrome on everyone's machines with Firefox. It's not even the same game any more though.
In the 90s there was still a fighting chance — and we won! — because the technologies at play were not entrenched in every day life. Now it's more akin to replacing every car with an EV.
I praise your effort, I'd like to see more diverse browser landscape as well.
But in this particular case I think that different browser (or particularly Firefox) would not possibly prevent that problem, since (Firefox) shares same "Safe Browsing" database with Google, for "Phishing Protection".
> (..) since Firefox shares same "Safe Browsing" database with Google
This. Firefox even sends the hash and filename of each download to Google. So if you use Firefox to download the Tor browser, an abortion price list or a secret PDF, Google knows you downloaded it.
Probably a sad reflection of recent prominent news stories regarding governments that don’t tell the truth to their people. By my calculation covering a sizeable chunk of the world’s population.
Google safe browser API returns safe for some URLs blocked in chrome so you don't even know what content is unsafe to fetch (like images) before your site gets this hideous red warning appear
While doing SEO and content for my startup, I have realised Google has way too much power to make or break an organisation. When it works in your favour you can build billion dollar businesses when it works against you it can kill billion dollar businesses.
I am so impressed by the transparent and accessible writing style of
uk.gov. As each paragraph raised questions in my mind, they were
precisely answered in the next.
For once I can't blame Google. Production should have known better
than to roll out some random .digital domain. I would definitely not
trust this if I had spotted it manually.
I had this too. Google decided to block my company's domain for no good reason. I also couldn't reach anyone in Google to sort it out - I eventually had a response that "it's an automated system, there is noone who looks after it or can do anything about it".
My company's email is hosted on GMail, and that also silently swallowed any in- or out-bound emails that as much as mentioned my domain. Including the email to our support inbox from our customer telling us they were having problems.
Running anything in production at the whim of Google is too much of a risk, and I've been migrating everything I can off them ever since. They're a menace.
37 comments
[ 9.6 ms ] story [ 84.1 ms ] threadTo make matters worse getting hold of anyone at Google is next to impossible and in our case it took 48 hours before the review we requested in their console had any effect. On top of that though it didn't only affect Google Chrome, it was all major browsers _and_ services such as NextDNS, or DNS services provided by ISPs(!) which took far longer than 48h to eventually drop off.
.digital is a real TLD, and govuk.digital is an actual domain, registered by "Government Digital Service" (created 2017-07-14T14:17:30Z) although the registrant's details are redacted "for privacy".
Is this domain really registered to some part of uk.gov? ...or is this some bad actor/squatter who found out about the internal naming and snapped up the matching public domain?
Q: Shouldn't gov.uk just stick to domains under .gov.uk?(!) Would make it easier to work out who's actually gov.uk and who's merely pretending :)
The main reason to buy those domains is to spend as little money as possible. Someone at gov.uk is a cheapskate or works with a really limited budget.
I guess there’s got to be some incentive for teams to not register hundreds of subdomains, but if they are turning to 3rd party TLDs to host government assets, I’d say something has gone wrong.
Isn't this basically the very definition of "shadow IT"?
Then you find that some prod[-ish] service relies on an external provider that's not listed as a provider in any of your org's financial systems, is paid by some random internal person's credit card and expensed back ... then that internal person leaves, and N weeks/months later all hell breaks loose when your service falls over and no-one knows why.
.gov.uk domains are registered through commercial registrars (who charge a fee), much like .co.uk domain names. The main difference being that you need to be an eligible organisation to register a .gov.uk domain name.
There are likely procedures in place and security checks and people involved in gov.uk domain control, that are a non issue if the team owns the tld, which makes internal testing easier.
That may also be (tangentially) true, but someone also paid _actual_ money to register the _actual_ .digital domain.
Was this 'someone' really uk.gov? If so, why?
- to keep test/stage as faithful a copy of prod as possible they will have a totally separate but the same DNS set up/CDN set up/ load balancing etc. Theoretically the only difference would need to be one routing rule rather than stuff that might start creating edge case bugs with certs/cookies etc where there are different numbers of segments in domains. Also allows for more certainty/confidence when something is tested in a lower environment that it will work when promoted to prod
TFA says this was the intention, and the govuk.digital reference never worked outside the internal network (the subdomain in question does resolve right now, but the pointed-to host seems to be firewalled). That alone is much better than I’ve seen banks do.
Given that *.production.govuk.digital is presumably accompanied by *.staging.govuk.digital or similar, I can see why they took out a separate domain for internal purposes, e.g. so that breaking DNS for it does not immediately put the whole of public-facing gov.uk on fire, or so that the CDN can pick up backend IP changes without modifications to the gov.uk DNS zone, or for some other infrastructural reason. It’s not like domains are expensive, and this one honestly looks like it might have been left over from the time they were rolling out the whole thing.
(TIL you can set favicon URLs for non-HTML documents: apparently putting rel="shortcut icon" in the HTTP Link header should work? Although gov.uk is not doing that right now.)
I believe it's fairly common to have a TLD that is only used for internal addresses, it helps to ensure they are only routed internally and not accidentally exposed. Using a "real" TLD also ensures you own the namespace and it can't be accidentally used by a third party.
If they were to use an *.gov.uk domain for internal services there is a higher chance that someone would inadvertently expose it to the internet.
https://www.gov.uk/government/organisations/government-digit...
Much smaller scale example: Where I work, our production web app is at `<subdomain>.<charity>.ac.uk`.
DNS and SSL certs are managed centrally by IT. We open a support ticket, someone in IT gets it done. DNS entries are usually done within a few hours, but they like to purchase SSL certs in batches so if we need a cert for a new domain/subdomain it could take a few days or weeks.
It's useful for us to spin up multiple different dev/test/staging copies of our app, which are otherwise nearly identical to production in terms of how the infrastructure is set up. These are short-lived and used for internal user acceptance testing before we roll out changes to the live app - they're only accessible on the local network (so, via VPN or onsite), but we want to reduce friction to our users, as the ones doing the acceptance testing are domain experts (i.e. scientists, researchers, electrical engineers) but don't want to be tinkering around with software dev stuff like "editing /etc/hosts" and "accepting self signed SSL certificates".
We don't want to wait around for a whole sprint length's amount of time waiting on new temp subdomains when we've got new stuff to demo to people. It's a lot easier to just claim a domain name of our own (i.e. `.ga` and `.ml` are free) and set them up with external DNS management (have used cloudflare before, because it's easy to do letsencrypt DNS challenge method with it), and use letsencrypt for the SSL certs.
We could theoretically ask IT to let us manage the DNS ourselves for everything under `<subdomain>.<charity>.ac.uk`, i.e. they point that subdomain to an `NS` record to an authoritative DNS server that we run ourselves, and we could ask them for some wildcard certificates so we could do stuff like `staging-env-123.<subdomain>.<charity>.ac.uk`.
But our team is small, and we don't really _want_ to manage yet another service (our own DNS)* and have it adequately locked down to appease "cyber essentials" and so forth :-) and neither do IT want to defer DNS to "knowledgeable, but still users" users of the corporate network. And wildcard certificates are iffy from a security standpoint also.
*In practise this'd be letting kubernetes components manage it, but k8s is so bloody brittle and oh god I cannot keep up with all of this devops stuff there's like 5 of us stop making us run servers on top of servers.
It would not surprise me if there's similar gatekeeping of .gov.uk subdomains and SSL certs, necessary-for-security bureaucracy given their high profile, which slows down the deployment of temp. staging / "internal user acceptance testing" environments, and so the internal dev teams at some point went "fuck it, we'll buy a new domain ourselves" and someone paid for the govuk.digital domain using a Government Purchase Card and claimed it on the relevant Project Code.
EDIT: Reading comments about buy-your-own-domain being "shadow IT". Salient points about random people in the org paying for third-party services with a company credit card and then leaving. It's important to get in the "bus factor" mindset with this stuff, e.g. "if I got hit by a bus today, would anything fuck up in work if I wasn't there to remember login details?". Make sure logins for things are shared with the whole team in a secure way, make use of tools which support organisation accounts, have a shared mailbox etc.... oh and make sure the companies/services gets registered as a supplier with finance.
So still operating in a pre "Let's Encrypt" period. Disappointing to hear a charity is wasting money on SSL certs.
LE is also verboten because of stipulations from Certain Government Departments. There was definitely pushback years ago about just using LE because "it's bad for security" which effectively translated to "we don't trust it because it's free and we have less control over the domains issued". Hopefully that mindset changes now that LE is well-established.
Also should be noted that technically most of the domains are owned/purchased by the research councils (govt level) and they buy certificates which cover like 100 different domains (wtf, this seems ripe for domain-squatting) at a time, so at least they're getting their moneys worth out of those certificate renewals. So it takes even longer for us to get certs because the _research council(s)_ are busy batch-renewing certs for multiple institutions they're involved with.
Individual research projects also end up with their own domains that one EU country pays for, another country hosts, and a third one manages, because they all met up for beers in Brussels for a kickoff meeting and decided that was the fairest way of doing it. Each institutions IT department trying to make sense of the chaos which the principal investigators are creating, "seek forgiveness rather than permission" :-)
For my part, I'm pushing as hard as I can to replace Chrome on everyone's machines with Firefox. It's not even the same game any more though.
In the 90s there was still a fighting chance — and we won! — because the technologies at play were not entrenched in every day life. Now it's more akin to replacing every car with an EV.
But in this particular case I think that different browser (or particularly Firefox) would not possibly prevent that problem, since (Firefox) shares same "Safe Browsing" database with Google, for "Phishing Protection".
https://wiki.mozilla.org/Security/Safe_Browsing
This. Firefox even sends the hash and filename of each download to Google. So if you use Firefox to download the Tor browser, an abortion price list or a secret PDF, Google knows you downloaded it.
https://www.pentagrid.ch/en/blog/block_browser_requests_to_g...
Disclaimer: The blogpost is two years old, but I assume this is still current.
For once I can't blame Google. Production should have known better than to roll out some random .digital domain. I would definitely not trust this if I had spotted it manually.
My company's email is hosted on GMail, and that also silently swallowed any in- or out-bound emails that as much as mentioned my domain. Including the email to our support inbox from our customer telling us they were having problems.
Running anything in production at the whim of Google is too much of a risk, and I've been migrating everything I can off them ever since. They're a menace.