HN Explorer

Tell HN: Microsoft.com added 192.168.1.1 to their DNS record

498 points by indosauros ↗ HN
Is this bad?

$ nslookup microsoft.com

Non-authoritative answer:

Name: microsoft.com Address: 192.168.1.0

Name: microsoft.com Address: 20.112.250.133

Name: microsoft.com Address: 20.231.239.246

Name: microsoft.com Address: 20.76.201.171

Name: microsoft.com Address: 20.70.246.20

Name: microsoft.com Address: 20.236.44.162

Name: microsoft.com Address: 192.168.1.1

147 comments

[ 2.7 ms ] story [ 207 ms ] thread
h2odragon
I get it too, with .1.0 as well

    Name: microsoft.com
    Address: 192.168.1.1

    Name: microsoft.com
    Address: 192.168.1.0
"ooopsie!"
beezle
Looks like somebody made a booboo
WallyFunk
Interesting

https://who.is/dns/microsoft.com

What are the potential ramifications of this?

bennysaurus
Potential timeouts for clients/workstations trying to reach microsoft.com.

Which entry is picked for use is generally random depending on the client.

Most systems will retry using another entry though on issues connecting through. That said, if you are on a network that is 192.168 based, trying to get to Microsoft.com may just send you to your local router!

justin_oaks
I'm trying to figure out how this could have happened, but I control so few IP addresses that many of my DNS entries are manually assigned. And you'd have to be incompetent if you have access to set DNS records and you set them to RFC 1918 addresses.

Anyone have any theories on how this could happen?

efortis
* Copy/Paste

* Copilot told me

* Sabotage (internal or external)

fomine3
my wild idea: counterattack for DDoS
[deleted]
(comment deleted)
whynotmaybe
I'll go with Joseph Conrad on this one.

"It's only those who do nothing that make no mistakes, I suppose."

Now the persons that did it have some proof that they did something.

They will surely put some check in place because there should be another adage somewhere that says that you only learn to use the handrails after you fell in the stairs.

colechristensen
Certainly just some automation bug, perhaps a few things strung together, like a dev environment setup that leaked into production. A human in the loop making a mistake probably as well.

This is the kind of thing you look at and put up a few guardrails to prevent it happening again.

keypusher
My guess is that someone at MS was testing Windows Updates or other changes from a local source. They also had some other DNS updates in their config they were testing. They took all of their config and pushed it out, when they should only have taken the other changes.
fsckboy
I hope that's not their workflow, but few people I've ever worked with have know how to create operating procedures that had half a chance of succeeding, usually it's based on "oh, don't worry, i would never do that"
bruce511
>> And you'd have to be incompetent if you have access to set DNS records and you set them to RFC 1918 addresses.

Clearly the following is not in play for a root domain (Microsoft.com) but assigning a DNS entry to a class C address does have a purpose.

If you have an intranet server, giving it a DNS name allows for HTTPS serving, with an automatic, CA signed, certificate. (Using say LE with DNS challenge.)

I provide this simply as an example of how this might come about.

kotaKat
Through a series of connections I know a guy that knows a guy that works at Microsoft that was made aware and the changes have been reverted. Give 'er 30 minutes TTL ;)
rubyfan
This is my favorite HN comment of 2023.
jgoodknight
This is my second favorite HN comment of 2023
rand1239
HN is a wormhole through which you can connect pretty much anyone in tech industry.
jaza
30 minutes minutes or 30 Windows minutes? :P
wkjagt
Actually, it’s looking more like 6 days. No wait, 30 seconds.
rasengan
This isn’t something that I think should be diluted.

If it’s that simple for a stray record to be included in the dns round robin it could have been bad if it was an external ip with a machine setup by a phisherman especially since control of a domain is all you need to get an ssl cert now.

Couple this with the fact that it’s Microsoft, one of the most relied on companies in our computer world, this is pretty darn horrible.

statictype
For all this to work you need to control the domain. Is that easier than simply breaking into their systems and owning their servers?
water9
Sometimes it’s easier to bribe then to break
Cpoll
That's exactly what they're saying.

> it could have been bad if it was an external ip with a machine setup by a phisherman

I.e. one of the IPs for microsoft.com belongs to $phisher, which means they control (a subset of the traffic going to) the domain. They can't add CNAME records for certificate validation, but LetsEncrypt for example offers HTTP-based validation.

Not sure how Microsoft sets up their certificate pinning, it might not be quite that easy.

rasengan
It might also be a highly targeted attack on someone with precious information wherein someone was able to hack a simple router and in order to get access to their actual microsoft.com account, they simply setup a phisherman's clone on the router and captured the login/password/2fa and got into the account.
notakio
For the Microsoft.com domain, proper, there seem to be no existing CAA rules, allowing each and every CA on earth to issue certificates based on whatever criteria the CA requires. What could possibly go wrong with that approach?
nullbytes
Microsoft also has some of the phishiest looking domains when you are redirected around the O365 cloud.
rvnx
100%. Starting with "onmicrosoft.com". A phisher wouldn't really have to control Microsoft.com to take advantage of confusion.
horusthegame
There were several phishing attempts from that domain, onmicrosoft.com, to my personal email account this past week.
rbanffy
Microsoft.com has been constantly trying to fool me into subscribing to their Office tools.
bombcar
The only thing that competes is the redirecting when you log into any health portal.
varun_ch
Absolutely, I am so glad I'm not the only person who feels this way. Microsoft does not understand domain names.

They use 1drv.ms as the domain in OneDrive emails, and sometimes it almost looks like the .ms ccTLD belongs to them - it very much doesn't, anyone can register a .ms domain.

windows.net being an Azure domain that third parties can have content under is fitting.

It sometimes looks like they want their users to be phished.

Microsoft is a smart company though, I really hope they can sort this out.

mschuster91
Yeah. Easy to spot when your session expires - first they're still using login.live.com, just to redirect you to login.microsoftonline.com.
stkai
I see them both. My TTL will run out at 16:39 PST, though.
petee
TTL appears to be set to an hour. But either way, its been 45 min and the primary ns1-39.azure-dns.com is still offering up 0.1
zenexer
192.168.1.1 is gone now, but all authoritative nameservers are still offering 192.168.1.0. Oops.
adolph
A few are dropping 192.168.1.0 now:

  as of 1703035296:
  ns1-39.azure-dns.com no longer has 192.168.1.0 for microsoft.com
  1.1.1.1 still has 192.168.1.0 for microsoft.com
  8.8.8.8 still has 192.168.1.0 for microsoft.com
  76.76.2.0 no longer has 192.168.1.0 for microsoft.com
  9.9.9.9 still has 192.168.1.0 for microsoft.com
  208.67.222.222 still has 192.168.1.0 for microsoft.com
  185.228.168.9 still has 192.168.1.0 for microsoft.com
  76.76.19.19 still has 192.168.1.0 for microsoft.com
  94.140.14.14 still has 192.168.1.0 for microsoft.com
zenexer
Only one of those is authoritative. All of the authoritative servers have dropped it. Microsoft has fixed the issue.
awill
Yea, sure, but that's like saying "company X stopped selling poison, it's Shop Y's fault that they're still selling it".

Nope. This is still Microsoft's fault while non-auth servers update.

zenexer
First we wanted to see Microsoft update the authoritative nameservers, then move on to monitoring propagation. Conflating the two makes it difficult to monitor whether Microsoft actually fixed it correctly this time, as they appear to have screwed up their first attempt.

I wasn't attempting to address blame since I figured that was obvious enough.

dboreham
Few months early April 1 stunt?
Geezus_42
I know myself through a series of connections as well. ;) ;)
workfromspace
Do we need to restart our Windows machines? :)
awill
that's the solution to all Windows problems, so yes :)
mi_lk
for uninitiated (me), why is it bad?
Racing0461
Since 2 out of the 7 IPs are 192.168 (private ips), 2/7 visitors to microsoft.com will load the private ones assumign equal weight and not get the page to load.
ajb
Well in my case (and a lot of other people), 192.168.1.1 is the local address of my home router. So if I go to microsoft.com I have a 1 in 7 chance of getting my home router instead (if I ignore the certificate warning). Other random breakage will happen depending on what that local address is assigned to for you.

In theory this could be leveraged for hacking, but I think that would require setup in advance.

Zuiii
yep. If a hacker can somehow control 192.168.1.1 or 192.168.0.1 they get access to your microsoft.com cookies at least. I'm sure there are more microsoft specific ways to leverage this too (e.g. data/updates hosted on microsoft.com that misuse HTTPS as a poor man's authentication. The curl | sh crowd are especially susceptible to this problem.)
marshray
They would still need the private key to a https cert that your browser considers valid.
Iwan-Zotow
So if you go to microsoft.com with probability 1/7 you'll hit 1.1 on your private network of 192.168 - likely router, and with probability 1/7 you'll hit 1.0 maybe printer
labster
I don’t know man, putting microsoft.com on your router sounds like a massive reduction in latency. Congrats on the achievement.
edgineer
What could a TLA do with this if it had time to plan ahead?
timschmidt
Serve malicious updates from a locally controlled machine, for one. Lord knows about auth.
wrboyce
Do most DNS forwarders not block addresses that resolve to a local IP these days? I know dnsmasq does, and NextDNS too I think.
drexlspivey
Why? Having local IPs on a public DNS is a legitimate use case.
wolverine876
In fact, some people block domains by routing them to 127.0.0.1 in their host files. I've used private ranges too, in places where loopback might possibly do something funky.
wrboyce
As another reply mentioned, to prevent DNS rebinding attacks. The general expectation is you will whitelist domains from which you expect RFC1918 responses.
charcircuit
Why doesn't windows update use authentication (eg https)?
acdha
They do: the updates are signed so our hypothetical spies would need to have a zero day in Authenticode or to have compromised the signing keys.
wolverine876
> Serve malicious updates from a locally controlled machine. Lord knows about auth.

Wouldn't they have to break into my local machine first, plant an update service, and an update? That doesn't seem to scale well at all, and wouldn't it be easier to just break into the machine they want to 'update'?

timschmidt
A fairly prominent update service already runs from the domain microsoft.com Many machines come with it preinstalled.
wolverine876
The erroneous DNS change wouldn't help that sort of exploit. It just redirects attempts to contact microsoft.com to a local address, probably a router.
timschmidt
That's exactly what I said in the first post.
keyle
Damn you Murphy's law.
_nickwhite
An entry-level admin is now unemployed, just before the holidays.
98codes
Nah, if it's already reverted, they're good to go. A post-mortem with how something like that got through will definitely be on the table though.
taspeotis
The seniors all go on leave and the interns are left to run the place. If they fired the juniors the seniors would have to come back from holiday!
johnnyanmac
I'm wondering how such a change would get "merged" in to begin with. I imagine even non-network engineers would get this huge itch having a large corporate contain a private IP in the changelist (I'm the non network engineer and can't really explain why it's bad. But it FEELS wrong and sometimes you at least need to use instinct to get another pair of eyes on something).
blorenz
I hope not. Failures are on a spectrum and this was unfortunate but probably not malicious. All things considered this should be a lesson learned. There should be more failsafe mechanisms in place so juniors can fail safely and learn from them. The absolute worst thing we can do is shame an individual so they don’t attempt to try new things in fear of ridicule.
wolverine876
> There should be more failsafe mechanisms in place so juniors can fail safely and learn from them.

And if not, whoever put the junior in that role is the person responsible for the problem.

doubloon
well theoretically you could argue the structure of this task should have 'dual control' / multiple people should be involved in the process checking each others work. preferably even split it up people who do not know or interact with each other on a regular basis. yes it would be slower but its important to get it correct.

might as well throw in some automated poke-yoke or whatever too.

in that case there is no fault in any of the juniors or operators, the fault is in management for failing to implement infrastructure to force a critical process to have more than one control

david_shaw
> An entry-level admin is now unemployed, just before the holidays.

I highly doubt that entry-level admins at Microsoft have access to DNS for their primary domain. My guess is that this incident is a lot more interesting than that.

jobs_throwaway
Yep, this doesn't seem like the kind of thing that you can just toss a couple approvals on and change at a company as big as Microsoft. How this made it through the review process would be very interesting
GuB-42
This is most likely a honest mistake. Smart managers don't fire employees for such mistakes unless their behavior regarding that mistake is inappropriate.

As the story goes, after a junior admin wiped a production database. The boss was asked if he should be fired. To what he answered: "Fire him? No way! Not after such an expensive training." Now, he knows.

dvaun
Maybe Sydney tried to breakout…
fouc
Good point. Let's watch Microsoft executives & employees closely for signs of panicking over an escaped AGI.
quickthrower2
For the uninitiated, can some traffic get sent to 192.168.1.1. Is it round robin?
stop50
Yes it is.
YetAnotherNick
I could be wrong but in my experience OS just selects one random and uses that for some time not round robins it.
quickthrower2
Even if that is the case, if it is random, some section of DNS would send traffic to it. Maybe it was OK because most resolvers would ignore the local address on the list??
iameli
Wait wait wait wait. Bunny.net accidentally changed their DNS to 127.0.0.1 and took a bunch of their CDN users down today too. Coincidence? Weird day.
coolspot
Employees mixing up SSH consoles while setting up their smart home Christmas lights over weekend.
[deleted]
(comment deleted)
ragebol
<singing voice="Chris Rea">Becoming 'home' for Christmas </singing>
account42
Probably just developers trying to meet deliveries/targets before the holidays.
Waterluvian
So let me see if I understand. With this DNS record, if me or Windows tries to hit “microsoft.com” there’s a 1/7 chance it hit my router instead?
bastard_op
How the hell did that pass any sort of responsible review process at Microsoft?

Now Microsoft owns all your home networks, only like the default address on every home router out there...

nightfly
Any risk here is nearly the opposite of what you seem to think it is
YetAnotherNick
No, it's that when you open microsoft.com it could open your router page.
plorkyeran
You have the danger of this backwards: this is a very bad security problem for Microsoft, and not a problem for people outside of MS (except to the extent that we're all indirectly reliant on MS being secure). Pointing a domain at an IP address does not give you any power over than IP address, and you can point a domain at anything you want.
sunnybeetroot
No they don’t. Going to Microsoft.com will take you to your router.
riffic
click click click -- "it's done, boss."

Microsoft's mindnumbingly dense ClickOps culture strikes again.

at a serious org this would have involved at least some level of oversight or intervention

[deleted]
(comment deleted)
aaomidi
Y'all, instead of the constant confirmed here. Just do an authoritative lookup.

dig +trace +short microsoft.com

NS a.root-servers.net. from server 100.100.100.100 in 10 ms.

NS b.root-servers.net. from server 100.100.100.100 in 10 ms.

NS c.root-servers.net. from server 100.100.100.100 in 10 ms.

NS d.root-servers.net. from server 100.100.100.100 in 10 ms.

NS e.root-servers.net. from server 100.100.100.100 in 10 ms.

NS f.root-servers.net. from server 100.100.100.100 in 10 ms.

NS g.root-servers.net. from server 100.100.100.100 in 10 ms.

NS h.root-servers.net. from server 100.100.100.100 in 10 ms.

NS i.root-servers.net. from server 100.100.100.100 in 10 ms.

NS j.root-servers.net. from server 100.100.100.100 in 10 ms.

NS k.root-servers.net. from server 100.100.100.100 in 10 ms.

NS l.root-servers.net. from server 100.100.100.100 in 10 ms.

NS m.root-servers.net. from server 100.100.100.100 in 10 ms.

RRSIG NS 8 0 518400 20240101050000 20231219040000 46780 . fG/YHtUJu3YMAm9Mlzzvp3xG4UCPG01aYNnlyF1HfAHdZpR+L88CVUcz NFHq9M45KjB7ZTlSFt2JvEyK/8FcavZLOthkXRREbJQswjLCbhiPQCbq tQLF+tKaNYUihqawCfjgZy1i5YwYjmphbjfzwoKo1POtepf0YCIcuLBi nQFw4Lr79O6cjyg6qlYnqaK6z4Xi5qt6ocohJafjs86LuuRo2WvmJ1IK k0ZUoAC6Qyjz4MVhqHMvQGdp7EnzjoL8Y9PTXeUuD6Ixp/Aklj2psLjD TZDPYN1q+zDd1giFyuwNRX9DG1zrxzN2lzQiLWmGKrzP3DvFWL1L2Ts1 FWjy/Q== from server 100.100.100.100 in 10 ms.

;; UDP setup with 2001:502:7094::30#53(2001:502:7094::30) for microsoft.com failed: network unreachable.

;; UDP setup with 2001:502:7094::30#53(2001:502:7094::30) for microsoft.com failed: network unreachable.

;; UDP setup with 2001:502:7094::30#53(2001:502:7094::30) for microsoft.com failed: network unreachable.

A 20.112.250.133 from server 150.171.10.39 in 20 ms.

A 20.231.239.246 from server 150.171.10.39 in 20 ms.

A 20.76.201.171 from server 150.171.10.39 in 20 ms.

A 20.70.246.20 from server 150.171.10.39 in 20 ms.

A 20.236.44.162 from server 150.171.10.39 in 20 ms.

A 192.168.1.0 from server 150.171.10.39 in 20 ms.

adolph
Or from a bunch of dnses:

  $ export srch="192.168.1.0"; echo "as of $(date '+%s';):"; for dns in 1.1.1.1 8.8.8.8 76.76.2.0 9.9.9.9 208.67.222.222 185.228.168.9 76.76.19.19 94.140.14.14; do dig @${dns} microsoft.com +short | grep "${srch}" > /dev/null; if [  $? == 0  ]; then echo "${dns} still has ${srch} for microsoft.com"; else echo "${dns} no longer has ${srch} for microsoft.com"; fi; done
  as of 1703033639:
  1.1.1.1 still has 192.168.1.0 for microsoft.com
  8.8.8.8 still has 192.168.1.0 for microsoft.com
  76.76.2.0 still has 192.168.1.0 for microsoft.com
  9.9.9.9 still has 192.168.1.0 for microsoft.com
  208.67.222.222 still has 192.168.1.0 for microsoft.com
  185.228.168.9 still has 192.168.1.0 for microsoft.com
  76.76.19.19 still has 192.168.1.0 for microsoft.com
  94.140.14.14 still has 192.168.1.0 for microsoft.com
  $ pbpaste | sed 's;^;  ;' | pbcopy
p1mrx
microsoft.com is currently IPv6-only on my network, because OpenWrt's DNS rebinding protection filters out the A records:

  $ ping -4 microsoft.com
  ping: microsoft.com: Address family for hostname not supported

  $ ping -6 microsoft.com
  PING microsoft.com(2603:1030:c02:8::14 (2603:1030:c02:8::14)) 56 data bytes
  64 bytes from 2603:1030:c02:8::14 (2603:1030:c02:8::14): icmp_seq=1 ttl=112 time=68.4 ms
TacticalCoder
Wait... Can DNS resolvers be configured so that RFC1918 is respected?

I mean: I don't expect anything less from Microsoft than doing stuff like that and it cannot affect me for I nullroute microsoft.com from my unbound server (unboud takes wildcard when nullrouting or NXDOMAINing crap domains like microsoft.com or meta.com etc., which is sweet).

However I'd expect my trusty DNS resolver to also prevent me from anyone not on my private LANs to impersonate addresses reserved for private uses.

Does anyone know here if it's easily doable?

dgl
You're looking for DNS rebinding protection, many DNS servers support it. However there are some cases where things do use private IPs in DNS records outside of the local domain, one example is Plex (e.g. https://support.plex.tv/articles/206225077-how-to-use-secure... suggests turning off DNS rebinding protection) -- although in some cases you can allow particular domains which is a much better way than turning it off entirely.

(See also the sibling comment about microsoft.com being IPv6 only as a result of a particular implementation of DNS rebinding protection: https://news.ycombinator.com/item?id=38704159)

WarOnPrivacy
My Unbound servers strip RFC out.

Public resolvers keep DNS answers intact because they can carry alt data like how dodgy a SMTP server is.

icedchai
I wouldn't expect it to. I have plenty of RFC-1918 addresses in a subdomain of my public DNS zone for my home network. It's been that way for decades. (Perhaps I should use split DNS, but...)
lsago
I was getting an empty answer for microsoft.com. Turns out my dnsmasq is blocking it:

  $ dig microsoft.com. | grep EDE
  ; EDE: 15 (Blocked)

  resolver.log:Dec 20 00:43:57 router dnsmasq[8172]: possible DNS-rebind attack detected: microsoft.com