HN Explorer

Ask HN: Was I pwned? [resolved]

189 points by wasipwned ↗ HN
A few days ago, I noticed that my home network performance would degrade substantially to the point of being unusable. I would just power-cycle all my switches, and the issue would resolve for a while. It happened again this morning, so I decided to try to look closer at what could be causing the issue. That's when I noticed that my Linux desktop was doing a lot of traffic, and here's what I observed:

- My desktop has a private IP address, let's say 10.0.0.2.

- Running `iftop`, I saw all the traffic coming from a different source IP address, 10.0.0.3. It was transferring ~300Mbps.

- Running `tcpdump`, I saw that all of this traffic was going to a public IP address (AT&T). All of the source port/dest were ipsec-nat-t.

- I saw that `10.0.0.3` showed up as a client on my switch with a randomized MAC address (presumably, since I couldn't find the MAC prefix in a vendor list).

- I could not find any references to `10.0.0.3` or the random MAC address on my desktop (looking at kernel logs, system logs, ip a, ifconfig).

- During this period, my network was degraded (high packet loss across my switches).

It was at this point that I decided to try blocking the MAC address from my switch, and performance immediately returned to normal. I tried unblocking the MAC a few minutes later, but it has yet to return. That plus the fact that the issue happens at seemingly random times (especially the middle of the night) makes me think that it's not automatically connecting and instead being triggered remotely.

I've since disconnected my desktop from the network and am in the process of rotating keys. I'm especially perplexed at the traffic showing up from a different source IP on my desktop, but I did not see any interface that matched. I tried to look and see if it was potentially a VM running, but I didn't see anything in virsh. I did have Docker containers running, but I assume I would have seen the IP address show up on one of my interfaces.

I'm at a bit of a loss and was wondering if anyone has ever seen anything like this before, and if there is any suggestions for things I should check.

93 comments

[ 3.2 ms ] story [ 167 ms ] thread
jatin2302
Looks like RAT

Or it could be torrent running in background or some sync services for any storage app.

cyberpunk
I mean. Yes, you're pwned. You need to reflash all possible firmware, dd your disks to zeros and start again.. I wouldn't even trust it then, personally.

What's the IP address it was talking to? Maybe we can help find out what it was?

wasipwned
Yeah. That's definitely the plan, but I want to see if there's anything I can learn from the machine before I even do so.

The IP address was 107.122.31.71.

cyberpunk
It looks like a consumer IP address (AT&T US); ungood. Either your attacker is unsophisticated and they actually had that IP, or they're bouncing through a pwned machine.

Kill it. Kill it with fire.

KomoD
Well, I've dug around, there's no hostname associated with it or pointing to it.

There's no obvious connections to any orgs or sites, and no entries in virustotal or abuseipdb, however there is an open port 179 (looks like BGP??).

Atlas22
Its a carrier grade NAT IP for ATT's cellular service, the BGP is likely just the ATT router.
wasipwned
Yeah, I'm now realizing that this might be multicast traffic I'm seeing from another device, and I do use AT&T which is making me think this may not actually be malicious.
wasipwned
Out of curiosity, how do you know it's CGNAT? Is it just because all of AT&T's mobile traffic is through CGNAT?
redleader55
If you are to go this far, don't stop at dd-ing disks to zero. Replace the SSD controller firmware as well + BIOS + all firmware everywhere.
FeistySkink
Why stop at the firmware? Shred the hardware altogether. /s
Wistar
… and bury it at least 8 feet deep on land no closer than a mile from your residence.
bitwize
Turn off your computer and make sure it powers down,

Drop it in a 35-foot hole in the ground,

Cover it completely, rocks and boulders should be fine...

FeistySkink
Pretty sure you could still recover the flash-based memory. Unless you're dropping into acid or lava.
wpasc
seems to be careless not to burn current residence to the ground as well
guerrilla
I think 6 feet should be enough in most cases, if you're short on time or energy.
yabones
I think we need more information. Do you run any services on that machine that would be exposed? Do you port-forward to that box? Use a VPN or something like Tailscale?

Or perhaps a sync client like syncthing, onedrive, nextcloud, etc. could be to blame.

One option would be to log all traffic on that machine to a .pcap and feed it through some IDS analyzers.

wasipwned
Nothing is exposed directly to the internet, but I had some development services that were accessible on my private network.

I do use Dropbox, but the odd part was it seemingly IPsec traffic.

I really should have grabbed a pcap when it was occurring. I only have a screenshot of tcpdump which is not very useful.

yabones
Was it IPsec traffic, or just something running on the IPsec port?
wasipwned
Unfortunately, I did not save any packet captures, so I only saw that it was on the IPsec port.
CamperBob2
Dropbox would be my #1 suspect. They like to play fast and loose with your networking resources. I have seen Dropbox open lots of weird, undocumented ports for purposes their own support people aren't aware of. Case in point: https://pdfhost.io/v/1C9zgOibj_port_1298_1299_dialogue_with_... (23 kB .PDF)

Kill your Dropbox process(es) -- there will probably be several, again for no clear reason -- and I'll bet this behavior stops. Whenever my system behaves in an unexpected way, I've learned to start the troubleshooting process by temporarily killing Dropbox.

bguebert
Some cell phones will generate a random MAC address. Is 10.0.0.3 given out from your dhcp server? Maybe a phone syncing video to a cloud service.
wasipwned
It was given an IP via DHCP (but not that specific IP, that was more for illustrative purposes).

I'm currently ruling out that it is any other device given I'm seeing the traffic from my desktop, and it shouldn't be acting as a router for another physical device. But I'd like to know if I could be wrong about htat.

tetraca
Do you have a corporate laptop or computer? Things like crowdstrike love to scan your network and phone home.
wasipwned
There are corporate laptops on my network. I had originally thought maybe that could be related. But I can't explain how this source IP was showing up on a tcpdump from my desktop if that was the case, so at this point I'm assuming it's an issue originating from my desktop.
userbinator
Another often-overlooked side of WFH. Devices you don't actually own should be kept on an isolated network away from everything else you do own. I also throttle their bandwidth and disconnect them outside of working hours.

If companies can say "our network, our rules", I'll also do the same with mine.

erdos4d
How is your desktop connected to the switch (ethernet or wifi)? If the computer is wired, maybe you have a virus on it and that is somehow using the wifi to get another IP? I would suggest you backup your user data and wipe/restore the desktop. If it comes back after that, I'd bet someone has cracked your wifi password and is getting in that way, or some other device on the network is the culprit and reinfecting your desktop.
wasipwned
Ethernet, and wifi is disabled. But even if it was wifi or another network interface getting the IP, I should have been able to find it in ip a / ifconfig I'm assuming
erdos4d
Maybe I misunderstood what you wrote, but you feel the traffic is actually originating from your desktop? I am completely unaware how someone could get 2 IP addresses for your desktop without that showing up somewhere on the box (agreed, you should have found it with the tools you ran). I still think wipe and wait, but this seems like you got seriously hacked. Good luck.
deusum
At least block the external IP and ports where the transfers are happening. Change the router password, some neighbor might be in the network.

It sounds like it might be part of a DDoS campaign, as well. Hard to diagnose here.

Kikawala
Do you happen to have a mobile phone with AT&T and are near Fremont, CA?
wasipwned
Yes and yes.
Kikawala
See if the traffic goes away if you disconnect the phone from your wifi. It's most likely AT&T's wifi calling feature.
wasipwned
So interestingly, it looks like Unifi did classify the traffic as wifi calling, but it was doing a lot of traffic in the middle of the night when I was asleep. And the biggest question mark in my head is: how is this traffic looking like it's coming from my desktop?
wasipwned
And by looking it's coming from my desktop, I mean the tcpdump was run directly on my desktop and I saw the traffic there. So I assume it had to be routed through my desktop unless I am missing something.
Kikawala
You can run the below command to see which process or PID is talking over ipsec-nat-t

sudo lsof -n -i :4500

FollowingTheDao
You are missing something. You are seeing BGP routing table updates on your network from an AT&T router. You have BGP running on your network somewhere.
wasipwned
I had run lsof on my desktop and I did not see any of the IP addresses in question. I did not check specifically for port 4500 though.
valleyer
That's often true, but not always. For example, if your desktop is connected to an Ethernet hub, you would usually expect to see all traffic from any machine on the hub. (Ethernet switches are a different story.)

Wi-Fi can work the same way, though it sometimes requires an extra step to place your machine's network card into "promiscuous mode" in order to see traffic neither to or from your machine.

veonik
WiFi calling is multicast, I think, so your phone just basically broadcasts it to the network and every device will see it, though only your router will actually do anything with it.

There's examples of WiFi calling causing this type of issue, described as a packet storm. For example, here's a reddit post with similar symptoms you're describing. https://www.reddit.com/r/networking/comments/3g31mc/iphone_w...

wasipwned
This actually makes the most sense so far. I hadn't even considered the possibility of multicast. Let me see if I can dig in further.
deusum
300mbps for Wifi calling? I'd say more like video streaming/calling.
kevin_nisbet
If a network loop is involved, it could be a small amount of traffic that is getting trapped in the loop. Would need to know way more about the network setup and config though, and I would've thought most cheap home switches would run some sort of spanning tree.
wasipwned
Circling back, this discussion thread seems to be the most likely culprit. In my panicked state, I didn't even consider multicast traffic being the reason why I saw this traffic in tcpdump. I'm digging into this a bit more. I probably wasn't pwned, but I am still currently operating as if I were. I appreciate everyone's response here.
[deleted]
(comment deleted)
kevin_thibedeau
Keep it as a honeypot and run the replacement in a vlaned subnet off the currently owned router.
[deleted]
(comment deleted)
FollowingTheDao
Are any of your switches running a BGP service you do not know about? Could it be trying to send or receive a huge routing table?
itake
Is your router patched? Maybe they hacked your router. Not sure why it would need to assign itself a new IP. Maybe there is a docker container running on the router?
wasipwned
I'm using a UDM Pro, and I had just recently patched. The only container running on the router is unifi-os itself. I keep repeating myself because I want to make sure I can't be missing something, but tcpdump on my desktop is showing the traffic which is why I'm assuming that it's my desktop that is compromised.
FollowingTheDao
> I'm using a UDM Pro, and I had just recently patched.

This is the issue. The patch either turned on BGP or has a BGP bug.

maineldc
Older versions of Unifi controller were subject to mining hacks because of the Log4J compromise. I would check to make sure you are running a recent version of the Unifi Controller.
andrewf
Are you running any virtual machines on your desktop? Because "My machine is 10.0.0.2 but there's something on the same physical host communicating as 10.0.0.3" is what it'd look like if a virtual machine with a "bridged" ethernet interface got its own IP address via DHCP and talked to the Internet.

This is speculation, I don't know whether you were owned.

wasipwned
Yeah, that was exactly what I was thinking as well. I do use KVM to run a few VMs, but they were the only VMs I could find, and they were both stopped the whole time.
guerby
iproute2 things you could look at:

ip ne # show the IP/MAC table

ip rule # show the source routing state

ip netns list # show network namespaces

You could also transfer a trusted "ip" binary from another system in case yours is compromised (kernel could be compromised too)

nonane
> - I saw that `10.0.0.3` showed up as a client on my switch with a randomized MAC address (presumably, since I couldn't find the MAC prefix in a vendor list).

MAC address randomization is enabled by default on iOS: https://www.linksys.com/support-article?articleNum=317709

wasipwned
Thanks for reminding me of this. This is looking less and less malicious at this point as `10.0.0.3` is my phone (using AT&T, which is where all the traffic was destined).
ghostbrainalpha
That still seems like an awful lot of traffic for your phone to be using.

Do you mind sharing what gets you that kind of data usage? Just hours of FaceTime calling or something else?

wasipwned
I can't explain this part yet. I was asleep when this happened, so I wasn't even using my phone. I may be wrong about 300Mbps being to the AT&T. public IP, as my router shows a much lower rate. That might have just been the total traffic I was seeing internally on my private network from multicast.
smoyer
Maybe they've got your phone generating traffic to bill you for going over your limit?
wasipwned
False alarm. Really appreciate everyone helping me sanity check this. The randomized MAC is part of iOS' Wi-Fi privacy, and my phone is using Wi-Fi calling for AT&T. The randomized MAC and the fact that I thought I saw the traffic originating from my desktop (it wasn't, it was just multicast traffic) really threw me off.
CamperBob2
At 300 Mb/sec?
wasipwned
I can't fully explain this part yet, but I am currently expecting that I will see the issue again even with my desktop disconnected.

I am also probably wrong about it being 300Mbps to AT&T. It was probably 300Mbps of multicast traffic internally.

kevin_nisbet
One possibility, although unlikely, is do you have a loop in your network. I think you mentioned multiple switches... does anything have multiple paths on the switching layer to form a loop?
wasipwned
Not to my knowledge, but I will double checking to make sure. The odd part is the issue only started recently, but I haven't made any major changes to my network recently.
kevin_nisbet
Well, the fun thing about network loops, is they can run for years without a problem on a switched network. It's only when you run into a learning issue that switches fall back to flooding all ports except the source port, which then allows packets to flow through the loop indefinitely.

This is why so many people footgun themselves disabling spanning tree, is it seems to work without it... until it doesn't.

AnotherGoodName
Do you have a Chromecast? I had one spamming ARP requests recently. Network reboots didn't fix it but rebooting the chromecast did

Took a loong time and Wireshark for me to find that one.

CamperBob2
Something I wish someone would build is a smarter Wireshark. "Something is hinky on my network. Figure out what it is and who's causing it."

Wireshark is awesome, but the problem is, I only need to use it about once every five years, which means I have to start the learning curve from scratch every single time. I end up following the same basic troubleshooting steps each time, but the process never gets any easier because I never remember what I did the last time.

bedast
My iPad once decided it had 100gb of data to back up and kept trying to back it up to icloud. It was killing my internet.

If you have icloud backups enabled, check the size of the backup.

This seems to be an issue that comes up from time to time on iOS and the only way to clear it is to temporarily disable backups, remove all backups, then re-enable the backup. If that doesn't clear out the oversized backup, you may need to factory reset. I ended up factory resetting my iPad to resolve my issue.

tomxor
I find Apple devices are the worst for disrespecting network use when they "sleep", my partner has a couple and as soon they think no one's looking they saturate the network downloading updates and "syncing" shit. It doesn't help that they started using massive undelta'd partition images to distribute minor updates to their OSs.

They also keep changing stuff on each update to make it harder to force the things to disconnect in sleep mode without a lot of fiddling.. so I've resorted to just adding them to a MAC based access control on the router which we toggle on when we want to use the internet for anything that needs more reliability or bandwidth. I also first tried an aluminium foil based Faraday cage/box to chuck them in with some degree of success but the foil can get breaky.

MarkSweep
You maybe be able to reduce this by turning off “background app refresh”.
tomxor
Maybe, but trying it on the device it's a constant battle, next update it will be something else.. Blocking it's access externally is reliable.
justsomehnguy
Some devices can convert muticast to unicast traffic.

Some devices can respond with errors to the traffic received, but not destined to them.

csw-001
I had a similar situation - but I figured it out pretty quickly because my Firewalla actually warned that device was an iphone in anon mode, and prompted to turn Mac spoofing off if I wanted device-level tracking. I highly recommend Firewalla for home networking issues of this sort, it’s the only thing that gives a relative networking novice like me a fighting chance.
guerrilla
> I'm especially perplexed at the traffic showing up from a different source IP on my desktop, but I did not see any interface that matched

This is easy to do with a raw socket, you just ARP for the IP. See fantaip in Unicornscan for example an example of software that can do that for you. So, all you need is root.

xur17
I experienced a somewhat similar issue yesterday on my network that I described in detail here [0].

Essentially one of the computers (running ubuntu) on my network started sending a VERY high volume (it measured 20gb for the day, and I think it was all over a 10 minute period) of DNS traffic to my router, which runs an unbound instance for my network. That traffic (or at least I think it was that traffic) brought down my network to the point where I could even ping an external or internal ip address.

Does tcpdump show the destination ip address the traffic was sent to on AT&T's network? Curious if that could be a dns server..

Also, what version of ubuntu is your desktop running, and what software does it have on it? Are you using canonical's livepatch service?

[0] https://forum.opnsense.org/index.php?topic=31284.0

wasipwned
Yes I posted the IP address here: https://news.ycombinator.com/item?id=33820749 and it appears to be AT&T's CGNAT IP address and communicating over port 4500 (IPsec), so the likely culprit is Wi-Fi calling which uses IPsec.

I'm running Ubuntu 20.04. I don't use livepatch, but I do update/reboot frequently. I'm mostly running Chrome, Firefox, and Docker. Occasionally GIMP and LibreOffice.

LocalPCGuy
Sounds like you found the issue but for future reference or for anyone stumbling across this later on, another thing to check is network ports in the "cheap" (i.e. generally most < $100) USB/USB-C hubs/port expanders with a power passthrough and a network port.

I had a pretty bizarre experience where it would work just fine during the day while the computer was on, but when I'd shut the lid of my work MacBook, the network port on that little USB-C hub would just start sending off ACK signals like crazy, killing my network for anything else trying to use it (effectively denial of service myself). It was really hard to track down also because it wasn't "traffic" really, and it didn't happen on the devices that were impacted (i.e. I'd be using my Windows PC in the evening and that was attached to my work computer). Even more perplexing because it was semi-random - turned out it wasn't "random", it was when I shut the lid of my work laptop vs. just leaving it up and walking away. I finally saw the flood of traffic by dumping network traffic and was able to trace it back to that hub (first I thought my laptop was pwned and was doing something like exfiltrating data or mining when I wasn't logged in, but it was very definitely the hub after a bit more digging).

Since discovering that, I have come across others that have written up the same or similar issues. With the power passthrough, the hub still has power, and if the network interface is flaky as many are, it can cause issues, particularly when the machine it's plugged into stops using it.

This post has links to a few various write-ups: https://mjtsai.com/blog/2022/05/11/usb-c-hubs-breaking-ether...

harel
Even though this was a false alarm in the end, the processes taken to investigate this merit an upvote and a save for future reference.
[deleted]
(comment deleted)