HN Explorer

Every day at the same time, my internet dies for 1 minute. How do I investigate?

308 points by overallorder ↗ HN
3:40pm every day my wifi loses internet access. My devices remain connected to the network, but all traffic dies. Almost exactly 1 minute later everything is resumed. I have no idea what the cause could be.

How could I begin investigating this? I have a spare Raspberry Pi and and old Android phone at my disposal, and some programming competency.

140 comments

[ 21.7 ms ] story [ 1162 ms ] thread
niklasmtj
Did you check if your public IP changes after this minute? I don't know if this still happens but back in the past my router did this every 24h to get a new IP.
njsubedi
I can confirm this. The time is around 7:00PM and the IP address changes, and I have to log in again to some websites because my IP has changed. Downtime is approximately 2 minutes.
pmiller2
This was my thought as well. It sounds like a 24 hour DHCP lease expiring.
rndgermandude
A lot of service providers do this because they do not want you to host static services from your "dynamic" home IP, as they are selling "business" lines with dedicated static IPs. Or at least they used to do this for that reason. By now it's just some thing a lot of ISP do. I have seen other possible explanations like ISP claiming this is a "security" feature because attackers cannot have permanent access without always learning the new IP, which is supposedly somehow hard. This is pretty bull tho. If attackers have access to your system they could just run software signalling them back any IP changes. And even nmapping the entire IP space of an ISP to find a service you were hitting again is very feasible.

Anyway, if the ISP does this, then you have to reconnect once every day (or sometimes every two days), and if you don't do so manually they will just cut your connection on their side and make you reconnect.

My (German) ISP does that too, the one I had before did it, the one before did it as well.

znpy
If you can replace the ISP provided router with your own you should be able to renew the lease.
olmideso
If your devices are kept connected this means that your wifi and router are still active, probably the issue somewhere behind your router, either a connection or provider. But due to exact timing it's likely something on the provider's side and not just some connection issue. I would personally start with contacting them to investigate.

You might also check the router's logs for that time, perhaps there's something useful information.

uncledave
Try and get hold of your Internet/WiFi router's logs. The answer will probably be in there.
elliekelly
Do you have xfinity by any chance? And are you using their x1 combination modem/router in bridge mode with custom (non-comcast) DNS settings?

Because the exact same thing was happening to me at the exact same time. I’m no networking expert but when I checked the logs it seemed that every day Comcast would try to “fix” my (not broken) DNS settings. I had been using mullvad DNS and open DNS as backup but since switching to nextDNS the issue has stopped.

I don’t know enough about networking or DNS to offer any explanation as to why.

dpedu
I have xfinity configured with custom DNS and used to have a somewhat similar problem. At the same time each day, DNS would stop working (but I could still reach plain IPs) for a couple minutes.

It went away after I called them and had them "de-provision" and "re-provision" my modem. Basically deleting it from their system and readding it. For whatever reason, whenever I'm having chronic comcast weirdness, having them do this solves it. It sucks that this is the way things are.

dickfickling
MY INTERNET HAS BEEN GOING OUT AT 3:40pm PT TOO!

Sorry for the all caps. I just scrolled through my texts and saw the “internet out?” texts are at the same time.

Hopefully that means it’s a provider issue behind us. Wonder what we have in common?

- South Lake Tahoe, CA

- Spectrum Gig internet

- Asus Zenwifi mesh

fisherjeff
Hmmm my Spectrum Gig in OR doesn’t go out at 3:40p but it does pretty much every day at 1pm. I was similarly excited to see this post...
onionisafruit
In my neighborhood it’s shortly after midnight. Or so I’ve heard from my kids and neighbors. I’m never awake to notice.

Also Spectrum, but no gigabit here.

beaugunderson
Spectrum in Huntington Beach, CA; ours does go out for one minute during the day as well though I didn't think to get systematic and note the times.
wasdfff
My spectrum cuts out pretty reliably at 12:30am every night. Lets me know its time for bed.
fisherjeff
Now that, I could get behind: “Attention Spectrum user, cease doomscrolling at once and get some sleep”
[deleted]
(comment deleted)
amelius
Get better hardware and software that properly reports error messages instead of hiding them.
derwiki
Disconnect your wifi gateway and plug a computer directly in to your modem. Eliminate or prove the router as the problem source.
xgbi
Launch ´mtr google.com’ on one of your devices (preferably connected via Ethernet to your router) a few minutes before the cut off, and watch what happens when the cut occurs. It will show you exactly which equipment on the route starts dropping.

Although I suspect your ISP has a 24h lease and your modem renegociates at 3:40 everyday

wjnc
So then the trick is to forgo internet by disconnecting power from a few minutes before that time to the middle of the night and get power back? I do feel for the ISP's with this setup because the right thing to do would be to write in the installation manual: "Only install in the deep of night".
milankragujevic
Well they could 1) renew the IP before it expires or 2) reboot the modem remotely at 3:30am for example.
hsbauauvhabzb
Or assign a lease that expires at 0330 instead of a lease that expires in 24 hours.

But I’ve not experienced similar in 20 years of dynamic ips.

rantwasp
it depends. depending on the dhcp implementation on the server side you may have the same problem (ie while you get a new lease it may expire at the same time). unlikely but I’ve seen some stuff when it comes to dhcp
throw0101a
Actually it seems that it is not renegotiating everyday: it is expiring and re-requesting.

The way DHCP 'should' work is that when a client is part-way through the lease it should try to renew the IP it already has:

* https://en.wikipedia.org/wiki/Dynamic_Host_Configuration_Pro...

* https://tools.ietf.org/html/rfc2131#section-4.4.5

This allows the client to (hopefully) keep extending the IP it has so that a re-IP does not cause current connections to drop. If the modem is not doing this (can the OP log into it in someway to see logs?), then it is acting as a 'non-ideal' DHCP client.

Until this is sorted out, try rebooting the modem an an 'odd' hour that will not disrupt you during the day. This is no guarantee though: the DHCP server may remember the old/current DHCP lease and simply re-issue it with the same expiration time.

I'm on DSL/PPPoE, so this may not apply, but: my Asus router has a setting that allows it to automatically reboot. I do this at ~04h00 to get a new IP every day so help with privacy concerns. I generally surf with cookies disabled, so these two things help with the low-hanging fruit of simple tracking techniques. (My DSL modem is bridged.)

dayjah
I had this exact issue in 2008 in SF on Comcast Business.

The lease would be dropped and renegotiated every few hours, we could notice it because our SSH tunnels would get torn down.

What was annoying was that the business plan had a static IP associated with it, presumably managed by their DHCP setup.

It took weeks to convince Comcast the issue wasn’t in our building, and moments for them to fix once they “got it”.

patneedham
What piece of evidence or logs finally convinced Comcast that the issue was on their side?
dayjah
Honestly my memory is not that good! I remember a few tech visits as they replaced modems, etc, and I eventually got to some “3rd level support person” that said something akin to “oh, that issue again”.
midasuni
Did you try saying shibboleet?

https://xkcd.com/806/

oxfeed65261
c.f. Judges 12:5-6 (https://www.bible.com/bible/114/JDG.12.5-6.NKJV):

> The Gileadites seized the fords of the Jordan before the Ephraimites arrived. And when any Ephraimite who escaped said, “Let me cross over,” the men of Gilead would say to him, “Are you an Ephraimite?” If he said, “No,” then they would say to him, “Then say, ‘Shibboleth’!” And he would say, “Sibboleth,” for he could not pronounce it right. Then they would take him and kill him at the fords of the Jordan. There fell at that time forty-two thousand Ephraimites.

barbarbar
Haha - This is and the comment above are awesome. But not really a backdoor.
landemva
I saw this daily drop on several small biz Comcast setups. I recall I found it while looking through all the settings on the router, though I do recall having to call Comcast and they set something so it never happened again. It was the Comcast device with built in wifi. Hardwired didn't glitch, just wifi.
torwayburger
I could be wrong but this sounds more like NAT port issue given the frequency and static lease agreement on IP.
runjake
This is the correct answer. Although typically leases will attempt a renewal halfway through the lease time, so it would be a 48 hour lease.

As OP said, mtr will tell you where in the line it’s happening.

KronisLV
Here's a summary from the man pages of mtr, for those who are not familiar with the tool but maybe are reading HN on their phones so can't check it out:

    mtr combines the functionality of the traceroute and ping programs in a single network diagnostic tool.
    
    As mtr starts, it investigates the network connection between the host mtr runs on and HOSTNAME by sending packets with purposely low TTLs. It continues to send packets with low TTL, noting the response time of the intervening routers. This allows mtr to print the response percentage and response times of the internet route to HOSTNAME. A sudden increase in packet loss or response time is often an indication of a bad (or simply overloaded) link.
    
    The results are usually reported as round-trip-response times in milliseconds and the percentage of packetloss.
This actually makes me wonder about whether there's a command that'll let me see short summaries of what every file under /usr/bin is, in the form of a list. Definitely wasn't aware of mtr as an average web developer up until now.
lambdaba
apropos -s 1 . will list a short description from the manpages (might have to run mandb first)
ksec
On the subject of Lease Time.

My router is currently showing a lease time of 3 Min. I think most router tends to renew their IP at 50% of ISP's lease time ( in this case 6 min )

What are the purpose of these ridiculously low lease time? I remember in the old days they tend to go for 48 hours if not longer.

igetspam
To get you to pay more money for a static IP.
ksec
Seems Reasonable, but I though most ISP dont want Home users to be using Static IPs.

And apology for asking an off topic question. I guess that is what the downvotes are coming from.

wernercd
There's a limited number of IPs so most homes CANT use static... we already don't have enough IPs for the mountains of devices.

But the other part is money... why give something away for free when you can make it a "feature" to pay for?

Tons of other reasons in various articles...

https://www.online-tech-tips.com/computer-tips/ott-explains-...

https://superuser.com/questions/590391/why-do-isps-change-yo...

eppp
I work at an ISP and setup both static and dhcp services. Statics take far more work and management than dhcp, ignoring the scarcity problem. We make you pay more because you are requesting something that requires both documentation and management on a home connection.

Id be glad to give them away for free but it is all downside for us and over 99% of residential customers dont want one to start with.

I think we have 3 residential customers out of 1500 that have statics.

wernercd
After having read some of the articles on it (including the two listed), the mechanics are something that are probably a lot more intensive to manage than I figured initially. Changing DHCP is a couple changes and then... wait. Changing hard coded paths and all that...

I'm actually glad I don't deal with that stuff on a day to day... but I like learning about it.

m463
un-ignoring the scarcity problem, when I've gotten a static IP I would usually get a block of 8 and they would waste the 0 for network and 7 for broadcast. I've wondered why they didn't just give you 8 usable addresses.

(or if I set up my own router could I just use the all?)

nikau
Most ISPs are using carrier grade NAT (CGNAT), so most users are sharing IPs with other customers.
JeremyNT
Is there a citation for the "most" here? I understand this is true in some cases, but I've never used an ISP that does carrier grade NAT. I'm on Spectrum cable now, which is not exactly known for being a good ISP, and I have my own address.
bobowzki
This could also be a power or electrical interference issue. I would start by logging the connection with ping every 1 sec or so to see if it's only 3:40pm.
ayewo
1. Start the ping command at 3:39pm and leave it running until 3:41pm then make a note of whether packet losses start at 3:40pm. (E.g. "ping yahoo.com" on Unix or "ping -t yahoo.com" if on Windows).

2. If there are packet losses, then you have your answer: the issue is from your ISP.

3. If there are no packet losses, then you'll need to look closer at your network. Check to see if some hardware on your network might be performing a reboot at the 3:40pm mark.

kratom_sandwich
Are you familiar with the "Reply all" podcast? They have a "recurring segment, called 'Super Tech Support' [in] which [...] the Reply All team [...] takes on odd or especially complex tech support issues that the listeners or friends of the hosts have encountered." Not sure if your situation qualifies, though.

[1] https://en.wikipedia.org/wiki/Reply_All_(podcast)

davidro
Change your DNS servers.. I use openDNS, but there are lots of other choices. If the Pi isn’t in use set it up as a Pi-hole (DNS server / ad blocker), so you’re not use your isp provided router for DNS.
netsharc
DNS probably has nothing to do with disconnects, this kind of random suggestion unrelated to the problem always makes me think of the Futurama Robot Judge: https://www.youtube.com/watch?v=Jqn8HB2w2Fs (skip to 0:15 for the punchline)

Or maybe it is related, in which case I'll eat at a hat.

mackatsol
I've had outages that were due to my ISP's DNS service going down... one of the reasons I started using OpenDNS.

I need to watch more Futurama. That was great!

detaro
Sounds like standard reconnect every 24h, many ISPs do that. reset your modem at a weird time to get the interruption at that weird time in the future.
bjeds
This.

I think the OP is so eager to get back online he sees 3:40 every day and is using that timepoint as his troubleshooting clue.

However If he were to disconnect 3:10 and then just be offline an hour until plugging the device back he would probably see the reset happen 4:10 going forward.

corysama
Does your WiFi provide a way to monitor the real time bandwidth usage of each device? I was experiencing randomly timed outages for a while. Turned out to be our various phones and other devices randomly saturating my low upload bandwidth with cloud backups. That shouldn’t kill my downstream bandwidth, but it does.
unilynx
It doesn't kill your downstream directly, but you can't send the acknowledgements for the download fast enough if the upload is saturated. That kills your download speed
milankragujevic
@OP did you try calling support to report the problem? It's almost impossible to diagnose ANY internet problems (not related to your own equipment of course) and the ISP has tools and access to do it. In any case, it's much easier and faster to contact support.

In my experience, even if I discover what is the problem, if it's on their side I still have to go through support to get it fixed, so I just wasted my time. And in most cases the lowest level of support doesn't even write down the information I give in the ticket, even if I send an email.

princevegeta89
Wow I have the same problem here. In my case I found out it is at&t themselves.
the8472
Some german ISPs used to force a reset of home internet connections every 24h to trigger IP changes so that static IPs would remain reserved for the business tier. The time of the reset could be shifted by preemptively reconnecting at a desired time, they only reset the link after 24h of actual uptime.
0xTJ
I get why they do it, but actually causing issues just to cycle IPs seems like a lot. I've had cases where I have the same IP assigned to me for quite a long time.
aeyes
They already did this on dialup and ISDN, back when routers didn't exist. When DSL first started appearing they just kept doing it. The idea was to not keep unused connections/IPs up, since you didn't have a router you would need to reconnect manually.

Also, it made it harder to run servers until we had services like DynDNS.

On newer connections it isn't common anymore, especially if you have a voice service.

Lev1a
On my family's FritzBox we can set the time for that connection reset. I think it's somewhere around 04:00.
cvandebroek
+1 on @detaro's message. Sounds like a 24h disconnect. Your router may have a feature to set dis/connect times manually ir you unplug/replug it at the time you wish to have the dis/connect to happen.
nicoffeine
Make sure all your network devices / computers are synced via NTP, or at least accurate to the second.

Hardwire the Pi to the network. Start a verbose traceroute and ping loop on a device on WiFi, as well as the Pi. Once it fails, observe the lights on your router/wifi device(s). Note time to the second of the start and finish of the outage, and the behavior of the lights. Check the results of your traceroutes/pings. Start digging through the logs of each network device.

Also, you can never quite rule out power, even if the lights stay on. A power conditioning UPS is useful here.

scotty79
It was easy way of dealing with router hanging up due to accumulated problems used by amateur internet providers. Just disconnect it from power once a day.

You can do that with cheap socket timer.

Not sure why would anyone set it to the middle of the day. Maybe error in setting times or clock or it was set intentionally so that you can service it at normal hours if it doesn't come back on.

The problem is you can put in 'off' time and 'on' time but they have to be at least 1 minute apart.

oger
"My internet" is a bit vague - so I am making some assumptions here. Scenario (A) could be that your WiFi dies at a specific point in time. You might then want to look at some source of electromagnetic interference (like a motor starting while creating sparks). Scenario (B) could be that your internet service provider is cutting your connection and provides a new IP address - which is common for many xDSL providers. You want to check if the external IP address of your router changes. You may also find some log files in your router that may give you more information. Some routers allow you to select the time when the router preemptively disconnects at a certain time in order to avoid the disconnect from the ISP at an arbitrary moment. You may want change this setting or to talk to your ISP and try to convince him to not cut the connection during the day but rather during the night. Scenario (C) could be a device on your local network creating interference. Your tools of choice will be tcpdump and WireShark to locate the culprit on your LAN. Enjoy your debugging and good luck!
2rsf
You don't need something fancy like motor sparks to kill a 2.4Ghz WiFi, a leaky Microwave will do that and perfectly explain the 1 minute length.

Switching to 5Ghz will help if this is the case

tyingq
At 3:39p, unplug your router and leave it unplugged for an hour, then plug it back in. If the outage then moves to 4:40p every day, it's a 24 hour IP address lease expiring.
spuz
Are you talking about IP leases from the router to the local network or IP leases from the ISP to the router?
mvolfik
likely router -> ISP - if it was the devices, they could show up as not being connected to the network. also weird that all devices' leases would expire at the same time
ourcat
Have you tried different WiFi channels?

I had a similar experience to this years ago while working at my mother's kitchen table for while. The network would keep going dead and I had no idea what was causing it.

Eventually, I discovered that it happened every time she used the microwave to heat up a drink or something.

Changing to a different WiFi channel fixed it. :)

bonestamp2
I came to suggest the microwave too. The 24h IP reset sounds more likely, but don't discount the microwave, they typically use the 2.4 GHz frequency (as do bluetooth and some wifi networks of course). So, if the 24h IP reset isn't it, try switching to a 5 GHz wifi network to see if the same dropout occurs. If that solves it, it could be a nearby microwave oven.
rsyring
Step 1: reboot all devices on the network: computers, router, cable modem (or whatever equivalent device your router plugs into).

If that doesn't fix it, below is how I'd proceed.

Use the trace route networking tool (google it, different syntax on different OS) to some known good public IP address. I usually use 1.1.1.1 or 8.8.8.8.

This tool shows you the path your traffic takes:

    $ traceroute 1.1.1.1
    traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 60 byte packets
    1  192.168.1.1 (192.168.1.1)  1.286 ms  1.386 ms  1.963 ms
    2  192.168.11.1 (192.168.11.1)  2.591 ms  4.342 ms  5.219 ms
    ...snip...
    16  0.ae19.GW8.CHI13.ALTER.NET (140.222.230.223)  42.888 ms  35.742 ms 0.ae20.GW8.CHI13.ALTER.NET (140.222.230.225)  41.060 ms
    17  152.179.105.202 (152.179.105.202)  51.913 ms  50.681 ms  55.442 ms
    18  one.one.one.one (1.1.1.1)  70.087 ms  65.130 ms  53.607 ms
The snippet above shows that I have two local devices (192.168.1.1, 192.168.11.1) my traffic passes through and 18 devices total before my traffic hits the public IP.

Run this command at 3:39. Then run it again at 3:40. Then again at 3:41. Compare results and that should give you a good idea which device is malfunctioning at that time.

Some routers also have a ping command baked into them. If you can make it constantly ping 8.8.8.8, while you run the test above, it may help you. In the case where your traceroute stops at your own router, you can look at the ping from the router itself, and see if it also stopped:

A) If the traceroute stops at your router, but the router ping is not interrupted, then something in your router is flaking out.

B) If the router ping is also interrupted, then the problem is "upstream" from you. Maybe the cable modem (or whatever device) that is upstream from your router.

If you decide the problem is upstream, contact your ISP. If it's local, investigate and/or replace the flaky device.

HTH.