NHacker Next
  • new
  • past
  • show
  • ask
  • show
  • jobs
  • submit
Every day at the same time, my internet dies for 1 minute. How do I investigate?
wruza 1186 days ago [-]
Your ISP does that. Apart from IP rotation theory, I heard that some ISP billing software requires a session reset to flush network statistics to a billing database, because it is accumulated in RAM. They have to do that to block/divert clients whose account reaches a limit, because you don’t normally upload payment info to your firewall/proxy and only check it for new sessions. (I’m not an ISP guy and this may simply be an urban legend from 90's)

My experience partially confirms that. I had an ISP who would redirect every syn packet on 80/443 to internal “low money” site in the night, but I had to reset my router after sending them money, cause otherwise that crippled session got stuck at their hardware until next autoreset.

1 minute may be a technical pppoe cleanup/reconnect timeout. Some shitty ISPs reset this timeout even for unsuccesful attempts, and this deadlock can last for half an hour until e.g. a router decides to back off for a while. Could be cured by turning it off for 3-5 minutes to cool off the ISP side. It’s rare these days.

samoa42 1185 days ago [-]
> 1 minute may be a technical pppoe cleanup/reconnect timeout.

i'd second this guess because its a common setup. OP could check the external ip before/after the reconnection and verify this way if it changed.

would also be helpful to know connection technology (dsl/cable/wifi/fibre..) and name of the isp

candiodari 1185 days ago [-]
If you want to confirm this, if you can pull the logs from your router you should be able to see this happen and getting log entries around that time.
edrobap 1185 days ago [-]
I also suspect that everyone has a regular reset interval. I had faced similar issue with cellular networks. I was working on sending accelerometer readings every few seconds for an earthquake detection project. While I was doing that, the Internet connection would stop working every night at 10pm and it required restart of the connection. There was no option but to write a daemon script that’d check health of the Internet connection and start the application required to establish connection again.
mikewarot 1186 days ago [-]
1. Unplug your Router at some other time, for at least 5 minutes, long enough for your ISP to notice it's absence and give you a different IP address. If your 3:40 PM changes, it's the 24 hour expiration of your external IP.

2. Do a ping 8.8.8.8 from your Linux machine (Raspberry Pi) or ping -t 8.8.8.8 from windows, and watch what happens at 3:40 PM

3. As others have said, turn off uPnP

Forbo 1186 days ago [-]
This assumes that the ISP has a short lived DHCP lease. Comcast Business once tried to charge me to roll my DHCP IP from them. They told me I would have to disconnect my equipment for three weeks for the lease to expire. I told them to go fuck themselves, got a minicipal fiber connection, and never looked back.
Triv888 1186 days ago [-]
With Comcast, if I spoof my router's MAC address, I get a new IP instantly... otherwise it is basically static (residential service).
m463 1185 days ago [-]
So does that mean they somehow detect the mac address has come from a new device and roll the ip address to prevent people from doing server kinds of things?
Triv888 1185 days ago [-]
I host servers on my home connection and never had issues (but I do use a dynamic dns service to make sure that I always have the correct IP)
_underfl0w_ 1186 days ago [-]
Must be nice! I wonder if OP even has the ability to switch ISPs (here in Texas, limited options)
joshspankit 1186 days ago [-]
A) There are a lot of potential hiccups with trying to get a new IP:

- Some ISPs assign statically (odd, but true) meaning you will always get the same public IP

- The DHCP lease time is different for many ISPs: 5min to weeks, meaning you’d have to be offline for at least that long in order to be assigned a new IP when connecting

- Some ISPs lease it to the modem and not the router or anything internal (does not apply to what you said, but another tactic: changing the MAC address)

B) OP asked how to investigate. If getting a new IP dodges the issue and it doesn’t happen again, then there’s basically zero chance that the actual cause will be known

scook 1186 days ago [-]
Consumer ISPs don't always refresh IP addresses like that. I have had the same IP address on my home internet for years and was told that I would need to purchase a block of static IPs in order to recieve a new one.
xgbi 1186 days ago [-]
Launch ´mtr google.com’ on one of your devices (preferably connected via Ethernet to your router) a few minutes before the cut off, and watch what happens when the cut occurs. It will show you exactly which equipment on the route starts dropping.

Although I suspect your ISP has a 24h lease and your modem renegociates at 3:40 everyday

KronisLV 1186 days ago [-]
Here's a summary from the man pages of mtr, for those who are not familiar with the tool but maybe are reading HN on their phones so can't check it out:

    mtr combines the functionality of the traceroute and ping programs in a single network diagnostic tool.
    
    As mtr starts, it investigates the network connection between the host mtr runs on and HOSTNAME by sending packets with purposely low TTLs. It continues to send packets with low TTL, noting the response time of the intervening routers. This allows mtr to print the response percentage and response times of the internet route to HOSTNAME. A sudden increase in packet loss or response time is often an indication of a bad (or simply overloaded) link.
    
    The results are usually reported as round-trip-response times in milliseconds and the percentage of packetloss.
This actually makes me wonder about whether there's a command that'll let me see short summaries of what every file under /usr/bin is, in the form of a list. Definitely wasn't aware of mtr as an average web developer up until now.
lambdaba 1186 days ago [-]
apropos -s 1 . will list a short description from the manpages (might have to run mandb first)
throw0101a 1186 days ago [-]
Actually it seems that it is not renegotiating everyday: it is expiring and re-requesting.

The way DHCP 'should' work is that when a client is part-way through the lease it should try to renew the IP it already has:

* https://en.wikipedia.org/wiki/Dynamic_Host_Configuration_Pro...

* https://tools.ietf.org/html/rfc2131#section-4.4.5

This allows the client to (hopefully) keep extending the IP it has so that a re-IP does not cause current connections to drop. If the modem is not doing this (can the OP log into it in someway to see logs?), then it is acting as a 'non-ideal' DHCP client.

Until this is sorted out, try rebooting the modem an an 'odd' hour that will not disrupt you during the day. This is no guarantee though: the DHCP server may remember the old/current DHCP lease and simply re-issue it with the same expiration time.

I'm on DSL/PPPoE, so this may not apply, but: my Asus router has a setting that allows it to automatically reboot. I do this at ~04h00 to get a new IP every day so help with privacy concerns. I generally surf with cookies disabled, so these two things help with the low-hanging fruit of simple tracking techniques. (My DSL modem is bridged.)

dayjah 1186 days ago [-]
I had this exact issue in 2008 in SF on Comcast Business.

The lease would be dropped and renegotiated every few hours, we could notice it because our SSH tunnels would get torn down.

What was annoying was that the business plan had a static IP associated with it, presumably managed by their DHCP setup.

It took weeks to convince Comcast the issue wasn’t in our building, and moments for them to fix once they “got it”.

landemva 1186 days ago [-]
I saw this daily drop on several small biz Comcast setups. I recall I found it while looking through all the settings on the router, though I do recall having to call Comcast and they set something so it never happened again. It was the Comcast device with built in wifi. Hardwired didn't glitch, just wifi.
patneedham 1186 days ago [-]
What piece of evidence or logs finally convinced Comcast that the issue was on their side?
dayjah 1186 days ago [-]
Honestly my memory is not that good! I remember a few tech visits as they replaced modems, etc, and I eventually got to some “3rd level support person” that said something akin to “oh, that issue again”.
midasuni 1186 days ago [-]
Did you try saying shibboleet?

https://xkcd.com/806/

oxfeed65261 1186 days ago [-]
c.f. Judges 12:5-6 (https://www.bible.com/bible/114/JDG.12.5-6.NKJV):

> The Gileadites seized the fords of the Jordan before the Ephraimites arrived. And when any Ephraimite who escaped said, “Let me cross over,” the men of Gilead would say to him, “Are you an Ephraimite?” If he said, “No,” then they would say to him, “Then say, ‘Shibboleth’!” And he would say, “Sibboleth,” for he could not pronounce it right. Then they would take him and kill him at the fords of the Jordan. There fell at that time forty-two thousand Ephraimites.

barbarbar 1185 days ago [-]
Haha - This is and the comment above are awesome. But not really a backdoor.
torwayburger 1186 days ago [-]
I could be wrong but this sounds more like NAT port issue given the frequency and static lease agreement on IP.
wjnc 1186 days ago [-]
So then the trick is to forgo internet by disconnecting power from a few minutes before that time to the middle of the night and get power back? I do feel for the ISP's with this setup because the right thing to do would be to write in the installation manual: "Only install in the deep of night".
milankragujevic 1186 days ago [-]
Well they could 1) renew the IP before it expires or 2) reboot the modem remotely at 3:30am for example.
hsbauauvhabzb 1186 days ago [-]
Or assign a lease that expires at 0330 instead of a lease that expires in 24 hours.

But I’ve not experienced similar in 20 years of dynamic ips.

rantwasp 1186 days ago [-]
it depends. depending on the dhcp implementation on the server side you may have the same problem (ie while you get a new lease it may expire at the same time). unlikely but I’ve seen some stuff when it comes to dhcp
ksec 1186 days ago [-]
On the subject of Lease Time.

My router is currently showing a lease time of 3 Min. I think most router tends to renew their IP at 50% of ISP's lease time ( in this case 6 min )

What are the purpose of these ridiculously low lease time? I remember in the old days they tend to go for 48 hours if not longer.

igetspam 1186 days ago [-]
To get you to pay more money for a static IP.
ksec 1186 days ago [-]
Seems Reasonable, but I though most ISP dont want Home users to be using Static IPs.

And apology for asking an off topic question. I guess that is what the downvotes are coming from.

wernercd 1186 days ago [-]
There's a limited number of IPs so most homes CANT use static... we already don't have enough IPs for the mountains of devices.

But the other part is money... why give something away for free when you can make it a "feature" to pay for?

Tons of other reasons in various articles...

https://www.online-tech-tips.com/computer-tips/ott-explains-...

https://superuser.com/questions/590391/why-do-isps-change-yo...

eppp 1186 days ago [-]
I work at an ISP and setup both static and dhcp services. Statics take far more work and management than dhcp, ignoring the scarcity problem. We make you pay more because you are requesting something that requires both documentation and management on a home connection.

Id be glad to give them away for free but it is all downside for us and over 99% of residential customers dont want one to start with.

I think we have 3 residential customers out of 1500 that have statics.

wernercd 1185 days ago [-]
After having read some of the articles on it (including the two listed), the mechanics are something that are probably a lot more intensive to manage than I figured initially. Changing DHCP is a couple changes and then... wait. Changing hard coded paths and all that...

I'm actually glad I don't deal with that stuff on a day to day... but I like learning about it.

m463 1185 days ago [-]
un-ignoring the scarcity problem, when I've gotten a static IP I would usually get a block of 8 and they would waste the 0 for network and 7 for broadcast. I've wondered why they didn't just give you 8 usable addresses.

(or if I set up my own router could I just use the all?)

nikau 1185 days ago [-]
Most ISPs are using carrier grade NAT (CGNAT), so most users are sharing IPs with other customers.
JeremyNT 1184 days ago [-]
Is there a citation for the "most" here? I understand this is true in some cases, but I've never used an ISP that does carrier grade NAT. I'm on Spectrum cable now, which is not exactly known for being a good ISP, and I have my own address.
runjake 1186 days ago [-]
This is the correct answer. Although typically leases will attempt a renewal halfway through the lease time, so it would be a 48 hour lease.

As OP said, mtr will tell you where in the line it’s happening.

dickfickling 1186 days ago [-]
MY INTERNET HAS BEEN GOING OUT AT 3:40pm PT TOO!

Sorry for the all caps. I just scrolled through my texts and saw the “internet out?” texts are at the same time.

Hopefully that means it’s a provider issue behind us. Wonder what we have in common?

- South Lake Tahoe, CA

- Spectrum Gig internet

- Asus Zenwifi mesh

fisherjeff 1186 days ago [-]
Hmmm my Spectrum Gig in OR doesn’t go out at 3:40p but it does pretty much every day at 1pm. I was similarly excited to see this post...
onionisafruit 1186 days ago [-]
In my neighborhood it’s shortly after midnight. Or so I’ve heard from my kids and neighbors. I’m never awake to notice.

Also Spectrum, but no gigabit here.

wasdfff 1186 days ago [-]
My spectrum cuts out pretty reliably at 12:30am every night. Lets me know its time for bed.
fisherjeff 1186 days ago [-]
Now that, I could get behind: “Attention Spectrum user, cease doomscrolling at once and get some sleep”
beaugunderson 1186 days ago [-]
Spectrum in Huntington Beach, CA; ours does go out for one minute during the day as well though I didn't think to get systematic and note the times.
1186 days ago [-]
elliekelly 1186 days ago [-]
Do you have xfinity by any chance? And are you using their x1 combination modem/router in bridge mode with custom (non-comcast) DNS settings?

Because the exact same thing was happening to me at the exact same time. I’m no networking expert but when I checked the logs it seemed that every day Comcast would try to “fix” my (not broken) DNS settings. I had been using mullvad DNS and open DNS as backup but since switching to nextDNS the issue has stopped.

I don’t know enough about networking or DNS to offer any explanation as to why.

dpedu 1186 days ago [-]
I have xfinity configured with custom DNS and used to have a somewhat similar problem. At the same time each day, DNS would stop working (but I could still reach plain IPs) for a couple minutes.

It went away after I called them and had them "de-provision" and "re-provision" my modem. Basically deleting it from their system and readding it. For whatever reason, whenever I'm having chronic comcast weirdness, having them do this solves it. It sucks that this is the way things are.

kratom_sandwich 1186 days ago [-]
Are you familiar with the "Reply all" podcast? They have a "recurring segment, called 'Super Tech Support' [in] which [...] the Reply All team [...] takes on odd or especially complex tech support issues that the listeners or friends of the hosts have encountered." Not sure if your situation qualifies, though.

[1] https://en.wikipedia.org/wiki/Reply_All_(podcast)

agotterer 1186 days ago [-]
I had a similar problem once. The issue ended up being a router power cycle which occurred at the same time every single day. It took several days to diagnose. First, during an outage I tried to login to my router. Just because the internet was down the local area network should have been up. I couldn’t access the admin panel for several minutes. Next, I started observing the lighting patterns on the router. I came to realize that it appeared to be restating itself for no good reason. In the end my internet provider replaced the router which fixed the problem. Tech support was pretty stumped. A few people on the internet complained about the same issue with the Fios router model I had (don’t remember which one it was). Good luck! Let us know what the problem was if you figure it out.
breckenedge 1186 days ago [-]
I had a similar problem with my Asus router. Traced it back to it being scanned for UPnP every day at the same time. Something about the scan caused the router to crash and reboot. Disabled UPnP and the problem disappeared.
rapjr9 1185 days ago [-]
This is a known "feature" of some access points (AP), especially those used on big corporate WAN's. They want to keep the hardware costs low and software development/testing costs low, so instead of making sure the AP is absolutely solid in operation, they reboot it once a day to clear any problems. Even if the AP has dropped off the network it reboots at a preprogrammed time and gets back on the network. Also saves sending someone out to press a reset button. It also gets rid of any attack code that may have taken over via RAM but which is not present in the firmware PROM. I worked on a campus where all our wireless AP's would reboot every night at around 3am. Data collection for experiments in labs would either have to have retries built in, or use the wired network. This is not an uncommon commercial tactic because it takes an enormous amount of testing to try every possible situation a router or AP could get into (malformed packets, unknown bugs in standardized protocols, compiler bugs, overheating when a sunbeam shines on the router, ...) You might be able to change the time this reboot happens in the admin settings of your router to be something more convenient for yourself. That could also ensure you that as to the cause of the restart.
niklasmtj 1186 days ago [-]
Did you check if your public IP changes after this minute? I don't know if this still happens but back in the past my router did this every 24h to get a new IP.
rndgermandude 1186 days ago [-]
A lot of service providers do this because they do not want you to host static services from your "dynamic" home IP, as they are selling "business" lines with dedicated static IPs. Or at least they used to do this for that reason. By now it's just some thing a lot of ISP do. I have seen other possible explanations like ISP claiming this is a "security" feature because attackers cannot have permanent access without always learning the new IP, which is supposedly somehow hard. This is pretty bull tho. If attackers have access to your system they could just run software signalling them back any IP changes. And even nmapping the entire IP space of an ISP to find a service you were hitting again is very feasible.

Anyway, if the ISP does this, then you have to reconnect once every day (or sometimes every two days), and if you don't do so manually they will just cut your connection on their side and make you reconnect.

My (German) ISP does that too, the one I had before did it, the one before did it as well.

znpy 1186 days ago [-]
If you can replace the ISP provided router with your own you should be able to renew the lease.
njsubedi 1186 days ago [-]
I can confirm this. The time is around 7:00PM and the IP address changes, and I have to log in again to some websites because my IP has changed. Downtime is approximately 2 minutes.
pmiller2 1186 days ago [-]
This was my thought as well. It sounds like a 24 hour DHCP lease expiring.
tyingq 1186 days ago [-]
At 3:39p, unplug your router and leave it unplugged for an hour, then plug it back in. If the outage then moves to 4:40p every day, it's a 24 hour IP address lease expiring.
spuz 1186 days ago [-]
Are you talking about IP leases from the router to the local network or IP leases from the ISP to the router?
mvolfik 1186 days ago [-]
likely router -> ISP - if it was the devices, they could show up as not being connected to the network. also weird that all devices' leases would expire at the same time
olmideso 1186 days ago [-]
If your devices are kept connected this means that your wifi and router are still active, probably the issue somewhere behind your router, either a connection or provider. But due to exact timing it's likely something on the provider's side and not just some connection issue. I would personally start with contacting them to investigate.

You might also check the router's logs for that time, perhaps there's something useful information.

the8472 1186 days ago [-]
Some german ISPs used to force a reset of home internet connections every 24h to trigger IP changes so that static IPs would remain reserved for the business tier. The time of the reset could be shifted by preemptively reconnecting at a desired time, they only reset the link after 24h of actual uptime.
0xTJ 1186 days ago [-]
I get why they do it, but actually causing issues just to cycle IPs seems like a lot. I've had cases where I have the same IP assigned to me for quite a long time.
aeyes 1186 days ago [-]
They already did this on dialup and ISDN, back when routers didn't exist. When DSL first started appearing they just kept doing it. The idea was to not keep unused connections/IPs up, since you didn't have a router you would need to reconnect manually.

Also, it made it harder to run servers until we had services like DynDNS.

On newer connections it isn't common anymore, especially if you have a voice service.

Lev1a 1186 days ago [-]
On my family's FritzBox we can set the time for that connection reset. I think it's somewhere around 04:00.
samradelie 1186 days ago [-]
Probably not your issue: Internet: Old TV caused village broadband outages for 18 months https://www.bbc.com/news/uk-wales-54239180
fersarr 1186 days ago [-]
I was thinking of this too.
derwiki 1186 days ago [-]
Disconnect your wifi gateway and plug a computer directly in to your modem. Eliminate or prove the router as the problem source.
detaro 1186 days ago [-]
Sounds like standard reconnect every 24h, many ISPs do that. reset your modem at a weird time to get the interruption at that weird time in the future.
bjeds 1186 days ago [-]
This.

I think the OP is so eager to get back online he sees 3:40 every day and is using that timepoint as his troubleshooting clue.

However If he were to disconnect 3:10 and then just be offline an hour until plugging the device back he would probably see the reset happen 4:10 going forward.

milankragujevic 1186 days ago [-]
@OP did you try calling support to report the problem? It's almost impossible to diagnose ANY internet problems (not related to your own equipment of course) and the ISP has tools and access to do it. In any case, it's much easier and faster to contact support.

In my experience, even if I discover what is the problem, if it's on their side I still have to go through support to get it fixed, so I just wasted my time. And in most cases the lowest level of support doesn't even write down the information I give in the ticket, even if I send an email.

ayewo 1186 days ago [-]
1. Start the ping command at 3:39pm and leave it running until 3:41pm then make a note of whether packet losses start at 3:40pm. (E.g. "ping yahoo.com" on Unix or "ping -t yahoo.com" if on Windows).

2. If there are packet losses, then you have your answer: the issue is from your ISP.

3. If there are no packet losses, then you'll need to look closer at your network. Check to see if some hardware on your network might be performing a reboot at the 3:40pm mark.

krona 1186 days ago [-]
Reminds me of this news article: Old TV caused village broadband outages for 18 months https://www.bbc.co.uk/news/uk-wales-54239180
to11mtm 1186 days ago [-]
Oy! You got a license for that broadband jammer mate?
oger 1186 days ago [-]
"My internet" is a bit vague - so I am making some assumptions here. Scenario (A) could be that your WiFi dies at a specific point in time. You might then want to look at some source of electromagnetic interference (like a motor starting while creating sparks). Scenario (B) could be that your internet service provider is cutting your connection and provides a new IP address - which is common for many xDSL providers. You want to check if the external IP address of your router changes. You may also find some log files in your router that may give you more information. Some routers allow you to select the time when the router preemptively disconnects at a certain time in order to avoid the disconnect from the ISP at an arbitrary moment. You may want change this setting or to talk to your ISP and try to convince him to not cut the connection during the day but rather during the night. Scenario (C) could be a device on your local network creating interference. Your tools of choice will be tcpdump and WireShark to locate the culprit on your LAN. Enjoy your debugging and good luck!
2rsf 1184 days ago [-]
You don't need something fancy like motor sparks to kill a 2.4Ghz WiFi, a leaky Microwave will do that and perfectly explain the 1 minute length.

Switching to 5Ghz will help if this is the case

tgeorge 1186 days ago [-]
I would check out DSLReports and ask them there. http://www.dslreports.com/forums/all

Chances are they experienced it there too. But I had an issue like that with Cablevision at 3-4am when they would push out firmware updates to the modem.

ourcat 1186 days ago [-]
Have you tried different WiFi channels?

I had a similar experience to this years ago while working at my mother's kitchen table for while. The network would keep going dead and I had no idea what was causing it.

Eventually, I discovered that it happened every time she used the microwave to heat up a drink or something.

Changing to a different WiFi channel fixed it. :)

bonestamp2 1186 days ago [-]
I came to suggest the microwave too. The 24h IP reset sounds more likely, but don't discount the microwave, they typically use the 2.4 GHz frequency (as do bluetooth and some wifi networks of course). So, if the 24h IP reset isn't it, try switching to a 5 GHz wifi network to see if the same dropout occurs. If that solves it, it could be a nearby microwave oven.
rsyring 1186 days ago [-]
Step 1: reboot all devices on the network: computers, router, cable modem (or whatever equivalent device your router plugs into).

If that doesn't fix it, below is how I'd proceed.

Use the trace route networking tool (google it, different syntax on different OS) to some known good public IP address. I usually use 1.1.1.1 or 8.8.8.8.

This tool shows you the path your traffic takes:

    $ traceroute 1.1.1.1
    traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 60 byte packets
    1  192.168.1.1 (192.168.1.1)  1.286 ms  1.386 ms  1.963 ms
    2  192.168.11.1 (192.168.11.1)  2.591 ms  4.342 ms  5.219 ms
    ...snip...
    16  0.ae19.GW8.CHI13.ALTER.NET (140.222.230.223)  42.888 ms  35.742 ms 0.ae20.GW8.CHI13.ALTER.NET (140.222.230.225)  41.060 ms
    17  152.179.105.202 (152.179.105.202)  51.913 ms  50.681 ms  55.442 ms
    18  one.one.one.one (1.1.1.1)  70.087 ms  65.130 ms  53.607 ms
The snippet above shows that I have two local devices (192.168.1.1, 192.168.11.1) my traffic passes through and 18 devices total before my traffic hits the public IP.

Run this command at 3:39. Then run it again at 3:40. Then again at 3:41. Compare results and that should give you a good idea which device is malfunctioning at that time.

Some routers also have a ping command baked into them. If you can make it constantly ping 8.8.8.8, while you run the test above, it may help you. In the case where your traceroute stops at your own router, you can look at the ping from the router itself, and see if it also stopped:

A) If the traceroute stops at your router, but the router ping is not interrupted, then something in your router is flaking out.

B) If the router ping is also interrupted, then the problem is "upstream" from you. Maybe the cable modem (or whatever device) that is upstream from your router.

If you decide the problem is upstream, contact your ISP. If it's local, investigate and/or replace the flaky device.

HTH.

dvcrn 1186 days ago [-]
Not likely your issue but I had something similar that exactly every day my Wi-Fi would act up and my computer disconnect. It also happened to be the same time a nearby church starts ringing it's bells and my room happened to be very close to it. It also didn't happen on days the church didn't use the bells.

Still don't know what exactly caused this and I don't live there anymore but because I was the only one in my family with these problems, and only my room was facing the church, I want to believe the bells rang at a frequency that messed with my old routers Wi-Fi.

If someone could tell me if this Is possible, I could finally get some closure on this.

rapjr9 1185 days ago [-]
A lot of church bells these days are not real bells, they are PA systems playing a recording of church bells. They are often programmable and wirelessly controlled, for example:

https://www.brgproducts.com/carillon_pricing/index.php

So it seems entirely possible the wireless part of the church bell system was interfering with your router. If the church did have real bells it seems very unlikely they would generate strong EM radiation at 2.4GHz or 5GHz. They might induce some vibration if they are really loud, which could trigger an intermittent connection in the router, though that seems pretty far fetched also.

MrStonedOne 1186 days ago [-]
Hack at the problem until its reduced to its purest form.

> My devices remain connected to the network, but all traffic dies.

Can the devices send packets within the network, but not the internet? Use ping to double check. if no, do you have any wired devices to confirm rather or not its wireless interference vs the router doing some sort of maintenance task.

Does this impact all protocols? tcp/udp/imcp. I've seen random network hiccups only impact tcp before.

ping tests imcp, dns (nslookup on windows/host on linux) tests udp (and can also test tcp) http tests udp and tcp, depending on browser and service. curl can let you confirm the test is going over tcp.

MTR is likely your final solution for investigating this. It is like a traceroute that rapid fires out to get second by second details about all the hops in a network path. The types of errors you get can also tell you why.

If it ends up being tcp only and internet based, things get harder, some mtr clients support using tcp instead of imcp to find hops where tcp breaks, but your isp routers might also refused to reply to such packets directly.

Finally, watching the lights on the router and modem (if seperate) can be illuminating. First, get a feel for normal light operation, normal activity blink speed, etc, then starting a few minutes before the event normally happens, just observe the lights for changes until the event ends while also using your phone to know when the event has started and ended.

On that note, look up how to access web consoles on the router (and modem if seperate). They may have event logs that tell you if anything is happening, like ip renewals, or if they are getting commands from the isp to do things.

sgt101 1186 days ago [-]
What type of connection are you on? Is it DSL or fibre. If it's copper then my guess is that there is a device (like a heating pump) that's on a timer with a big switch thats creating a pulse of radio noise that's then echoing up and down the copper and pushing the session out of bounds. At that point the far router (in the dsl box on the road) will try and resync the connection.
rsecora 1186 days ago [-]
First step check if the external wan IP address has changed.

Second Check the logs of the pppoe daemon in your router. (If the router provides such logs).

drewpc 1186 days ago [-]
I'm actually surprised by all of the comments. The community jumped right into solving OP's problem and has largely ignored the OP's original question of "how could I begin investigating this?" Another variation of this question could be "how do I troubleshoot complex network issues?"

After 20 years of network/system/software engineering across a wide variety of network sizes and levels of complexity, I teach people the following method to troubleshoot complex network issues:

1. Write down the symptoms that you're seeing. How do you know something is wrong? What do you see happening?

2. Draw a diagram that includes all of the components and nodes involved. (this is hard)

3. Develop some hypotheses that could be worth testing. (this is hard and highly variable based on knowledge/experience)

4. Establish a "test plan" that allows you to prove/disprove the hypotheses while making minimal changes to the system. Start from one source device and work your way out to the farthest component you identified in your diagram. Start at the lowest OSI layer and work your way up.

5. Methodically test components, step by step, along the diagram.

6. As you test, record your results. Add new hypothesis to the list as you gather more information but don't start testing them right away! You can develop a new test plan after you finish the first one that proves/disproves your new hypotheses.

7. Repeat all steps, adding new information, until you find a solution.

I know this seems like a lot of steps when you're just troubleshooting a WiFi issue at home--it may be overkill. However, this framework applies to that scenario or when you're diagnosing any complex network issue. You'll learn more about the systems, protocols, and devices that truly make up the Internet than you could ever imagine. Through repetition and experience, these steps will get easier and some of them can happen quickly and in your head.

To help you get started, here's some info for steps 1, 2, and 3:

1. From your post: "3:40pm every day my wifi loses internet access. My devices remain connected to the network, but all traffic dies. Almost exactly 1 minute later everything is resumed"

2. Start your diagram off as an "equipment string diagram". This will include all physical devices between you and the Internet. Along the way, you may need to modify the equipment string diagram to include "virtual" devices, network segments, various protocols/servers, etc. Your diagram should include at least the following items:

  - Your laptop/desktop.  If it's happening to all devices, then pick one.  Identify your IP and MAC addresses
  - The WiFi access point that device is connected to.  Identify the IP and MAC addresses
  - Any switches, firewalls, modems, etc that connect from your WiFi access point to your Internet Service Provider (ISP) and their IP and MAC addresses.
  - The "next hop" from your modem into the ISP's network.  You can use a cloud to represent the ISP's network that you don't know/understand, but always identify the IP address of the device that your modem first reaches.  If you can find the MAC address (or other OSI Layer 2 address) as well, even better.
That's the equipment string between you and "The Internet". For now, you can ignore the complexity inside the ISP's network and beyond. You might have to add more of that later, but start small.

We know there are other components involved in making the Internet work that introduce complexity and can cause issues along the way. Let's list them on your diagram and identify what servers are used and where they might be located.

  - DHCP (local to your network and also between your modem and ISP)
  - DNS (could be local to your network and often is a third party service either run by your ISP or not)
  - Encryption (VPNs, SSL certificates, network device clock settings, etc)
  - IP routing (what devices do IP routing?  Hint: all devices that operate at Layer 3, using IP, do IP routing--including your workstations)
3. Some hypotheses (some were identified in the comments):

  - Is there a device in the equipment string that is rebooting every day?
  - Is DNS intermittently failing?
  - Is DHCP releasing/renewing your IP address assignment?  This could be from your device -> local DHCP server OR your modem -> ISP DHCP server
  - Is there an upstream connectivity issue with your ISP?
  - Is your WiFi access point losing connectivity?
tacon 1186 days ago [-]
The Universal Troubleshooting Process has been a web site since the mid-90s:

http://troubleshooters.com/tuni.htm

drewpc 1186 days ago [-]
Thanks for sharing!! I've never seen this before.
grogenaut 1186 days ago [-]
I will point out to all the people surprised at the time lining up that if the issue is a timer then this is likely very common. There are only 1440 minutes in a day, and waking hours are half that brinignrh it very close to the birthday paradox (double). https://en.m.wikipedia.org/wiki/Birthday_problem.

If you consider 4 time zones in the us it's actually more common than birthdays since you'd divide by four again and people would suspect both a company wide tx synced issue and a per tx issue.

With 23 people about 1/3rd of the people commenting at this point we get a chance of 2 people colliding as:

Full day: 16% 12 Hour: 29% Across time zones: 79%

nicoffeine 1186 days ago [-]
Make sure all your network devices / computers are synced via NTP, or at least accurate to the second.

Hardwire the Pi to the network. Start a verbose traceroute and ping loop on a device on WiFi, as well as the Pi. Once it fails, observe the lights on your router/wifi device(s). Note time to the second of the start and finish of the outage, and the behavior of the lights. Check the results of your traceroutes/pings. Start digging through the logs of each network device.

Also, you can never quite rule out power, even if the lights stay on. A power conditioning UPS is useful here.

petercooper 1186 days ago [-]
Just how I'd approach it but I'd plug my machine (wired) into the closest egress point possible (cable modem, DSL modem router, whatever), open up the Network pane, run a reliable ping, load up the modem/router's status page, then see exactly what happens at that time.

This would let me pick up on disconnects, DHCP lease issues, and a variety of other things in one hit. And if it wasn't any of those, would allow me to explore some other possibilities such as the problem being at the ISP's end.

jamescun 1186 days ago [-]
Sounds very regular, may be worth contacting your ISP to see if that is their maintainence period or any other regular operation they may perform.

I am with a broadband provider in the UK called Zen, at 23:15 every Sunday our PPPoE session is terminated, and it takes a couple of minutes before it successfully resumes. Occasionally it will happen mid-week, however still at 23:15. At one point I suspected the provided FRTIZBox router, however the problem persisted once we moved to an MT992 modem and a Unifi USG.

xbkingx 1186 days ago [-]
I had the same problem with and older Netgear router. It turned out to be a bug in the scheduling settings. It had the ability to disable network access between specific times on a daily schedule. The feature was enabled by default, but no times were set to disconnect. You'd think that would mean the connection would never go down, but a firmware update changed that. Every morning at ~3:00am the network would go down for 1 minute. Disabling the feature entirely resolved the problem.
jrockway 1185 days ago [-]
I had this problem on my wired connection with Windows. Exactly 24 hours after boot, I would lose my connection with certain services (mostly realtime games! very annoying!) After some investigation, I realized that my computer renewed its DHCP lease with the route at this time, and switched to a static IP allocation. Problem solved.

Honestly, this should not cause anything to lose its connection (the renewed IP address is the same), but it did, so I worked around it.

danesparza 1186 days ago [-]
Many people have mentioned DHCP -- and a misbehaving DHCP client could certainly be the culprit.

In doing some of my own network investigating recently, I discovered that services like Suricata will (by default) take down network interfaces and restart them when they get new rules. I wonder if something software on your router is doing something similar: Are you running Suricata or Snort?

Also: Logs are your friend. If you have access, review the logs of your firewall / router / etc.

analog31 1186 days ago [-]
One thing is to find out if something has a midnight bug, but has its internal clock set to something other than your actual time zone. I once committed a midnight bug in a manufacturing plant, and it took me a while to diagnose, because none of the PC's were networked, and I had never bothered to set the time on them. I puzzled over it until one of the operators said: "It always happens at the same time every day during my shift."

Disclosure: I'm smarter now.

scotty79 1186 days ago [-]
It was easy way of dealing with router hanging up due to accumulated problems used by amateur internet providers. Just disconnect it from power once a day.

You can do that with cheap socket timer.

Not sure why would anyone set it to the middle of the day. Maybe error in setting times or clock or it was set intentionally so that you can service it at normal hours if it doesn't come back on.

The problem is you can put in 'off' time and 'on' time but they have to be at least 1 minute apart.

rem1313 1186 days ago [-]
What channels do you use on wifi? If you you use DFS channels on 5GHz it could be a radar issue that forces your router to switch channels. I have it as well, but it happens at night so I'm not terribly concerned about it (my router sends me notification thats how I know).

Edit: https://en.wikipedia.org/wiki/Dynamic_frequency_selection

qwertox 1186 days ago [-]
The devices are staying connected, so I doubt that this could be the source of the problem.
bobowzki 1186 days ago [-]
This could also be a power or electrical interference issue. I would start by logging the connection with ping every 1 sec or so to see if it's only 3:40pm.
freitasm 1186 days ago [-]
Are you on a xDSL service of fibre?

On xDSL services, some monitored alarms may cause line problems when they "check in". In these cases an in-line DSL filter should do the trick.

Another possibility is that some modem/routers have a "self healing feature" that is just really a restart. See if there's anything like this in the admin/system part of the configuration settings.

mtippett 1186 days ago [-]
Similar situation. Occasionally super bad packet loss within the network. https://use-cases.org/2020/12/29/diagnosing-eero-issues-with...

Still haven't resolved it, but it is likely Fing on Big Sur.

uzgz 1186 days ago [-]
Check this article. Might be some other faulty electrical device: https://www.ispreview.co.uk/index.php/2020/09/18-months-of-v...
jedimastert 1186 days ago [-]
I love that almost everyone here has some sort of "family IT syndrome" but still got nerdsniped but the question
corysama 1186 days ago [-]
Does your WiFi provide a way to monitor the real time bandwidth usage of each device? I was experiencing randomly timed outages for a while. Turned out to be our various phones and other devices randomly saturating my low upload bandwidth with cloud backups. That shouldn’t kill my downstream bandwidth, but it does.
unilynx 1186 days ago [-]
It doesn't kill your downstream directly, but you can't send the acknowledgements for the download fast enough if the upload is saturated. That kills your download speed
merpnderp 1186 days ago [-]
The sun was hitting my cable line at the same time every day causing my internet to go out, my router to reboot and the line power to be automatically increased. I finally called out a tech who fixed some old splitters and now the strength isn’t just barely marginal enough that the sun would cause a drop.
uzgz 1186 days ago [-]
Second possibility: Did you connect a Sonos Device to your router using ethernet? If so, does your Sonos send on the same wifi spectrum your router does? My Play:5 sent on channel 11, so did my router, mesh repeater could not send to base while all connections appeared to be just fine.
tdiff 1186 days ago [-]
I am using LTE internet via Huawei 3372 and it stops for about a minute or two exactly 6 hours after starting the modem (of course during some important call). After it, existing connections continue to work. I'm curious to know what might be causing this.
dougSF70 1186 days ago [-]
I have the same wifi issue. At 3pm (approx) my comcast business wifi drops for a minute or two.
zaidazmi 1186 days ago [-]
Some routers allow to reboot daily at a set time. Worth checking the settings of your router.
1186 days ago [-]
princevegeta89 1186 days ago [-]
Wow I have the same problem here. In my case I found out it is at&t themselves.
hepinhei 1186 days ago [-]
It is likely to be the automatic channel selection. It is a process of selecting the best channel and normally you lose connectivity. If not that, you should be a scheduled maintenance procedure from your isp
dwt204 1186 days ago [-]
I am not an expert whatsoever. I just notice a similar thing happening at around the same time, the wireless and sometimes ethernet connection dies. Usually a minute up to three minutes the connection resumes.
Brian_K_White 1186 days ago [-]
At 3:30, disconnect your router from your modem and connect a laptop directly to the modem. No wifi. Start a speed test or long video or something and just watch the modem through to 4:00.

HAVE a separate modem and router.

tekknolagi 1186 days ago [-]
We had this problem in our house and it turns out it's because I accidentally plugged the modem into one of those auto power switches meant for turning off lights periodically.
pxeboot 1186 days ago [-]
I had a connection through a small municipal ISP until moving recently. My connection would drop for several seconds at exactly 23:59 UTC every day. Never was able to figure it out.
andreasha 1186 days ago [-]
Maybe the router is using DFS channels and some radar station takes a reading at that time every day and forces your router to change channel on its 5GHz network.
raarts 1186 days ago [-]
Switch off you WiFi (and your modem if they're different). Stay up very late and switch them back on at 2am or so. Maybe the problem now occurs at 2am.
not2b 1186 days ago [-]
It's probably the 24 hour IP reset, as many commenters have said. You could try time-shifting it to a time when you're normally asleep.
jb775 1186 days ago [-]
Go to 192.168.0.1, log into your router/modem and see if there's an admin "Event Log"...could find some answers there.
Siira 1186 days ago [-]
Friendly advice, investigating this can only be worth the time it takes if you care about the learning or treat it as entertainment.
mrkramer 1186 days ago [-]
This is most probably because you have dynamic IP address and your router resets at the same time every day to fetch new IP address.
kvhdude 1186 days ago [-]
did you check your router (sys)log messages. Many of the routers will allow enabling debug logs as well.
uncledave 1186 days ago [-]
Try and get hold of your Internet/WiFi router's logs. The answer will probably be in there.
29athrowaway 1186 days ago [-]
This sounds like your DHCP lease is expiring and you have to request a new one each time it does.
_important 1181 days ago [-]
somebody (like neo) is trying to get in/out of the Matrix. Your internet connection is the most suitable path for them.
ews 1186 days ago [-]
Is your ISP att/fiber? I have exactly the same problem every night at 4am.
cvandebroek 1186 days ago [-]
+1 on @detaro's message. Sounds like a 24h disconnect. Your router may have a feature to set dis/connect times manually ir you unplug/replug it at the time you wish to have the dis/connect to happen.
davidro 1186 days ago [-]
Change your DNS servers.. I use openDNS, but there are lots of other choices. If the Pi isn’t in use set it up as a Pi-hole (DNS server / ad blocker), so you’re not use your isp provided router for DNS.
netsharc 1186 days ago [-]
DNS probably has nothing to do with disconnects, this kind of random suggestion unrelated to the problem always makes me think of the Futurama Robot Judge: https://www.youtube.com/watch?v=Jqn8HB2w2Fs (skip to 0:15 for the punchline)

Or maybe it is related, in which case I'll eat at a hat.

mackatsol 1186 days ago [-]
I've had outages that were due to my ISP's DNS service going down... one of the reasons I started using OpenDNS.

I need to watch more Futurama. That was great!

ffpip 1186 days ago [-]
Maybe your ISP resets your external IP at that time.
technick 1186 days ago [-]
Are you using Spectrum / Charter by chance?
lmilcin 1186 days ago [-]
Use traceroute to find where your packets die?
raullen 1186 days ago [-]
Mine dies for 10min
atomashpolskiy 1186 days ago [-]
traceroute?
alisausaaaaaa 1185 days ago [-]
Wanna have hot-lovin' conversations? You’re on the right way! - https://adultlove.life
amelius 1186 days ago [-]
Get better hardware and software that properly reports error messages instead of hiding them.
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact
Rendered at 12:13:10 GMT+0000 (Coordinated Universal Time) with Vercel.