NewNotSYN Dropping my Broadband Connection

[mpyusko]

Recently, about mid day, but maybe other times (the graph doesn’t allow scrolling back to different times) I’ve been busy working when my Internet connection craps out. My SSH, RDP, and VPN connections all drop. my VOIP calls cut-out u but restore themselves. My Firewall optios are set to log “NewNotSYN” but I’m not sure if plainly disabling the logging, or adding a proper Rule will solve the issue.

As you can see, I’m not even scratching the surface of my CPU, or NIC (25x480Mbps) but the sudden influx is enough to flog my connection. Why? On the backend I have a SATA SSD for logging, with plenty of space. Thoughts?
(I didn’t get a screencap of the external Network traffic, but it was minimal.)

Here is one from last week…

I have two more screencaps from today, but I’m only allowed 1 image at a time.

[homercide]

I don’t know about the logging issue but it would appear something may be trying to get updates or do some other activity, not being able to the review the logs will likely make it hard.
Do you have any new devices on your network from about the same time the issue started?
I am guessing you are working from home these days, maybe have a look around and see what all is running?

[mpyusko]

I’ve always worked from home. That being said, I am typicically the only one home during the day, and although I do have a decent sized network, most of the devices are idle. I looked through the logs and at the time it cut out, I had 1.2 to 1.4 thousand open connections and the NewNotSYN log entries show about a hundred different IPs and mostly Amazon, but from several different subnets (green and blue, static and DHCP). I know what all my devices are that are on my network. A lot of the local devices aren’t even amazon devices (not saying it matters much the way things get cloud-hosted). What I’m trying to figure out is how to stop it from happening again.

[homercide]

I had something crashing my ethernet switch a while back, turns out it was my wireless printer going to sleep, it for some reason barfed in the last use, also I had to actually remove the power cord from it, I read some forums posts awhile back, some AP’S we’re doing strange things and required a reboot even though they were working correctly, maybe unplug everything in the house and bring them back online in order of modem, firewall and so on.

My first guess says it Windows 10 trying to talk to the mother ship.

[mpyusko]

Power went out within the past week. PITA getting it all back online again, even with a plethora of batteries. Kinda odd how it’s like all at once a handful of different devices suddenly shoot up to ~100 connections each… all at the same time.

[homercide]

Yeah I had a similar issue with power going out also, it just blinked out for a few seconds but I was back to rebooting everything again, I cheated with the big wall mounted tv though, Firetv stick is behind the TV, same with power outlet, I was lazy on that one, just walked over to the breaker box and killed it for a few minutes, much easier then removing a large TV on the wall to reboot a streaming stick.

I would guess one device is having issues and causing the other ones to act up, just a guess since I have seen it.

[jon]

If the logging is on:

Then you should see lots of entires in the Firewall Log. From that you can see the IP address (and the MAC address) of the troublemaker.

[homercide]

Thanks jon, it simplifies the process.

[mpyusko]

Yes logging is on… I figured I’d wait until it happend again then I could see where and what in realtime. I wasn’t disappointed. Amazing this can stall my Internet connection. I mean, I notice it because SSH, VPN, and VOIP all start crapping out. Is it because my firewall is just too weak? It doesn’t look like it’s even scratching the surface. CPU ~15% It wasn’t even enough to raise the clock on either core. As far as Process count goes, ~400m… it’s about half of what the peak for the day was. The SSD writes, minimal, RAM 71% free (no change in a week).

So should I just drop all NewNotSYN to fix it? I’m still evaluating the logs to see where the culprit is.

[jon]

This will go back about 2 weeks to look for DROP_NEWNOTSYN in the message log:

for logf in $(ls /var/log/messages* | sort -rV | tail -2) ; do ls -al $logf ; zgrep -i "DROP_NEWNOTSYN" $logf ; done

Then you can match past peaks with the log...

[homercide]

It’s a bit hard seeing exact times but to me it appears something on blue starts sending before all the inbound traffic.

[mpyusko]

@jon

for logf in $(ls /var/log/messages* | sort -rV | tail -2) ; do ls -al $logf ; zgrep -i "DROP_NEWNOTSYN" $logf ; done

That’s a 10 MB (output) file LOL.
So I was just passing through the WebGUI And noticed they seem to be inboud TO my public IP. But the source MAC does not match anything on my network (still verifying) and the inbound NEWNOTSYN have numerous Source IPs.

Restated… Inbound NewNotSYN:
Numerous Source IPs (varying subnets and GeoIP ranges)
Single Source MAC
Destination is my Public IP, not anything NAT behind the Firewall.

From 14:27 to 14:31 today there are only 211 Lines.
Of those, 198 are “IN=red0”
Of those, there are 27 unique IP’s with a single MAC:

[root@skynet ~]# grep -e "IN=red0" temp1.txt |sort -t'=' -k4|cut -d ' ' -f 5-11|sort -u
skynet kernel: DROP_NEWNOTSYN IN=red0 OUT= MAC=00:1c:c0:43:47:2f:00:01:5c:8b:9e:46:08:00 SRC=104.16.109.79
skynet kernel: DROP_NEWNOTSYN IN=red0 OUT= MAC=00:1c:c0:43:47:2f:00:01:5c:8b:9e:46:08:00 SRC=157.245.185.115
skynet kernel: DROP_NEWNOTSYN IN=red0 OUT= MAC=00:1c:c0:43:47:2f:00:01:5c:8b:9e:46:08:00 SRC=162.125.19.131
skynet kernel: DROP_NEWNOTSYN IN=red0 OUT= MAC=00:1c:c0:43:47:2f:00:01:5c:8b:9e:46:08:00 SRC=162.159.130.233
skynet kernel: DROP_NEWNOTSYN IN=red0 OUT= MAC=00:1c:c0:43:47:2f:00:01:5c:8b:9e:46:08:00 SRC=162.159.136.234
skynet kernel: DROP_NEWNOTSYN IN=red0 OUT= MAC=00:1c:c0:43:47:2f:00:01:5c:8b:9e:46:08:00 SRC=172.217.165.142
skynet kernel: DROP_NEWNOTSYN IN=red0 OUT= MAC=00:1c:c0:43:47:2f:00:01:5c:8b:9e:46:08:00 SRC=172.217.9.238
skynet kernel: DROP_NEWNOTSYN IN=red0 OUT= MAC=00:1c:c0:43:47:2f:00:01:5c:8b:9e:46:08:00 SRC=191.101.50.190
skynet kernel: DROP_NEWNOTSYN IN=red0 OUT= MAC=00:1c:c0:43:47:2f:00:01:5c:8b:9e:46:08:00 SRC=198.22.253.113
skynet kernel: DROP_NEWNOTSYN IN=red0 OUT= MAC=00:1c:c0:43:47:2f:00:01:5c:8b:9e:46:08:00 SRC=23.10.88.237
skynet kernel: DROP_NEWNOTSYN IN=red0 OUT= MAC=00:1c:c0:43:47:2f:00:01:5c:8b:9e:46:08:00 SRC=23.223.156.43
skynet kernel: DROP_NEWNOTSYN IN=red0 OUT= MAC=00:1c:c0:43:47:2f:00:01:5c:8b:9e:46:08:00 SRC=3.213.182.132
skynet kernel: DROP_NEWNOTSYN IN=red0 OUT= MAC=00:1c:c0:43:47:2f:00:01:5c:8b:9e:46:08:00 SRC=3.226.165.42
skynet kernel: DROP_NEWNOTSYN IN=red0 OUT= MAC=00:1c:c0:43:47:2f:00:01:5c:8b:9e:46:08:00 SRC=3.231.74.94
skynet kernel: DROP_NEWNOTSYN IN=red0 OUT= MAC=00:1c:c0:43:47:2f:00:01:5c:8b:9e:46:08:00 SRC=34.200.63.6
skynet kernel: DROP_NEWNOTSYN IN=red0 OUT= MAC=00:1c:c0:43:47:2f:00:01:5c:8b:9e:46:08:00 SRC=34.233.202.213
skynet kernel: DROP_NEWNOTSYN IN=red0 OUT= MAC=00:1c:c0:43:47:2f:00:01:5c:8b:9e:46:08:00 SRC=34.238.26.171
skynet kernel: DROP_NEWNOTSYN IN=red0 OUT= MAC=00:1c:c0:43:47:2f:00:01:5c:8b:9e:46:08:00 SRC=35.153.172.172
skynet kernel: DROP_NEWNOTSYN IN=red0 OUT= MAC=00:1c:c0:43:47:2f:00:01:5c:8b:9e:46:08:00 SRC=35.172.64.65
skynet kernel: DROP_NEWNOTSYN IN=red0 OUT= MAC=00:1c:c0:43:47:2f:00:01:5c:8b:9e:46:08:00 SRC=35.186.224.44
skynet kernel: DROP_NEWNOTSYN IN=red0 OUT= MAC=00:1c:c0:43:47:2f:00:01:5c:8b:9e:46:08:00 SRC=40.126.28.13
skynet kernel: DROP_NEWNOTSYN IN=red0 OUT= MAC=00:1c:c0:43:47:2f:00:01:5c:8b:9e:46:08:00 SRC=52.167.253.237
skynet kernel: DROP_NEWNOTSYN IN=red0 OUT= MAC=00:1c:c0:43:47:2f:00:01:5c:8b:9e:46:08:00 SRC=52.230.222.68
skynet kernel: DROP_NEWNOTSYN IN=red0 OUT= MAC=00:1c:c0:43:47:2f:00:01:5c:8b:9e:46:08:00 SRC=52.242.211.89
skynet kernel: DROP_NEWNOTSYN IN=red0 OUT= MAC=00:1c:c0:43:47:2f:00:01:5c:8b:9e:46:08:00 SRC=52.32.34.32
skynet kernel: DROP_NEWNOTSYN IN=red0 OUT= MAC=00:1c:c0:43:47:2f:00:01:5c:8b:9e:46:08:00 SRC=52.84.130.63
skynet kernel: DROP_NEWNOTSYN IN=red0 OUT= MAC=00:1c:c0:43:47:2f:00:01:5c:8b:9e:46:08:00 SRC=52.84.53.61

[root@skynet ~]# grep -e "IN=red0" temp1.txt |sort -t'=' -k4|cut -d ' ' -f 5-11|sort -u|grep -c .
27

104.16.109.79
157.245.185.115
162.125.19.131
162.159.130.233
162.159.136.234
172.217.165.142
172.217.9.238
191.101.50.190
198.22.253.113
23.10.88.237
23.223.156.43
3.213.182.132
3.226.165.42
3.231.74.94
34.200.63.6
34.233.202.213
34.238.26.171
35.153.172.172
35.172.64.65
35.186.224.44
40.126.28.13
52.167.253.237
52.230.222.68
52.242.211.89
52.32.34.32
52.84.130.63
52.84.53.61

[homercide]

Do you have an eero access point with Vlan ?

[mpyusko]

???
I have a Arris Cable Modem on Red0
I have a Linksys Wifi Router and a Linksys AP on Blue0
I have a Cisco Switch on Green0
I have nothing (right now) on Orange0

So I was discussing this with a peer and he suggested “it looks like the Firewall forgot who was talking to what”. “Like the connection got reset”. Well none of the major devices rebooted, and I didn’t cycle anything. I took a look at the Cable Modem logs and got this: (note the time is actually off by an hour on the Cable Modem)

This is also consistent with the last time I noticed an issue. I called My ISP and the said they weren’t experiencing any issues, but whenever it happens, this is what the cable modem log looks like. All my Uppstream and DownStream signals look good and well within the margins. (currently) It also says it’s been up for over 12 days (since the last power outage.) Note it, like everything else, is on redundant UPS and was powered off cleanly when the battery went down. No surge/dip.

I’m thinking the Cable Modem itself is actually triggering the issue and it’s a sign the Cable modem needs to be replaced(?)

[homercide]

That makes since, from syslog I broke down the MAC address, one appears to be Intel, the second was from Cadent Inc. which was purchased by Arris in 2001, the last two octets I would assume are Vlan frames.

Do you have Vlan on the WAN connection by chance?

I thought I read some eeros products may have a Cadent chip in them so it was a long shot.

Maybe a longer power down on that modem and it will behave, I would try that first before shelling out the cash for a replacement.
Maybe you can get the ISP to send you something to test with?

[mpyusko]

The ISP owns the cable modem, so money isn’t the issue.
I do not have vLAN enabled on the WAN unless the ISP is using it. I believe the Arris can manage DHCP for up to 4 clients if I connect it to a switch. So maybe that’s how it’s assigning itself the 192.168.100.1 net while still allowing the firewall to connect as a public IP?

[homercide]

@mpyusko

192.168.100.1

I think most all cable modems have that for diagnostics.
Since they own the modem see if they will send a new one, likely you will end up with a newer model, seems the network issues and then time on the forum are eating up your work day, maybe the new one can adjust the clock for daylight savings ?

[mpyusko]

FYI, It’s been getting worse.

Spectrum just verified the Cable Modem looks faulty. I’m going to take it in and swap it out… Hopefully this should solve the problem.

[homercide]

Wow, it looks like it is totally bombing on you, did you notice some of the dates being 1/1/1970 ?
Looks like it tried to renew the DHCP lease and puked downloading a config file or something using TFTP, time update then failed.
I think it is trash now, a new one should get you all fixed.

[mpyusko]

That’s what I’m thinking, but I won’t be able to until tomorrow.