NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 5600 ms

Hello all, wondering if you can suggest some place to look…

I have an issue after some changes, my ISP sent out a new cable modem and upgraded my connection to 1100MBits/s Down and 100MBits/s Up (From 750MBits/s up and 80MBits/s Down).

Since then we are having stability issues when running bigish downloads from the internet.

I can not reproduce issues copying between the Green/Orange networks or when copying to/from the IPfire server. So it looks like I get the adapters resetting only when the Red network is involved with the Green or Orange Networks.

To summaries:
-On the IPfire server itself downloading an iso with wget from the internet works perfectly. (the examples I used downloaded at only about 20MB/s and 50MB/s)

-From the Green or Orange networks downloading an iso from the internet causes the respective Green or Orange adapter to keep reseting. Similar download speeds are reached but the reset will start happen with in 1 seconds and 10 seconds of the download starting)

Apart from the new modem, the only change I can see it will have had on the IPFire is that the connection between the modem and IPFire is now running at 2500Mb/s instead of 1000Mb/s.

I have tried downgrading the connection on IPFire red0 interface to 1Gbps but that did not help.
Wiregauard/OpenVPN stopped.
No IPS running.
Only service I am running is location blocking.

IPFire: IPFire 2.29 (x86_64) - core196
Hardware: www.ipfire.org - Profile a03839f5aa3287cef3cfb5e11cef10ff4f1614dd

Tests:
Can not really find anything in the logs I looked at…
IPFIRE Ethernet Controller I225-V 2.5Gbps <switch 2.5Gbps> 1 & 2.5 Gbps `Windows & Linux Desktops

dmesg
...
[ 5154.806182] igc 0000:01:00.0 green0: NETDEV WATCHDOG: CPU: 0: transmit queue 0 timed out 5600 ms
[ 5154.806532] igc 0000:01:00.0 green0: Register Dump
[ 5154.806536] igc 0000:01:00.0 green0: Register Name   Value
[ 5154.806542] igc 0000:01:00.0 green0: CTRL            181c0641
[ 5154.806548] igc 0000:01:00.0 green0: STATUS          40680683
[ 5154.806554] igc 0000:01:00.0 green0: CTRL_EXT        10000040
[ 5154.806559] igc 0000:01:00.0 green0: MDIC            1805dde1
[ 5154.806564] igc 0000:01:00.0 green0: ICR             00000081
[ 5154.806569] igc 0000:01:00.0 green0: RCTL            04408022
[ 5154.806578] igc 0000:01:00.0 green0: RDLEN[0-3]      00001000 00001000 00001000 00001000
[ 5154.806588] igc 0000:01:00.0 green0: RDH[0-3]        0000007b 00000034 000000e4 00000028
[ 5154.806598] igc 0000:01:00.0 green0: RDT[0-3]        0000007a 00000033 000000e3 00000027
[ 5154.806607] igc 0000:01:00.0 green0: RXDCTL[0-3]     02040808 02040808 02040808 02040808
[ 5154.806617] igc 0000:01:00.0 green0: RDBAL[0-3]      ffffb000 ffffa000 ffff9000 ffff8000
[ 5154.806627] igc 0000:01:00.0 green0: RDBAH[0-3]      00000000 00000000 00000000 00000000
[ 5154.806633] igc 0000:01:00.0 green0: TCTL            a503f0fa
[ 5154.806642] igc 0000:01:00.0 green0: TDBAL[0-3]      fffff000 ffffe000 ffffd000 ffffc000
[ 5154.806651] igc 0000:01:00.0 green0: TDBAH[0-3]      00000000 00000000 00000000 00000000
[ 5154.806660] igc 0000:01:00.0 green0: TDLEN[0-3]      00001000 00001000 00001000 00001000
[ 5154.806670] igc 0000:01:00.0 green0: TDH[0-3]        0000002c 00000034 00000016 0000005e
[ 5154.806679] igc 0000:01:00.0 green0: TDT[0-3]        0000002c 00000034 00000016 0000005e
[ 5154.806689] igc 0000:01:00.0 green0: TXDCTL[0-3]     02100108 02100108 02100108 02100108
[ 5154.806694] igc 0000:01:00.0 green0: Reset adapter
[ 5158.602317] igc 0000:01:00.0 green0: NIC Link is Up 2500 Mbps Full Duplex, Flow Control: RX/TX
...

If I start a big Download from my a computer on my orange Network:
IPFIRE Ethernet Controller I225-V 1Gbps Nuc 1Gbps (Linux)

...
[ 4199.811950] igc 0000:03:00.0 orange0: NETDEV WATCHDOG: CPU: 0: transmit queue 3 timed out 5379 ms
[ 4199.812277] igc 0000:03:00.0 orange0: Register Dump
[ 4199.812286] igc 0000:03:00.0 orange0: Register Name   Value
[ 4199.812294] igc 0000:03:00.0 orange0: CTRL            181c0641
[ 4199.812302] igc 0000:03:00.0 orange0: STATUS          40280683
[ 4199.812309] igc 0000:03:00.0 orange0: CTRL_EXT        10000040
[ 4199.812317] igc 0000:03:00.0 orange0: MDIC            180a7800
[ 4199.812324] igc 0000:03:00.0 orange0: ICR             00000081
[ 4199.812331] igc 0000:03:00.0 orange0: RCTL            04408022
[ 4199.812342] igc 0000:03:00.0 orange0: RDLEN[0-3]      00001000 00001000 00001000 00001000
[ 4199.812354] igc 0000:03:00.0 orange0: RDH[0-3]        00000026 00000087 000000f9 00000019
[ 4199.812607] igc 0000:03:00.0 orange0: RDT[0-3]        00000025 00000086 000000f8 00000018
[ 4199.812621] igc 0000:03:00.0 orange0: RXDCTL[0-3]     02040808 02040808 02040808 02040808
[ 4199.812632] igc 0000:03:00.0 orange0: RDBAL[0-3]      ffffb000 ffffa000 ffff9000 ffff8000
[ 4199.812644] igc 0000:03:00.0 orange0: RDBAH[0-3]      00000000 00000000 00000000 00000000
[ 4199.812652] igc 0000:03:00.0 orange0: TCTL            a503f0fa
[ 4199.812661] igc 0000:03:00.0 orange0: TDBAL[0-3]      fffff000 ffffe000 ffffd000 ffffc000
[ 4199.812672] igc 0000:03:00.0 orange0: TDBAH[0-3]      00000000 00000000 00000000 00000000
[ 4199.812683] igc 0000:03:00.0 orange0: TDLEN[0-3]      00001000 00001000 00001000 00001000
[ 4199.812696] igc 0000:03:00.0 orange0: TDH[0-3]        000000e0 000000e6 00000011 00000064
[ 4199.812708] igc 0000:03:00.0 orange0: TDT[0-3]        000000e0 000000e6 00000011 00000064
[ 4199.812731] igc 0000:03:00.0 orange0: TXDCTL[0-3]     02100108 02100108 02100108 02100108
[ 4199.812740] igc 0000:03:00.0 orange0: Reset adapter
[ 4203.392356] igc 0000:03:00.0 orange0: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
...

If I run IPERF3 for 10 minutes on the Green Orange Network its solid and never hiccups.
From the test machines above I get ~760MBits/sec

If I run I perf3 on the IFFire machine <Orange Direct Cable Connection 1Gbps> Orange Server ~940MBits/sec its solid.

Running speedtest from IPFIRE I am getting between 900 and 1050Mbits/s Down and 95Mbits/s Up depending on time of day.

Hello,

I cannot confirm that this is a general issue. I am using the same chipset in a couple of places and it is rock-solid.

Is it maybe getting too hot or has some power supply issues? You could try updating to Core Update 197 which has a new kernel to see if there are any stability fixes in there that solve this.

-Michael

1 Like

So grabbed my old/standby IPFire device running some 186, plugged it in and everything worked great could not get it to give reset but then all connections ran at 1Gbps. Even upgraded to 196 and all was still well all day.

Took problematic device, put a new SSD in it since the old one was some wonky make and I had bought a replacement some time ago. Installed Windows 10 to see if I could reproduce any issues it promptly upgraded its self to Windows 11, so that was hours lost before I could test it properly and measure the throughput with the newest Intel drivers.

I could not get it to misbehave but also could not get anywhere near the 2+Gbps speeds I get under linux. I reinstalled IPfire 196, configured it loaded in the backup I took. Ran tests all day with red connected to a dummy server with 2.5Gbps interface and Green connected to a laptop with 2.5 Gbps interface and let it run. Tried to get it to over heat too by running Stress during the uploads/download, No problems..

Just plugged it into the internet cable modem and network proper, all the tests I have run so far have worked flawlessly. So I have no idea what was wrong.

  1. New SSD even though there were no signs this was necessary.
  2. Installed Windows, maybe installing the latest Intel drivers corrected something?
  3. Reinstalled 196 afresh, maybe the various upgrades left something behind or some config was problematic?

Who knows, maybe the communities positive thoughts were what fixed it.

Thanks.