I226-V rev 04 rate adaptation

Hardware
1

Hello,
I am seeking an explanation + additional tweaks for the below improvement after stopping I226-V rev 04 to do 2,5G speed negotiation.

My box has 6 x Ethernet controller [0200]: Intel Corporation Ethernet Controller I226-VV [8086:125c] (rev 04)
Subsystem: Intel Corporation Device [8086:0000]
Kernel driver in use: igc
Kernel modules: igc

The box has Intel N100 CPU, 8 GB RAM DDR5 and 128 GB nVme disk - the box is manufactured to be used as router / FW.

My ISP uses FTTH & ONT (some Huawei ONT) as standard config. My subscription is 500 Mb/s simetric speed. Connection is PPoE.

One of above I226-V is connected to 1Gb/s port from the ONT that my ISP uses to give me internet.

The ipfire box has all security features powered on: geoIP, IPS, RPZ, etc.

I noticed that after telling red0 to stop advertising 2,5G the upload speed increased by 15% - finally reaching and sometimes bypassing the ISP speed from subscription!

Before this change, my speed was constantly 470-480 Mb/s upload.

What I’ve changed: rc.local has this line added

# Intel I226-V: allow only 1G for red0 port, disable 2.5G
/sbin/ethtool -s red0 autoneg on advertise 0x20

Result:

 ethtool red0
Settings for red0:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
                                2500baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  1000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 1000Mb/s
        Duplex: Full
        Auto-negotiation: on
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: internal
        MDI-X: off (auto)
        Supports Wake-on: pumbg
        Wake-on: g
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes

Reading some materials about the I226-V I have learned that this card had some past problems with “TX scheduling / flow control”

Question - anybody tweaking the I226-V to solve past problems that reduced the throughput?

Some AI suggest that I optimized TX/RX ring buffers (TX que is filled faster when using 2,5G, generating TX stalls - TX pauses) and also the I226-V no longer transmit toward Huawei ONT the 2,5G pause frames so ONT no longer loses time trying to parse those.

My IPS hosts a speedtest server - and distance between my location and ISP main datacenter is around 4 KM - so I am pretty close to that server, plus the provider uses only fiber in my city.

Retrieving ``speedtest.net`` server list…
Selecting best server based on ping…
Hosted by My-ISP [3.45 km]: 15.781 ms
Testing download speed…
Download: 483.84 Mbit/s
Testing upload speed…
Upload: 515.19 Mbit/s

2

Hello H&M,

what is the problem that you are trying to solve here?

3

Hello Michael,

Initially I had a flapping problem between my Meraki MR33 AP and I226-v card.

That was solved by telling I226 to stop advertising 2,5G : after that, Meraki stopped flapping.

Then, I thought to do the exact same setting for the card used for red0 - reason being that ISP ONT had a 1Gb network card in it. I never had problems with red0 connection - I226 seems to work well with the Huawei ONT.

But, I noticed that turning off 2,5G for red0 I get a bit more upload speed.

Searching for TX problems on I226 I found that this card had a handful of problems, all being addressed by Intel via software drivers - although some voices claim that problems are HW based.

Therefore, I have a better uplink speed now (TX improved). Explanation AI provided was that by disabling 2,5G some special Ethernet frames ?) are no longer sent by I226, frames that 1Gb card from ONT do not understand. So ONT no longer loses times to parse and discard those frames.

I am wondering if on top of this setting there are any other tweaks to improve the performance of the card when connected to a 1Gb card?

I have little hopes - many voices claim that HW in I226 is the problem and any software on top of it will not make miracles.

Thank you.

4

How did you measure speeds? You should try from local machine to local machine so you can see if issue is with your local setup or with ISP devices.

I also have router with 6 x I226-V Nics but don’t remember if it having similar issues and my CPU is slower Celeron N5105.

Default IPFire settings should be enough for 2.5Gbps speeds. I get 2400Mbps in and out from my server Orange ↔ Green zones.

image
image637×366 29.1 KB

But with iperf I get only around 500-1000Mbps when IPS is enabled. Disabling IPS from green and orange speed goes to stable 2.35 Gbps.

IPS is these days much more heavy for cpu since new version of Suricata scans more traffic now.

5

Internet speed is measured via speedtest-cli : my ISP hosts an speedtest server in their datacenter. My ipfire box is at 4 Km distance from that server and ISP uses only fiber optic between my location and that datacenter.
I posted this in HARDWARE section because this is NOT a SW problem IMHO - based on some articles about I226-V rev 04 the card had some severe problems that Intel tried to solve via driver (software solution for sub-optimal HW design to be polite)

To be clear: I226-V connected directly to Intel Corporation Ethernet Controller I225-V (rev 03) negociate 2,5 G speed and iperf3 between them shows 2,35-2,42 Gbits/sec (I have one N5105 box with I225-V rev 03 firmware-version: 1057:8754 connected to N100 box that has I226-v rev 04 firmware-version: 2013:8877, each running ipfire and iperf3)

As far as I could find on internet each I225/I226 runs the latest firmware.

But! The problem is NOT Intel I225-I226 link but rather I226 to Meraki or to Huawei ONT (the Fiber to copper media convertor my ISP)

Meraki and Huawei have 1Gb/s NIC and Meraki card was having some hard time to keep the 1Gb/s negociated speed while I226 was allowed (!) to megociate 2,5G

As soon as I turned I226 2,5G negotiation off the Meraki NIC got stable on 1Gb/s link (i.e. Meraki no longer logged NIC speed changes)

Seeing this improvement I thought that such improvement will help also Huawei ONT link so I applied it to the port where ONT is connected.

I discovered that speedtest-cli reported a 10-15% higher speed after that - and constantly over the subscription I paid (!!).

At this moment I am almost sure that disabling 2,5G in I226 cards connected to 1Gb cards did improved the stability of the line (for Meraki) and also the Upload speed (for ONT).
Question is: why that happened?
Second aspect: is there any tweak I can apply to those I226 cards to improve even better the links toward 1Gb NICs. Again: the I226 is a blast when connected to I225 but I feel it is glitchy when connected to other, lower speed cards!

Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by MyIPS (MyCity) [3.45 km]: 18.681 ms
Testing download speed................................................................................
Download: 480.68 Mbit/s
Testing upload speed......................................................................................................
Upload: 540.97 Mbit/s

I225 to I226 link (3m cat6 cable between them): 2 IPFIRE boxes, one with N5105 CPU and I225 rev 03 running iperf3 as server and the other with N100 CPU and I226 rev04 runing iperf3 client. Both boxes have 8GB RAM and 128 GB nVME disks, both being manufactured to be used as FW/Routers. Each Intel card has latest firmware (as far as I can tell there is no newer firmware compared with teh one ethtool retrieves)

Iperf3 Server logs:

[  5]  16.00-17.00  sec  46.8 MBytes   392 Mbits/sec
[  8]  16.00-17.00  sec  93.4 MBytes   783 Mbits/sec
[ 10]  16.00-17.00  sec  46.9 MBytes   393 Mbits/sec
[ 12]  16.00-17.00  sec  93.5 MBytes   784 Mbits/sec
[SUM]  16.00-17.00  sec   280 MBytes  2.35 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-17.00  sec   819 MBytes   404 Mbits/sec                  receiver
[  8]   0.00-17.00  sec  1.59 GBytes   802 Mbits/sec                  receiver
[ 10]   0.00-17.00  sec   820 MBytes   405 Mbits/sec                  receiver
[ 12]   0.00-17.00  sec  1.59 GBytes   802 Mbits/sec                  receiver
[SUM]   0.00-17.00  sec  4.78 GBytes  2.41 Gbits/sec                  receiver
iperf3: the client has terminated

What iperf3 client logged:


[  5]  16.00-17.00  sec  47.6 MBytes   400 Mbits/sec    0    277 KBytes
[  7]  16.00-17.00  sec  93.4 MBytes   783 Mbits/sec    0    263 KBytes
[  9]  16.00-17.00  sec  47.5 MBytes   398 Mbits/sec    0    274 KBytes
[ 11]  16.00-17.00  sec  93.5 MBytes   784 Mbits/sec    0    260 KBytes
[SUM]  16.00-17.00  sec   282 MBytes  2.37 Gbits/sec    0
^C- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  17.00-17.57  sec  26.0 MBytes   386 Mbits/sec    0    288 KBytes
[  7]  17.00-17.57  sec  52.1 MBytes   774 Mbits/sec    0    266 KBytes
[  9]  17.00-17.57  sec  26.1 MBytes   388 Mbits/sec    0    286 KBytes
[ 11]  17.00-17.57  sec  52.9 MBytes   785 Mbits/sec    0    263 KBytes
[SUM]  17.00-17.57  sec   157 MBytes  2.33 Gbits/sec    0




[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-17.57  sec   821 MBytes   392 Mbits/sec    0            sender
[  5]   0.00-17.57  sec  0.00 Bytes  0.00 bits/sec                  receiver
[  7]   0.00-17.57  sec  1.59 GBytes   777 Mbits/sec    0            sender
[  7]   0.00-17.57  sec  0.00 Bytes  0.00 bits/sec                  receiver
[  9]   0.00-17.57  sec   822 MBytes   393 Mbits/sec    0            sender
[  9]   0.00-17.57  sec  0.00 Bytes  0.00 bits/sec                  receiver
[ 11]   0.00-17.57  sec  1.59 GBytes   777 Mbits/sec    0            sender
[ 11]   0.00-17.57  sec  0.00 Bytes  0.00 bits/sec                  receiver
[SUM]   0.00-17.57  sec  4.78 GBytes  2.34 Gbits/sec    0             sender
[SUM]   0.00-17.57  sec  0.00 Bytes  0.00 bits/sec                  receiver
iperf3: interrupt - the client has terminated by signal Interrupt(2)

N100 box with I226-V rev 04

ethtool -i green0p3
driver: igc
version: 6.12.41-ipfire
firmware-version: 2013:8877
expansion-rom-version:
bus-info: 0000:05:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

ethtool green0p3
Settings for green0p3:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
2500baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
2500baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 2500Mb/s
Duplex: Full
Auto-negotiation: on
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
MDI-X: off (auto)
Supports Wake-on: pumbg
Wake-on: g
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes

N5105 with I225-v rev 03

ethtool -i green0p0
driver: igc
version: 6.12.41-ipfire
firmware-version: 1057:8754
expansion-rom-version:
bus-info: 0000:04:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

ethtool green0p0
Settings for green0p0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
2500baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
2500baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 2500Mb/s
Duplex: Full
Auto-negotiation: on
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
MDI-X: off (auto)
Supports Wake-on: pumbg
Wake-on: g
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes