The cable from the ISP router into the ipFire device “red” network port has 300Mbps download speed tested and confirmed. The cable from the “green” network port tests ~30Mbps download speed. CPU 60% to 99% free (avg 97%) over the past week and Memory is consistently 80% free on ipFire device. NICs Reaktek RTL 8111/8168/8411PCIe Gb devices.
Turned off IPS and IP Blocklists. Tried different network cables. Thoughts? Ideas? Questions? I’m stumped.
I would repeat here same observations I made in the past. So, I prefer to link one of the treads. Try those tests and let us know how it goes. Do not forget to look at the cpu with top, while you do an iperf3 test. Also tail -f /var/log/messages while doing iperf3.
Findings:
Running this command from a green-side client machine: iperf3 -t 120 -c <ip of **green** NIC> shows 866 Mbits/sec sender and 865 Mbits/sec reciever. top command running on the ipFire machine shows CPU0 maxes out. tail command of the messages file is clean while iperf3 -c is running.
iperf3 -t 120 -c <ip of **red** NIC> shows 906 Mbits/sec sender and 906 Mbits/sec reciever. top command running on the ipFire machine shows CPU0 and CPU1 are both busy. Between them, they total ~130%. CPU0 is busier than CPU1. tail command of the messages file is clean
I’m not sure what to make of these results.
that’s your bottle neck, I think. The card needs the CPU to transfer packets and only one core is used. Probably, it gets worst when also the other card is engaged at the same time, as both will need CPU cycles, both cores 0 and 1 get engaged to near max (0 for sure will be maxed out) the queues are not cleared fast enough and the firmware of the cards drops the speed.
You need cards that have their own silicon for queue management, like the intel i210 that can sustain two queues in both direction without needing the CPU. That’s the card of the APU2, which is my machine; when I do the iperf3 test, the cpu cores are very far from being saturated.
QOS and IPS disabled and settings saved? Full duplex GB set in network settings on green client?
Yes, BUT the CPU0 is maxing out under iperf3 at transfer speeds much higher than I am getting when running a speed test at fast.com, for example. Running a speed test from a machine connected to the green firewall port measures the speed at <30 Mb/s and the CPU0 load is less than 10%. That tells me that effectively the CPUs and the NICs are not the bottleneck.
I don’t think so, you cannot rely on the data shown by the web user interface. The granularity of that measurement is not the same that you get from top in real time.
I believe that if you try your speedtest while measuring the cpu activity using top, as you did for iperf3, you should see briefly the same congestion, before the drop of speed, as both cards will be engaged at the same time, the queues will not be cleared, and the speed will drop reducing the CPU engagement.
If you are looking from the beginning with high granularity, you should see this happening, but if you miss the CPU maxing out before the drop of the speed, you won’t see this reflected in IPFire logs.
I have an SSH connection to the ipFire device, and I’m looking at the ‘top’ command output while downloading a 1.5 GB file from the internet. CPU0 and CPU1 are both working, but not hard. CPU0 is around 12% while downloading and CPU1 is about 6%. At no time are any of the CPUs working hard.
If you are downloading from the internet you get a maximal bandwidth of the ‘weakest’ segment of the route your data flow. Therefore you seldomly reach nearly the capacity of your NIC, and the CPU(s) aren’t really loaded with packet transport.
Then, I am out of ideas. However, I think it is possible that your green side ethernet card switches from gigabit to megabit speed while you download your files. Firmware malfunctioning? If you ever figure it out, please let us know.
I would do another two tests, just for satisfying a certain curiosity. First, an internet speed test from IPFire machine using speedtest-cli package. The second, to do an iperf3 test at the same time for both cards. Just wondering if you create two band-saturating streams on both interfaces, how your system will behave.
John,
Please share the link to your fireinfo profile.
Running speedtest on the ipFire device gives the slow download ~30Mbs speeds. Does this mean that it’s a hardware (driver?) problem on the ipFire device? According to the lights on the ISPs router, there’s a Gbs connection between the router and the ipFire device.
Re BBitsch comment, connecting a laptop to the ISPs router, speed test shows download ~300 Mbs. Connecting the same laptop to the ipFire device and running the same speed test, download speed is ~30Mbs.
Okay, that means there’s really a problem with IPFire’s NIC or driver.
I’ve loaded ipFire CU 172 on another box with Intel I211 Gb NICs (as reported in the ipFire setup module). Connecting it to the ISP router and running speedtest-cli on the ipFire device I get ~60 Mbs download speed. Better, but it’s still an 80% haircut on a 300 Mbs circuit.
In another location, I have an ipFire device with Intel I210 Gb NICs deployed. It is getting speedtest-cli download speeds that are 85 Mbs where the ISP is providing 100 Mbs service.
If the I210 is working but the i211 is not, what does that say about the drivers? Is it safe to say that ipFire is correctly identifying the make and model of the NICs and using the right drivers?
At another location where I have an ipFire device connected and getting full speed (~100Mbs running speedtest-cli) the device has Intel I211 NICs. I’m going to do a fresh install on one of the underperforming devices and see if that helps.
I backed up one of the underperforming devices and then did a reload of ipFire CU 172. Speed test on my laptop was ~300 Mbs. Restored the ipFire backup and the speed test stayed at ~300 Mbs. Problem solved!
ipFire: When in doubt, reload.
Thanks.
Congratulations for solving the problem. This was a learning experience, I would not have guessed reinstalling a linux system was the way to go. Thanks for reporting it.
Keep in mind for future situations that also the QoS might influence the traffic speed. Also, i211 has only one bidirectional queue while i210 has two.