IPFire slow when download > 1GB files

pslpsl · 6 July 2023 10:17

I observed similar issue today with IPFire 2.27 (x86_64) - Core-Update 175. I cannot explain it, not easy to replicate and I do not remember similar issue with IPcop 1.4.21 (and that box was low spec, i686 type CPU, 512MB of SDRAM…)

I started to download several larger files (zip, mp4) from gdrive (Google drive), I cannot share links because those files are not public; 4 files with size about 1-3GB. The first thing I noticed was that I was not able to open new URL in my browser. Then I tried to connect to IPfire web interface but I was not able to do it. Then I tried to login with ssh but again, I was not able to do it (but this could be other mysterious issue - I receive error debug1: rekey after 4294967296 blocks and I have it on this box when I tried to connect to IP fire before; sometimes I can connect, sometimes I cannot connect, I am not sure who is troublemaker in this case). In summary, my local network was not working until I downloaded files. Once huge files were downloaded, I was able to continue my work. I have no idea why this download affected IPfire gateway so badly. It runs transparent web proxy, it could be related. On the other side, downloads were HTTPS protocol, these should be ignored by HTTP proxy… Downloads were requested from desktop PC with Linux Mint 19.3 that was connected to LAN (GREEN).

I can generate similar heavy traffic with torrent downloads but it doesn’t “disable” IPfire, I see high peak in IRQ requests but I still can access IPfire web interface, it is just slower…

My IPfire runs at mini PC Fujitsu S900. It has two LAN cards based on Realtek chipset (RTL8111/8168/8411). Internet connectivity is about 6 MBps (60mbps) - it is “VDSL 100/20 Mbps”.

Here are three screenshots documenting situation. Notice high IRQ spike in the system graph that is related to RED traffic on the second screenshot. The last screenshot only shows, that there was no memory issue on my gateway.

This is output from irqtop for torrent download:

irqtop | total: 3183322107 delta: 50565 | ipfire.home | 2023-07-06 12:04:19+02:00

         cpu0
  %irq: 100.0
%delta: 100.0

          IRQ         TOTAL         DELTA NAME                                                                                              
           27     921935054         22808 PCI-MSI 524288-edge red0
           28     886393534         23092 PCI-MSI 2097152-edge green0
          LOC     809201049          3546 Local timer interrupts
          IWI     293282358           677 IRQ work interrupts
           20     246260120           438 IO-APIC 20-fasteoi ath9k
           19      26187966             4 IO-APIC 19-fasteoi ahci[0000:00:11.0]
          NMI         27571             0 Non-maskable interrupts
          PMI         27571             0 Performance monitoring interrupts
          MCP          5949             0 Machine check polls
           16           632             0 IO-APIC 16-fasteoi snd_hda_intel:card1
...

irqtop for “idle” gateway:

irqtop | total: 3195200048 delta: 2357 | ipfire.home | 2023-07-06 12:20:59+02:00

         cpu0
  %irq: 100.0
%delta: 100.0

          IRQ         TOTAL         DELTA NAME                                                                                              
           27     927181310           369 PCI-MSI 524288-edge red0
           28     891712486           370 PCI-MSI 2097152-edge green0
          LOC     810142672           800 Local timer interrupts
          IWI     293501095           431 IRQ work interrupts
           20     246398967           387 IO-APIC 20-fasteoi ath9k
           19      26201363             0 IO-APIC 19-fasteoi ahci[0000:00:11.0]
          NMI         27634             0 Non-maskable interrupts
          PMI         27634             0 Performance monitoring interrupts
          MCP          5952             0 Machine check polls
           16           632             0 IO-APIC 16-fasteoi snd_hda_intel:card1
            0           159             0 IO-APIC 2-edge timer
...

pslpsl · 13 July 2023 19:23

I can replicate the issue. I just start download of bigger file, like Ubuntu Desktop ISO. The issue is that there are many running processes on gateway /srv/web/ipfire/cgi-bin/getrrdimage.cgi and these processes are just CPU hungry and load on gateway goes high; result is that GUI is not working. Not good… I will attach a screenshot of top running on gateway when big file is downloaded from the internet.

current_user · 15 July 2023 11:58

is pppoe involved here
like: Minimum CPU Performance for a Gigabit WAN Link via PPPoE?

pslpsl · 17 July 2023 15:57

Yes, I have VDSL2, it is PPPoE and it implies pppoe…

I opened another thread because I noticed that a lot of my low-end CPU power is consumed by script /srv/web/ipfire/cgi-bin/getrrdimage.cgi that can run several times, it depends how many WUI connections are opened to IPfire gateway…

cwok · 29 July 2023 19:13

if you are using PPPoE you may want to look at receive packet steering. It does not save a slow cpu, but it helps, if all cores were not used. In my testing atom C2758 with PPPoE on a gigabit connection went from 200mbit to about 600mbit including IPS.
you can check with:
“cat /sys/class/net/ppp0/queues/rx-0/rps_cpus”
response on 8 thread cpu will be something like:
0f

this is a bitmask for cpu threads, on 8 thread cpu you want to set it to ff with:
“echo ff > /sys/class/net/ppp0/queues/rx-0/rps_cpus”

now the bad thing, this has to be done everytime after the PPPoE connects/reconnects, i was too lazy to find out how to make this permanent.

cfusco · 31 July 2023 13:19

You can add the command to the rc.local script, which is the final script executed during the IPFire boot process.

current_user · 31 July 2023 20:57

@cwok
excellent
almost doubled the bandwith

@cfusco
reconnect ≠ reboot

cfusco · 1 August 2023 08:04

Do you need to issue that command at every reconnect event?

current_user · 1 August 2023 09:30

cwok · 9 August 2023 08:25

Yes, but my connection gets mostly interrupted by restart after update/upgrade.

And rc.local does not work reliably, as sometimes the PPPoE connection starts after rc.local is run.

eth0 · 17 August 2023 06:35

@pslpsl I guess you have the Fujitsu Futro S900 with the AMD Bobcat G-T44R CPU which is only single core. But even with one core there shouldn’t be such a blockage, maybe the RRDtool CGI-script can be optimized for single core CPUs?

pike_it · 17 August 2023 13:29

Why you think so?
IpFire, even with “hyperthreading” disabled, take advantage as any current OS from multiple cores.
Multi core for common people started with Athlon 64 x2 and Pentium D in 2005; since than, any OS scheduler was refined then rebuilt considering/assuming multi-core CPUs.
In currenty days, mono-core CPUs are not only slower, but at the same benchmark rate, penalyzing any OS/software that will run (probably except consoles and some other simpler things).

If is confirmed that CPU (AMD Bobcat G-T44R) it’s far more power efficient than 3 years older Pentium IV
https://www.cpubenchmark.net/compare/266vs1067/AMD-G-T44R-vs-Intel-Pentium-4-2.40GHz
Neverthless, both are not enough for almost any internet speed (7mbit down), expecially if “loaded” from PPPoE client process.

eth0 · 17 August 2023 14:02

I meant the IRQ interrupts, even with one core it should be distributed more evenly and the download shouldn’t block other processes like the WUI. But I think it is some bug in the RRDtool.

In the old Astaro SG firewall there was also a bug in RRDtool reported recently:

It seems under certain cirtumstances RRDtool can get in some loop which causes high CPU usage. I guess only the developers can really find the cause.

jon · 17 August 2023 17:18

maybe try this to make sure it is RRD

# press q to exit top
top -c -p $(pgrep -d',' -f collectd)

pike_it · 17 August 2023 20:03

For the sake of my bad memory, I looked for the release date of latest ipcop 1.4. I did not found that, only the copyright declare for installation manual. 2009.
And for a box dated 2011, IPCop 1.4 would run nicely. I was using M0n0wall with 64mb of ram, during that days, and for 0.6/4mbit connection was great.

Considering the ramsize (512mb is not that much for IPFire, developers reccomend 1gb at least) and the CPU benchmark i posted, IMVHO an hardware upgrade is advisable.
A thin client with this APU (the first in the comparison)
AMD Embedded G-Series GX-420GI Radeon R7E vs AMD G-T44R [cpubenchmark.net] by PassMark Software
can be found.
Need only some tinkering for install adapter like these

for have more lan cards than one.