IPFire Kernel Optimizations for 10Gb WAN + LAN

pscar13 · 16 December 2025 18:10

After the not very constructive discussion about 10Gb WAN problems, I looked into the differences in Linux kernel configuration between VyOS and IPFire.

I extracted some parameters and made a patch for the Kernel configuration file

--- ./kernel.config.x86_64-ipfire.ori	2025-12-16 12:03:23.219641848 +0100
+++ ./kernel.config.x86_64-ipfire	2025-12-16 12:21:38.711686554 +0100
@@ -191,6 +191,8 @@
 CONFIG_CC_NO_STRINGOP_OVERFLOW=y
 CONFIG_ARCH_SUPPORTS_INT128=y
 CONFIG_SLAB_OBJ_EXT=y
+CONFIG_NUMA_BALANCING=y
+CONFIG_NUMA_BALANCING_DEFAULT_ENABLED=y
 CONFIG_CGROUPS=y
 CONFIG_PAGE_COUNTER=y
 CONFIG_CGROUP_FAVOR_DYNMODS=y
@@ -442,7 +444,7 @@
 # CONFIG_X86_5LEVEL is not set
 CONFIG_X86_DIRECT_GBPAGES=y
 # CONFIG_AMD_MEM_ENCRYPT is not set
-# CONFIG_NUMA is not set
+CONFIG_NUMA=y
 CONFIG_ARCH_SPARSEMEM_ENABLE=y
 CONFIG_ARCH_SPARSEMEM_DEFAULT=y
 CONFIG_ILLEGAL_POINTER_VALUE=0xdead000000000000
@@ -606,6 +608,10 @@
 CONFIG_ACPI_NHLT=y
 CONFIG_ACPI_NFIT=m
 # CONFIG_NFIT_SECURITY_DEBUG is not set
+CONFIG_ACPI_NUMA=y
+CONFIG_AMD_NUMA=y
+CONFIG_X86_64_ACPI_NUMA=y
+CONFIG_NODES_SHIFT=10
 CONFIG_HAVE_ACPI_APEI=y
 CONFIG_HAVE_ACPI_APEI_NMI=y
 CONFIG_ACPI_APEI=y
@@ -1061,6 +1067,7 @@
 CONFIG_ARCH_WANT_OPTIMIZE_DAX_VMEMMAP=y
 CONFIG_ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP=y
 CONFIG_HAVE_GUP_FAST=y
+CONFIG_NUMA_KEEP_MEMINFO=y
 CONFIG_MEMORY_ISOLATION=y
 CONFIG_EXCLUSIVE_SYSTEM_RAM=y
 CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
@@ -1102,7 +1109,7 @@
 CONFIG_HAVE_SETUP_PER_CPU_AREA=y
 CONFIG_CMA=y
 # CONFIG_CMA_SYSFS is not set
-CONFIG_CMA_AREAS=8
+CONFIG_CMA_AREAS=19
 CONFIG_GENERIC_EARLY_IOREMAP=y
 # CONFIG_DEFERRED_STRUCT_PAGE_INIT is not set
 # CONFIG_IDLE_PAGE_TRACKING is not set
@@ -6976,7 +6983,7 @@
 
 CONFIG_IOMMU_DEFAULT_DMA_STRICT=y
 # CONFIG_IOMMU_DEFAULT_DMA_LAZY is not set
-# CONFIG_IOMMU_DEFAULT_PASSTHROUGH is not set
+CONFIG_IOMMU_DEFAULT_PASSTHROUGH=y
 CONFIG_IOMMU_DMA=y
 CONFIG_IOMMU_SVA=y
 CONFIG_IOMMU_IOPF=y

So I recompiled a modified CU199 version based on these new parameters
And I get the following results :

With the modified version CU199 (initial installation Red + Green)

F:\tools\iperf3>iperf3 -c mrs.bbr.iperf.bytel.fr -R -p9201
Connecting to host mrs.bbr.iperf.bytel.fr, port 9201
Reverse mode, remote host mrs.bbr.iperf.bytel.fr is sending
[  5] local 192.168.20.100 port 12578 connected to 31.33.13.121 port 9201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.01   sec   936 MBytes  7.81 Gbits/sec
[  5]   1.01-2.00   sec   955 MBytes  8.04 Gbits/sec
[  5]   2.00-3.01   sec   954 MBytes  7.97 Gbits/sec
[  5]   3.01-4.00   sec   942 MBytes  7.95 Gbits/sec
[  5]   4.00-5.01   sec   938 MBytes  7.80 Gbits/sec
[  5]   5.01-6.01   sec   943 MBytes  7.87 Gbits/sec
[  5]   6.01-7.01   sec   934 MBytes  7.89 Gbits/sec
[  5]   7.01-8.00   sec   945 MBytes  7.94 Gbits/sec
[  5]   8.00-9.00   sec   927 MBytes  7.80 Gbits/sec
[  5]   9.00-10.01  sec   955 MBytes  7.97 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.05  sec  9.22 GBytes  7.88 Gbits/sec  102845             sender
[  5]   0.00-10.01  sec  9.21 GBytes  7.90 Gbits/sec                  receiver

iperf Done.

With the official CU199 version, and the same test conditions, I only obtained

F:\tools\iperf3>iperf3 -c mrs.bbr.iperf.bytel.fr -R -p9201
Connecting to host mrs.bbr.iperf.bytel.fr, port 9201
Reverse mode, remote host mrs.bbr.iperf.bytel.fr is sending
[  5] local 192.168.20.100 port 50321 connected to 31.33.13.121 port 9201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.01   sec   382 MBytes  3.17 Gbits/sec
[  5]   1.01-2.01   sec   384 MBytes  3.22 Gbits/sec
[  5]   2.01-3.00   sec   380 MBytes  3.21 Gbits/sec
[  5]   3.00-4.00   sec   386 MBytes  3.24 Gbits/sec
[  5]   4.00-5.00   sec   385 MBytes  3.24 Gbits/sec
[  5]   5.00-6.00   sec   386 MBytes  3.24 Gbits/sec
[  5]   6.00-7.01   sec   389 MBytes  3.23 Gbits/sec
[  5]   7.01-8.01   sec   385 MBytes  3.24 Gbits/sec
[  5]   8.01-9.01   sec   386 MBytes  3.24 Gbits/sec
[  5]   9.01-10.01  sec   385 MBytes  3.24 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.05  sec  3.77 GBytes  3.22 Gbits/sec    0             sender
[  5]   0.00-10.01  sec  3.76 GBytes  3.23 Gbits/sec                  receiver

iperf Done.

That’s more than double the download bandwidth.

These tests were performed with my 8Gbps Down/1Gbps Up internet connection on my mini PC
Qotom Q11032H6 S13 miniPC, equipped with a Twin Lake-N Core i3 N355 processor, 8GB RAM, 2 NICS Marvell AQC113 10 Gigabit LAN interfaces
My resources are limited to push these tests further, and my knowledge of the Linux kernel is insufficient to know if these modifications might negatively impact other systems.

If the IPFire development team could verify and test these settings on the different X86_64 platforms, would it be possible to use them in a future version?

I performed a complete reinstall from my CU198 backup + upgraded to CU199 (test).
I then performed another backup and reinstalled the modified CU199.
And I’m still getting good performance with SpeedTest, with Suricata enabled.

But not for iperf3 with suricata

F:\tools\iperf3>iperf3 -c mrs.bbr.iperf.bytel.fr -R -p9201
Connecting to host mrs.bbr.iperf.bytel.fr, port 9201
Reverse mode, remote host mrs.bbr.iperf.bytel.fr is sending
[  5] local 192.168.20.32 port 52464 connected to 31.33.13.121 port 9201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.01   sec  81.4 MBytes   673 Mbits/sec
[  5]   1.01-2.00   sec  83.2 MBytes   708 Mbits/sec
[  5]   2.00-3.00   sec  84.9 MBytes   712 Mbits/sec
[  5]   3.00-4.01   sec  86.4 MBytes   716 Mbits/sec
[  5]   4.01-5.01   sec  82.8 MBytes   697 Mbits/sec
[  5]   5.01-6.01   sec  87.4 MBytes   733 Mbits/sec
[  5]   6.01-7.01   sec  96.8 MBytes   812 Mbits/sec
[  5]   7.01-8.01   sec   106 MBytes   894 Mbits/sec
[  5]   8.01-9.00   sec   107 MBytes   898 Mbits/sec
[  5]   9.00-10.00  sec   106 MBytes   889 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.04  sec   927 MBytes   774 Mbits/sec    0             sender
[  5]   0.00-10.00  sec   922 MBytes   773 Mbits/sec                  receiver

iperf Done.

hvacguy · 17 December 2025 00:49

Thank you for never giving up.

Great find

arne_f · 17 December 2025 07:54

numa should only affect systems with more than one CPU socket which have seperated ram for every socket.

Can you test the other changes seperat?

Im also not sure about CONFIG_IOMMU_DEFAULT_PASSTHROUGH this is maybee a security risk…

pscar13 · 17 December 2025 09:32

I performed a general comparison of the parameters, without a detailed analysis of each one.
Rebuilding IPFire for each parameter is very time-consuming, and I don’t have a test platform to verify the different possible configurations.
I’ll leave that analysis to the specialists. If it raises security concerns, I trust you to identify them.

porkyle · 17 December 2025 12:29

I think IOMMU is mostly used for virtualisation when sharing device(s) to virtual machines. Don’t know if it has any usecase outside any virtualisation tasks. Atleast it makes GPU and other PCIe device passthru easier in proxmox.

nickh · 19 December 2025 11:24

Interesting article on tuning Linux for network speed at Fornax's Guide To Ridiculously Fast Ethernet ~ pixeldrain, but keep clear of IOMMU, it seems.

pscar13 · 19 December 2025 12:29

Thanks for all this information; it will certainly be useful for the development of IPFire.
As far as I’m concerned, I understand that NUMA doesn’t really affect my platform.

However, I can’t explain why, but the IOMMU options does have an impact.

I simply added these options to the boot order of my CU198, and my SpeedTest more than doubled, and iperf3 reached its maximum (with Suricata disabled).

# Enable iommu Passthrough in GRUB defaults
sed -i 's/^\(GRUB_CMDLINE_LINUX=".*\)"/\1 intel_iommu=on iommu=pt"/' /etc/default/grub

# Regenerate GRUB configuration
/usr/sbin/grub-mkconfig -o /boot/grub/grub.cfg

reboot

Again, I don’t know if these options have any impact on security on this version of the Linux kernel.

ms · 19 December 2025 12:35

Hello,

Well, let’s make this into a constructive one…

The NUMA changes won’t have any effect on your system because the i3 CPU that you are using won’t have more than one NUMA node.

As @arne_f has already said, this will introduce some security issues because the hardware would be able to access any memory it desires. Any compromised firmware and so on would be able to read security credentials.

With IPFire, we usually harden the OS against compromised hardware. Obviously it is not possible to do this 100%, because if the hardware is fully compromised, the OS cannot trust anything, but at the same time, there could only be some smaller parts of the hardware being vulnerable. With more and bigger firmware blobs loaded into network interfaces, WiFi modules, storage controllers and so on; with a huge UEFI/BIOS that cannot be audited at all, this seems to be a sensible choice on our behalf.

So why do you see such a big change on your system? It can only be explained that the CPU is quite poor with any kind of IOMMU operations. Whenever the driver wants to send or receive data from the NIC, it has to do some extra work if running in strict mode. That should normally have very little overhead (a few percent, if even). But if the IOTLB or the DMA pages don’t fit into the CPU’s caches, the CPU is spending a lot of time waiting for data. This is quite likely happening here. I don’t think that this is for example repeatable on an Intel Xeon CPU that is simply designed for heavy I/O load compared to the rather power-efficient i3 series.

The huge drop in IPS throughput can also be explained by the CPU having too small buffers and the Spectre/Meltdown measures having a particularly high toll.

Since this is all a runtime setting, you could try to boot the regular IPFire kernel and pass “iommu.passthrough=1” to the kernel command line. That will enable the passthrough mode for you (if you want to trust your hardware) and you should be able to see the faster benchmark results. That would also help us to rule out NUMA.

Although these guides are not wrong per se, they simply don’t apply in the general case. Someone is optimising for a very specific task and aims to bring up throughput (at all costs). Having good latency is at least as important as having good throughput, if not even more important, so we simply cannot take all these settings and apply them to IPFire unless you want to have broken up calls, and small packets being queued inside buffers for seconds.

IPFire is deployed in so many all-round situations. Therefore we are making compromises. IPFire is also running all sorts of hardware, and once again, that has to come with some hard choices that very often sacrifice performance.

The easiest way to keep things fast and secure at the same time would simply be to upgrade the hardware. A desktop-grade i3 CPU is not designed to push tens of gigabits of packets. It can do that, somewhat, but in this specific case it seems that we have to turn off all the security measures.

I would rather replace my hardware before I have my network compromised though…

pscar13 · 19 December 2025 12:39

Our answers crossed paths…

Michael Tremer:

pscar13:
CONFIG_IOMMU_DEFAULT_PASSTHROUGH=y
As @arne_f has already said, this will introduce some security issues because the hardware would be able to access any memory it desires. Any compromised firmware and so on would be able to read security credentials.

Thanks for the explanation. I understand the theoretical risk associated with DMA in passthrough mode.
I don’t know if it poses a real problem in my specific case.
And I understand that it can’t be recommended as the default setting for IPFire.

My only goal is to share my experiences with the community.

hvacguy · 19 December 2025 13:14

This has bin a very informative conversation.

It does explain a lot.

ms · 19 December 2025 14:57

Hehe, yes they did.

This is a very difficult question indeed. Maybe we can collect a little bit more feedback from other people with hopefully vastly different hardware, so that we can get a better feeling how many people are impacted by this and how hard the impact is going to be.

I believe that your specific hardware could be hit worst and is an outlier, but that should be confirmed by having more people provide some data if possible.

I don’t want IPFire to be generally half the speed of other alternatives, but I would always favourite better security over throughput.

I am very happy to hear that and hopefully from more people.

nickh · 19 December 2025 15:15

ms:

Although these guides are not wrong per se, they simply don’t apply in the general case. Someone is optimising for a very specific task and aims to bring up throughput (at all costs). Having good latency is at least as important as having good throughput, if not even more important, so we simply cannot take all these settings and apply them to IPFire unless you want to have broken up calls, and small packets being queued inside buffers for seconds.

IPFire is deployed in so many all-round situations. Therefore we are making compromises. IPFire is also running all sorts of hardware, and once again, that has to come with some hard choices that very often sacrifice performance.

The easiest way to keep things fast and secure at the same time would simply be to upgrade the hardware. A desktop-grade i3 CPU is not designed to push tens of gigabits of packets. It can do that, somewhat, but in this specific case it seems that we have to turn off all the security measures.

I would rather replace my hardware before I have my network compromised though…

TBH, I expected that sort of reply, to completely dismiss the article and that everything is already being done right in IPF. I presume you have previously considered all these tweaks in order to be able to dismiss them out of hand?

ms · 19 December 2025 15:22

I didn’t dismiss them straight away. I am just saying that our goals and the goals of the author are not always the same.

Looking at our own hardware, yes, we regularly perform a lot of tests and tweak a lot of knobs to get something extra out of the system. That might however not always translate to all other systems.

Just increasing throughout at any cost is an objective that I would personally object to. That won’t create a good network and IPFire has so many ways to actually give you a great networking experience, even on weaker commodity hardware. If you are looking for more throughput, you will have to upgrade as there is a ceiling to what we can get out of the entire system.

I would be happy to hear if anyone has made some discoveries about what could be changed to improve IPFire for all users. The emphasis here is on all users. It is a constant process and IPFire should never stand still.

In testing, we will soon be looking at Linux 6.18 (a test version is already available), and we will re-evaluate a lot of decisions because the kernel moves on, too.

pscar13 · 19 December 2025 15:41

IPFire has opted for Optimum security, which is more than commendable.

But as we see here, with ISPs increasing bandwidth at a low cost, this translates into a choice between security and performance.

Users who can’t afford enterprise-grade hardware are likely to turn to other solutions, even paid ones, as we’ve seen.

Perhaps we could offer this choice within the IPFire setup and security levels?

But I admit, this would require extensive testing and a huge amount of research.

It’s difficult to remain free

ms · 19 December 2025 16:26

This is a very dangerous statement. You deploy IPFire to have security. Obviously that will always somewhat harm your performance. The goal is to do that as little as possible, but the strong security features like the IPS don’t come for free.

Enterprise-grade hardware has different goals. Performance? Yes, of course. But there is also the goal to have no downtime, longevity, reliability to buy the same product again in case your business is expanding, etc.

Is there a lot of really bad hardware out there? Yes. But it won’t be possible to ever make that run actually smooth. The vendor has already made compromises to offer a very cheap product and therefore, the software cannot make up for what the hardware does not have to offer.

“Paid” solutions don’t actually use any different hardware. They just have customised their solutions to a very specific hardware. We also have that in stock in our store. That is exactly the reason why this exists. They will get the best out of IPFire, because we have put in the work. If you buy your own hardware, you will have to put that work in yourself potentially.

Generally, this won’t be possible. In this IOMMU case, changes could be made at runtime, but a lot of other options can only be compiled in. Offering multiple different versions with different flavour is just going to make this all crazy.

Yes. Hardware can be brilliant blackboxes. This is going to be a lot of work indeed.

No, this has nothing to do with free vs paid software. It is just a matter of whether you are putting the work in yourself or you buy it in. The choice is yours. And that is why we call it free software - as in freedom, and not beer.

Although we do collect donations so I can afford to buy myself a beer

porkyle · 20 December 2025 12:23

That’s nice to hear. I’ve been thinking getting 10G nic and with newer kernel Realtek rtl8127 would be compatible but I have no idea if upstream driver is better or worse than Realteks proprietary. Atleast upstream driver for rtl8125 is better in kernel 6.18 in my experience.

ag · 20 December 2025 12:48

Hi all.

It’s easy to focus on the bigger throughput numbers and assume that automatically means “better”, but an important detail seems to be overlooked here: stability.

In the “optimised” iperf3 result, the sender reports Retr: 102845. That means a very large number of packets had to be retransmitted. While the raw throughput looks much higher, this usually indicates packet loss somewhere.

Even if the latency look fine in a speed test, in real-world scenarios (VoIP, video calls, online gaming, live streaming), that level of retransmission would translate into jitter, pauses, or brief dropouts while TCP recovers.

So throughput alone doesn’t tell the whole story. Latency, packet loss, and stability matter just as much, especially for a firewall sitting in front of mixed, real-time traffic.

Thanks,
A G

pscar13 · 20 December 2025 17:39

Thanks for the insightful comment.

This was a one-off test intended to analyze differences in Linux kernel settings, and it does not aim to represent real-world traffic patterns.

The figure of 102,845 retransmissions needs to be interpreted in the context of the test itself: approximately 9.2 GB transferred in 10 seconds at sustained near-line-rate throughput. Despite this, the throughput remains stable throughout the test.

In practice, even multiple concurrent VoIP calls or live streams would not generate traffic patterns comparable to a single TCP iperf3 stream saturating the link. Assessing latency, jitter, or real-time performance would indeed require different and more appropriate tests.

To be clear, I am not trying to impose a conclusion or prove a point; I am simply sharing test results out of technical curiosity and in the interest of constructive discussion.

ms · 23 December 2025 12:41

We don’t ship any proprietary drivers in IPFire. There usually is no need to, they are much less tested than what is being included in the Linux kernel, vendors have uncoordinated release schedules and often lack behind the kernel’s development. So in essence it is a lot of time for a bad result.

We have a HCL, so in case you want to purchase some hardware and you are not sure, check there first whether it is being supported.

xperimental · 3 January 2026 09:13

I did a short test:

Test Server (red):

My first built server (very old by now – anno 2016)
Intel Xeon Processor E5-2699 v4
128GB DDR4
Mellanox ConnectX-5
Windows Server 2022 – iperf3

Test IPFire

New Test system
Intel Core Ultra 7 Prozessor 265K
96GB DDR5
Intel Ethernet Network Adapter X710 (for green + red)
IPFire 2.29 core update 199 testing @ stock - iperf3

Test Client (green)

VMware ESXi 8.0 U3
AMD EPYC 7282 (8 cores)
128GB DDR4 (16GB)
Dell QLogic 57810
Manjaro Linux 25.0.10 - iperf3

Test switch (green)

D-Link DGS-1250-28X

Test Client ←> Switch ←> IPFire (green): normal

Test IPFire ←> Server (red): normal

Test Client ←> Switch ←> IPFire ←> Server: bad

I repeated the test several times. The important last test always performed very bad!