I got a brand new APU4C4 Board for use as Router on my 500/200MBit FTTH. Due to performance issues with (OPN|PF)Sense and beeing more familiar with Linux I decided tu use IPFire on it.
Installation and configuration went fine. I’m missing some features but this is configurable via CLI for me and I’m fine with that.
Unfortunately I’ve some massive performance problems. With fresh installed IPFire and only a PPPoE Connection I got only 330-350MBit Downstream on multi connection. If I put a Linux Notebook directly to the Fiber-Modem and configure the PPPoE I got arround 510-530MBit.
I’ve read tons of documentations to improve the performance. Classical intel NIC tunig as described in many howtos won’t bring a significant improvement. Only pinning the IRQs of the NICS to a dedicated CPU-Core per NIC improves speed massively to 450-470MBit. But there are nevertheless 60-80MBit left. I’ve read that linux based Routerdistributions like IPFire or OpenWRT would be able to max out GBit Connections so I’m a little courious why it is not working on my setup, even with massive tuning.
Can anyone confirm that the APU is able to handle such connections with full speed? Is there anything I’ve missed? I’m wondering about the integrated CPU would not be able to handle 500MBit Network connections.
It generally can transfer a Gigabit in a second. But encapsulating the PPP packets takes some time and is entirely CPU-bound which is not a strong point of this hardware.
If you need something that is a lot faster than this hardware, it is very hard to find something in the same form factor.
DJ has already suggested something, but depending on where you are going to use it (at home?) that might be too large (form factor-wise) or just simply too expensive.
We believe (and have been fed this back by PC Engines) that IPFire performs best on this board - by far.
Also… tuning can help avoiding performance wasting, but sometimes all you have to do is increase displacement (sometimes there’s replacement for displacement, but not so much)
I don’t have any Firewall Roles, Proxy or QoS configured yet. I’m wondering why this CPU is not beeing able to handle only the half of the NIC-Speed.
Consumer ready routers like FritzBox and so on are much slower than the Jaguar chip. Would it be usefull to try a normal distribution like arch? I don’t need a webinterface.
Is there any significant improvement with a recent BIOS/Firmware? I don’t have checked the installed Version yet but I’ve in mind thate the version was lower than the version numbers I read in several forums.
Also I’ve read something about “coreboost”. Is that eventually not activated on my installation?
EDIT:
I’ve read about that the performance Problem could be PPPoE. This information is from pfSense which is BSD but could this also be the bottleneck in linux? What implementation is used in IPFire? The kernel ppp implementation or rp-pppoe? Could it be worth to test rp-pppoe? Nevertheless I will try it but if you say it will not have any impact I could save this time.
EDIT2:
I gave OpenWRT a chance and what should I say. It works out of the box. Same downlink speed as with notebook directly attached to the modem. 533.22MBit at first test. I wonder why IPFire is not bee able to reach this values. It’s also linux based.
No, the bottleneck simply is the hardware. This one is optimised to consume little power and there have to be compromises somewhere.
I am not aware of any performance improvements through BIOS updates. Minor ones potentially, but nothing major.
That would be enabled in the version that LWL is shipping. However, it is best that that is not activating itself, because IPFire balances the load across all CPU cores which are much faster combined than a single one.
It is hard to compare different distributions just like this.
IPFire filters packets by default. If OpenWRT does not do that your comparison is flawed. If you disable all features in IPFire, you might of course get more performance, but a network that is less secure and probably not as much fun to use.
But I never enabled any features in IPFire. Firewallrules where empty, no QoS, no VPN. All in all the same Configuration as in OpenWRT. With top I checked the load of the cpu. In IPFire one Core goes to 99% when the “maximum” speed is reached. OpenWRT 0%. I think it’s something NIC-Configuration related issue. But this is only a guess. The differences are so significant that I cannot believe in slow hardware only. I’m ok with OpenWRT but it is too bad for IPFire.
It might be true. I’ve run openWRT on a TP-Line MR-3020 - Atheros AR9331 CPU with 4 MB flash and 32 MB RAM. It runs an SPI firewall fairly responsively. The same software running on APU4c4 might well report < 1 % CPU utilisation.
But openWRT does not provide any of the additional firewalling capabilities of IPFire
But I had no firewall rules in IPFire at all. Firewall configuration was empty. Now I have some simple rules in OpenWRT for testing and have no performance impact. Even some port forwarding so NAT should be enabled.
What are the additional firewall capatibilities of IPFire I’ve missed?
IPFire performs many more things on a packet that it forwards than the standard ISP router. That is because we have so much more CPU power available (usually) and that allows us for deeper inspection of packets and many other things.
On PPPoE: The encapsulation of the packets will in both systems come from the Linux kernel. So there should be no difference in throughput unless you configured something differently.
is there any way to disable firewall in IPFire completely to do some tests? It would be interresting at all to look where the gap is. I think a 4C 1GHz System has to be able to handle a 500MBit Connection even with a better firewall. I only want to prove myself beeing wrong with this
OpenWRT is not a standard ISP-Router at all. As far as I know OpenWRT and IPFire both using netfilter and iptables. I still think it is NIC-Configuration related due to the significant performance improvement I’ve made with pinning the NIC-IRQs to specific CPU Cores (that should not be the solution at all).
Digging deeper on this could lead to a massive improvement of IPFire.
@superdachs, can you update about your APU4 adventures?
Speaking of OpenWRT and others (Mikrotik), I belive the main difference is the hardware acceleration. Those routers do not inspect every packet indeed. The typical cause of action: inspect the connection, once the connection is recognized as safe, the thing just dumps it into hardware and essentially forget about it.
At the same time, IRQ balancing inefficiency almost halves IPFire performance on low power CPUs. I’m not sure who to blame here. It looks more like a general Linux IRQ balancer daemon failure to properly allocate IRQ over cores. It loads a single core with all irqs instead of spreading it evenly.
IPFire also skip many things at processing of approved connections. (It’s a statefull firewall.)
Also irq and cpu-core pinning should done by IPFire since core160
But you can enable some features that need more processing power. (IPS, QoS …)
