Network Performance Issues on APU4c4

Hello,

I got a brand new APU4C4 Board for use as Router on my 500/200MBit FTTH. Due to performance issues with (OPN|PF)Sense and beeing more familiar with Linux I decided tu use IPFire on it.
Installation and configuration went fine. I’m missing some features but this is configurable via CLI for me and I’m fine with that.

Unfortunately I’ve some massive performance problems. With fresh installed IPFire and only a PPPoE Connection I got only 330-350MBit Downstream on multi connection. If I put a Linux Notebook directly to the Fiber-Modem and configure the PPPoE I got arround 510-530MBit.

I’ve read tons of documentations to improve the performance. Classical intel NIC tunig as described in many howtos won’t bring a significant improvement. Only pinning the IRQs of the NICS to a dedicated CPU-Core per NIC improves speed massively to 450-470MBit. But there are nevertheless 60-80MBit left. I’ve read that linux based Routerdistributions like IPFire or OpenWRT would be able to max out GBit Connections so I’m a little courious why it is not working on my setup, even with massive tuning.

Can anyone confirm that the APU is able to handle such connections with full speed? Is there anything I’ve missed? I’m wondering about the integrated CPU would not be able to handle 500MBit Network connections.

Welcome,

do you use ips? maybe some rules are need to much cpu power?

https://wiki.ipfire.org/hardware/lightningwirelabs/mini

maybe this would be a better choice https://wiki.ipfire.org/hardware/lightningwirelabs/business

This is quite tight for this hardware.

It generally can transfer a Gigabit in a second. But encapsulating the PPP packets takes some time and is entirely CPU-bound which is not a strong point of this hardware.

If you add more things on top (IPS, QoS, proxy, etc…) this will only slow it down. The IPS is the biggest offender here and you won’t be able to get anywhere near 500 MBit/s. Expect maybe around 130 MBit/s (https://wiki.ipfire.org/configuration/firewall/ips/performance-considerations).

If you need something that is a lot faster than this hardware, it is very hard to find something in the same form factor.

DJ has already suggested something, but depending on where you are going to use it (at home?) that might be too large (form factor-wise) or just simply too expensive.

We believe (and have been fed this back by PC Engines) that IPFire performs best on this board - by far.

Under the hood there’s a 4 core Geode Jaguar Chip, with a remarkably low consumption (6W) and about 1500 score to CPUBenchmark.
https://www.cpubenchmark.net/cpu.php?cpu=AMD+GX-412HC&id=2473
(benchmarked version HC has a graphic core installed, with a total consumption of 7W)
https://www.amd.com/en/system/files?file=2017-06/g-series-soc-product-brief.pdf
But sooner or later 10GBe and 2.5GBe will be accessible. So question is: how much the TDP will be raised to follow up this performance need?

Also… tuning can help avoiding performance wasting, but sometimes all you have to do is increase displacement (sometimes there’s replacement for displacement, but not so much)

Thanks,

I don’t have any Firewall Roles, Proxy or QoS configured yet. I’m wondering why this CPU is not beeing able to handle only the half of the NIC-Speed.
Consumer ready routers like FritzBox and so on are much slower than the Jaguar chip. Would it be usefull to try a normal distribution like arch? I don’t need a webinterface.

Is there any significant improvement with a recent BIOS/Firmware? I don’t have checked the installed Version yet but I’ve in mind thate the version was lower than the version numbers I read in several forums.
Also I’ve read something about “coreboost”. Is that eventually not activated on my installation?

EDIT:
I’ve read about that the performance Problem could be PPPoE. This information is from pfSense which is BSD but could this also be the bottleneck in linux? What implementation is used in IPFire? The kernel ppp implementation or rp-pppoe? Could it be worth to test rp-pppoe? Nevertheless I will try it but if you say it will not have any impact I could save this time. :slight_smile:

EDIT2:
I gave OpenWRT a chance and what should I say. It works out of the box. Same downlink speed as with notebook directly attached to the modem. 533.22MBit at first test. I wonder why IPFire is not bee able to reach this values. It’s also linux based.

No, the bottleneck simply is the hardware. This one is optimised to consume little power and there have to be compromises somewhere.

I am not aware of any performance improvements through BIOS updates. Minor ones potentially, but nothing major.

That would be enabled in the version that LWL is shipping. However, it is best that that is not activating itself, because IPFire balances the load across all CPU cores which are much faster combined than a single one.

https://wiki.ipfire.org/hardware/mythbusters/single-core-performance

IPFire is using the kernel implementation which I consider the faster one.

But I am happy to be proven wrong.

It is hard to compare different distributions just like this.

IPFire filters packets by default. If OpenWRT does not do that your comparison is flawed. If you disable all features in IPFire, you might of course get more performance, but a network that is less secure and probably not as much fun to use.

But I never enabled any features in IPFire. Firewallrules where empty, no QoS, no VPN. All in all the same Configuration as in OpenWRT. With top I checked the load of the cpu. In IPFire one Core goes to 99% when the “maximum” speed is reached. OpenWRT 0%. I think it’s something NIC-Configuration related issue. But this is only a guess. The differences are so significant that I cannot believe in slow hardware only. I’m ok with OpenWRT but it is too bad for IPFire.

OpenWRT does not use any CPU? That cannot be true.

It might be true. I’ve run openWRT on a TP-Line MR-3020 - Atheros AR9331 CPU with 4 MB flash and 32 MB RAM. It runs an SPI firewall fairly responsively. The same software running on APU4c4 might well report < 1 % CPU utilisation.

But openWRT does not provide any of the additional firewalling capabilities of IPFire

But I had no firewall rules in IPFire at all. Firewall configuration was empty. Now I have some simple rules in OpenWRT for testing and have no performance impact. Even some port forwarding so NAT should be enabled.
What are the additional firewall capatibilities of IPFire I’ve missed?

IPFire performs many more things on a packet that it forwards than the standard ISP router. That is because we have so much more CPU power available (usually) and that allows us for deeper inspection of packets and many other things.

On PPPoE: The encapsulation of the packets will in both systems come from the Linux kernel. So there should be no difference in throughput unless you configured something differently.

is there any way to disable firewall in IPFire completely to do some tests? It would be interresting at all to look where the gap is. I think a 4C 1GHz System has to be able to handle a 500MBit Connection even with a better firewall. I only want to prove myself beeing wrong with this :smiley:
OpenWRT is not a standard ISP-Router at all. As far as I know OpenWRT and IPFire both using netfilter and iptables. I still think it is NIC-Configuration related due to the significant performance improvement I’ve made with pinning the NIC-IRQs to specific CPU Cores (that should not be the solution at all).
Digging deeper on this could lead to a massive improvement of IPFire.

@superdachs: maybe my problem is related to yours or vice versa

BTW: with running OpenVPN and active Firewall still over 500MBit in OpenWRT.

Have you tested basic nic speed with iperf2 and compared with the Wiki?
https://wiki.ipfire.org/hardware/pcengines/apu2b4
APU2 and 4 should have the same performance.

I can repeat the tests at next weekend if needed.

Not yet but I will next weekend.

I have repeatet the tests:
unidirektional 920Mbit from a client in green to red or vice versa and 730mBit bidirectional.

arne@thinkpad ~ $ iperf -c 192.168.200.80 -d
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.200.80, TCP port 5001
TCP window size:  298 KByte (default)
------------------------------------------------------------
[  5] local 192.168.20.18 port 56386 connected with 192.168.200.80 port 5001
[  4] local 192.168.20.18 port 5001 connected with 192.168.200.80 port 59356
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0-10.0 sec   872 MBytes   731 Mbits/sec
[  4]  0.0-10.0 sec   919 MBytes   770 Mbits/sec

Tested with an APU2B4 on core139 with a DNAT port5001 rule to the client ip of the client in green.