openVPN Roadwarriors slow throughput

Hi,

since core 202 I am having a problem with openvpn.
The ipfire I’m connecting to has 1Gig/1Gig Internet connection.
And it worked as expected until last update.

I made an iperf3 request through openvpn tunnel and one without.

This one is through openvpn:

I already changed from tcp to udp, but there its even worse and unstable.
I changed the MTU from 1400 to 1420.
Enabled/disabled DCO.
Rebooted the ipfire.
Changed the cryptgraphy to something less secure.

Maybe someone can help me with this strange problem please.

Kind Regards

Jan B

This one is without ovpn:

Would help if you detail the hardware on both ends: IPFire and the RW client.

This is inline with what I get on my boxes (ranging from PcEngines APU2D4 to dedicated architecture for firewall with N100 CPU & DDR5 memory)

But my “test” clients are always phones (medium range ones - nothing fancy) using GSM/mobile data and therefore I suspect that phone capability plays a role in that (GSM line usually delivers 150 Mb/s because I keep the phones on PowerSave ON that disables 5G) [Late edit: I disabled Power Save on one phone, Mobile data switched to 4G+ (?) and VPN speed increased from 30 Mb/s to 48-50 Mb/s. On another phone with a premiun range CPU (7 years ago was on top CPU) and 5G connection I got 100 Mb/s over VPN, but different GSM operator - better 5G coverage in the area I am now. The router was always the same - the N100 CPU box with DDR5 memory, Ipfire CU200]

To your question: one element that in the past helped me was LZO compression - nowadays is off by default, and I do not see it in UI anymore.
You can turn it on by editing settings file and reload the service

grep -i lzo /var/ipfire/ovpn/settings
COMPLZO=off
DCOMPLZO=off

Back in 2019 LZO was on UI, now I do not see it anymore OpenVPN: Is LZO-Compression now effectively disabled? - VPNs / OpenVPN - IPFire Community

Note that LZO compression is considered a security risk.

Security Advisory: The VORACLE attack vulnerability | OpenVPN

OK, the hardware running ipfire is an IPU445 with Intel Core i7-4500U and 8GB DDR3L-RAM, internet connection is 1Gig/1Gig.
The connecting client is Windows 11 with AMD Ryzen7 9700X and 64GB DDR5 RAM, internet connection hybrid bonding 5G with this throughput: Speedtest by Ookla - The Global Broadband Speed Test

I added
COMPLZO=off
DCOMPLZO=off
to /var/ipfire/ovpn/settings and made /etc/rc.d/init.d/openvpn-rw restart
same result :frowning:
Also added
COMPLZO=on
DCOMPLZO=on
and restarted. Still the same.

I get about 2-7 MB/s which was 45 MB/s before.

Thanks

Which OpenVPN client (Community or “official”) and which version?

community version and the most recent one: 2.7.4, I updated the version yesterday in hope to solve the problem. I don’t remember which version it was before, but for sure community version.

Hallo @janb

Welcome to the IPFire community.

Generally UDP should work fine with OpenVPN. If it is not and is unstable that would suggest to me that you are losing packets somewhere, which TCP is able to compensate most of the time but of course it then adds overhead to the traffic being sent.

If you also get a very unstable situation with UDP when running the iperf3 test from your red interface to a client on your green network then I would look at the drop stats for your nics to see if there is a problem with the hardware.

ip -s link show green0

replace green0 with red0 or blue0 or orange0 for any of the other interfaces.

Your drop rate should be 0 or very low in comparison to the transmitted or received packets.

ip -s link show red0
6: red0@eth1: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx3 brd ff:ff:ff:ff:ff:ff
    RX:  bytes packets errors dropped  missed   mcast           
    9541589163 5875288      0       0       0       0 
    TX:  bytes packets errors dropped carrier collsns           
    2389314977 5990443      0       0       0       0 

ip -s link show green0
2: green0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
    RX:  bytes packets errors dropped  missed   mcast           
     561099649 3512108      0      47       0   10729 
    TX:  bytes packets errors dropped carrier collsns           
    4974827767 6042566      0       0       0       0 

If the dropped stats are 0 or very low on your nics then it would tend to eliminate a nic hardware problem.

It was removed from the interface with the update to OpenVPN-2.6 in CU197.

See the compression section in the CU197 release announcement.

https://www.ipfire.org/blog/ipfire-2-29-core-update-197-released#openvpn-26

Since OpenVPN-2.4 the comp-lzo option has been ignored anyway by OpenVPN and if you look in your openvpn logs there would have been a warning about its use.
The same is true for the compress option but since OpenVPN-2.5

[root@ipfire~]# ip -s link show red0
3: red0: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
    RX:   bytes  packets errors dropped  missed   mcast
     5864483854  6529111      0       0       0     380
    TX:   bytes  packets errors dropped carrier collsns
    11964089588 10170872      0     844       0       0

[root@ipfire~]# ip -s link show green0
8: green0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
    RX:   bytes packets errors dropped  missed   mcast
    11133223459 4045622      0   32365       0  181296
    TX:   bytes packets errors dropped carrier collsns
     4751237241 5900897      0       2       0       0

green0 is a bridge interface to integrate wifi clients from specific ssid.

Now I have called my ISP for resetting gateway and port.
Both didn’t resolve the problem. Still very low rate.
Generally I would like to stick with TCP since it was okay (45 MB/s) until core 202.

for further investigation: There was an opnsense firewall installed which had the exact same problem - then i replaced the opnsense with this ipfire. The problem was gone and I had nice 45MB/s. Now yesterday I tested again and I get only 2-7 MB/s (like with the opnsense)

ISP says everything is fine on the line.
And i get the 1000 MBit with other applications like speedtest etc.
Only openvpn makes problems.

Try Wireguard to see if you get better results

It’s getting even worse with wireguard. I get 355 KB/s

A quick note.

obraz

In the test above, we see data transfer from the client to the server.

Adding the -R switch shows data transfer from the server to the client.
e.q. iperf3 -c 192.168.1.119 -R

I think it’s worth taking both tests.

over openvpn:

i think the slow startrate is caused by hybrid bonding.
But if I had 123 MBit after some seconds it would be OK. But it is not getting higher than 7 MB/s from server machine to my ovpn client in copy data.

WireGuard uses UDP so that looks like there are lots of lost packets that UDP cannot recover and something that is common for both WireGuard and OpenVPN which would either be the hardware you are running IPFire on or the connection from the ISP.

That seems like a lot of dropped packets, especially when TCP is being used.

If you run

netstat -st

it will tell you info for all the tcp related ports.

Tcp:
    27193 active connection openings
    12036 passive connection openings
    7860 failed connection attempts
    595 connection resets received
    40 connections established
    3898754 segments received
    3198686 segments sent out
    20406 segments retransmitted
    0 bad segments received
    4882 resets sent

What number of segments retransmitted do you have and do you have any bad segments received?

Just opened the speed test of the client - it tops at 40+ Mb/s upload.

And, as IPTom already spotted, the iperf is showing the upload direction only - both are inline.

Same happens with my asymmetrical mobile data links: the download for mobile is usually 3 fold faster than uplink - so I get max 50 Mb/s uplink directly and 10 Mb/s over VPN.

As IPTom suggested, would be good to focus on the direction that has big throughputs.

Also, please check QOS and run init.d/qos stop and then try again.

Some versions ago QoS was present although UI show never setup/not started.

Just to be sure that is not back

Tcp:
    545 active connection openings
    178 passive connection openings
    0 failed connection attempts
    6 connection resets received
    8 connections established
    812457 segments received
    1712810 segments sent out
    471 segments retransmitted
    0 bad segments received
    422 resets sent

No QoS active and also no qos binary in /etc/init.d

I think the problem lies elsewhere.
I just ran a test with my Android phone on 5G using IPFire CU202.

I get the following with WireGuard:

[root@ipfire ~]# iperf3 - s
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
ccepted conection from 10.0.20.1, port 43562
[  5] local 192.168.20.1 port 5201 connected to 10.0.20.1 port 43574
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  5.62 MBytes  47.2 Mbits/sec
[  5]   1.00-2.00   sec  6.88 MBytes  57.7 Mbits/sec
[  5]   2.00-3.00   sec  7.12 MBytes  59.8 Mbits/sec
[  5]   3.00-4.00   sec  6.75 MBytes  56.6 Mbits/sec
[  5]   4.00-5.00   sec  7.25 MBytes  60.8 Mbits/sec
[  5]   5.00-5.38   sec  3.25 MBytes  71.4 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-5.38   sec  36.9 MBytes  57.5 Mbits/sec                  receiver
---------------------------------------------------------
Server listening on 5201 (test #2)
-----------------------------------------------------------
Accepted connection from 10.0.20.1, port 57108
[  5] local 192.168.20.1 port 5201 connected to 10.0.20.1 port 57114
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  21.5 MBytes   180 Mbits/sec    0   1.07 MBytes
[  5]   1.00-2.00   sec  24.5 MBytes   206 Mbits/sec    0   1.21 MBytes
[  5]   2.00-3.00   sec  23.5 MBytes   197 Mbits/sec    0   1.16 MBytes
[  5]   3.00-4.00   sec  26.1 MBytes   219 Mbits/sec    0   2.07 MBytes
[  5]   4.00-5.00   sec  24.2 MBytes   203 Mbits/sec    0   2.07 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-5.01   sec   120 MBytes   201 Mbits/sec    0            sender

And with OpenVPN

-----------------------------------------------------------
Server listening on 5201 (test #3)
-----------------------------------------------------------
Accepted connection from 10.235.155.3, port 37610
[  5] local 192.168.20.1 port 5201 connected to 10.235.155.3 port 37624
[ ID] Intevl       Tase     Bitrate
[  5]   0.00-1.00   sec  6.50 MBytes  54.5 Mbits/sec
[  5]   1.00-2.00   sec  7.75 MBytes  65.0 Mbits/sec
[  5]   2.00-3.00   sec  7.38 MBytes  61.9 Mbits/sec
[  5]   3.00-4.00   sec  7.12 MBytes  59.8 Mbits/sec
[  5]   4.050   sec  7.50 MBytes  62.9 Mbits/sec
[  5]   5.00-5.22   sec  1.88 MBytes  70.3 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-5.22   sec  38.1 MBytes  61.2 Mbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201 (test #4)
-----------------------------------------------------------
Accepted connection from 10.235.155.3, port 38544
[  5] local 192.168.20.1 port 5201 connected to 10.235.155.3 port 45898
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  14.5 MBytes   122 Mbits/sec  725    202 KBytes
[  5]   1.00-2.00   sec  12.9 MBytes   108 Mbits/sec  764    823 KBytes
[  5]   2.00-3.00   sec  14.1 MBytes   118 Mbits/sec  1902    903 KBytes
[  5]   3.00-4.00   se 1.4 MBytes   137 Mbits/sec  2499    636 KBytes
[  5]   4.00-.0 sec  .2Myes  80.7 Mbits/sec  1094    743 KBytes
[  5]   5.00-5.15  sec  .0 Bytes  0.00 bits/sec   95    480 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-5.15   sec  67.5 MBytes   110 Mbits/sec  7079            sender
----------------------------------------------------------
Server listening on 5201 (test #5)
-----------------------------------------------------------

The same download speed and double the upload speed with Wireguard