WireGuard & OpenVPN throughput collapses only when the tunnel is terminated on IPFire — forwarding (Red↔Green) and a NAS-terminated tunnel behind the same box both run at full line speed

[jpanse]

Hi all,

I’m chasing a throughput problem that I’ve now narrowed down quite far, but I’m stuck on the root cause. I’d appreciate fresh eyes before I do anything drastic.

The core observation

• Plain routing Red ↔ Green runs at full line speed. The hardware, NICs and routing engine are clearly fine.

• The moment a VPN server terminated on the IPFire itself sits in the path, throughput collapses to ~10–12 Mbit/s.

• This happens with both WireGuard and OpenVPN (RoadWarrior), to the same degree.

• Crucially: the same WireGuard tunnel terminated on a NAS behind the IPFire (IPFire only forwards the UDP port) runs at ~260 Mbit/s over the identical WAN connection, path and latency.

So the only variable that matters is where the tunnel is terminated: terminate it on the NAS → 260 Mbit; terminate it on the IPFire → 12 Mbit. Same box, same line, same client, same path.

Setup

• IPFire Core Update 202 (so the Core 196 WireGuard improvements — per-device kernel thread, GSO backport — are already in place).

• CPU: Intel Xeon Silver 4309Y (8C/16T). /proc/cpuinfo flags include aes vaes pclmulqdq avx512 sha_ni — full crypto acceleration.

• WAN (red0) on fiber. Client is remote with ~210 ms RTT.

• Note on latency: the ~210 ms is identical for the fast NAS-terminated tunnel and the slow IPFire-terminated tunnel, so latency is not the differentiating factor (the NAS tunnel proves the path is fine at that latency).

Why hardware is ruled out

The same physical box forwards Red<->Green at line rate and the NAS tunnel passing through it hits 260 Mbit. If raw capacity (CPU, crypto, NIC, line) were the limit, those couldn’t run at full speed. During the slow VPN transfers the box shows essentially no CPU load (top ≈ 0.2 %, no openvpn/ksoftirqd process pegged). This looks like a software/termination-path limit, not a capacity limit.

What I have already tested and ruled out

• IPS (Suricata): stopped completely → no change.

• QoS / shaping: tc qdisc show shows only fq_codel/pfifo_fast, no htb/tbf, 0 drops. No traffic shaping configured.

• Firewall rate limits: iptables-save | grep -i limit → only LOG-rate limits, no rate limit on VPN traffic.

• Conntrack: conntrack -C = ~2985 of nf_conntrack_max 262144 (~1 %). Not full.

• MTU / fragmentation: ping -f -l 1400 8.8.8.8 through the tunnel → 0 % loss. wg0 MTU is 1420 on both the IPFire and the NAS; the WireGuard client config has no MTU override on either side.

• Kernel vs userspace WireGuard: lsmod | grep wireguard shows the kernel module loaded; no wireguard-go process. Native kernel WG, same as the NAS.

• Socket buffers: net.core.rmem_max / wmem_max = 4194304 (4 MB) → allows ~150 Mbit at 210 ms RTT, so buffers are not the cap.

• Parallel streams: iperf3 -c <ipfire-wg-ip> -P 8 -t 20 → [SUM] ~10.2 Mbit/s total, i.e. the same as a single stream — throughput does not scale with parallel streams.

• VPN port: changed the WireGuard listen/endpoint port (10443 → another port) → no change.

• Egress / routing: tracert -d 8.8.8.8 through the IPFire-terminated tunnel shows traffic leaving directly via the firewall’s own WAN IP (hop 2 = our public IP, redacted), straight to the ISP backbone — no second VPN hop, no detour. The decrypted traffic exits cleanly via Red0.

Why a reinstall is (probably) not the answer

A reinstall would be restored from backup, and the backup contains the current configuration — so if the cause is in the config, it would simply come back. More importantly, two independent VPN stacks (WireGuard and OpenVPN) are affected identically, which argues against a corrupted per-service config that a clean install would fix. That points at something systemic in the termination/egress path (a driver / NIC-offload behaviour on the decrypt-and-reinject path, or a kernel/sysctl/config aspect) — something a reinstall+restore would either preserve or reproduce.

My question

On capable hardware, what could cap locally terminated VPN throughput at ~10–12 Mbit/s while:

• plain Red<->Green forwarding runs at full line speed,

• a tunnel terminated on a NAS behind the same firewall runs at ~260 Mbit/s over the identical path/latency,

• both WireGuard and OpenVPN are hit equally,

• and the CPU stays near-idle?

Has anyone seen VPN-terminated throughput pinned like this independent of protocol, port, IPS, shaping and buffers? Where in the IPFire VPN-termination path would you look next — NIC offload on the decrypt path, IRQ/queue handling for tunnel traffic, or something in the server-side config?

Happy to provide any specific command output. Thanks!

[bassie (Bas Heijermans)]

Try these parameters in /etc/sysctl.conf :

# Increase kernel backlog queue:
net.core.netdev_max_backlog=4000
# Packets queued per CPU between NIC and socket layer

# Increase connection tracking:
net.netfilter.nf_conntrack_max = 1048576

I know, contrack doesn’t show many connections.

But the backlog is the NIC RX/TX ringbuffer and if it’s overloaded it will drop packets that will never make it to your daemons.

Typical ringbuffers are set for home-use and not heavy traffic.

Give it a shot. It helped me, but with high-traffic NTP-server.

[jpanse]

Thanks for the suggestion. Tested it since we had a similar symptom: single TCP stream over WireGuard, throughput starting low and climbing slowly.

Baseline (before your settings, single iperf3 -R stream, 60s):
Average: 21.0 Mbit/s
End of test: 45.1 Mbit/s

After setting net.core.netdev_max_backlog=4000 and net.netfilter.nf_conntrack_max=1048576, same test:
Average: 14.3 Mbit/s
End of test: 28.1 Mbit/s

No improvement, slightly worse actually, but within normal run to run variance. conntrack usage was never close to the old limit (2985/262144, about 1%), so raising nf_conntrack_max couldn’t have mattered here. RX ring wasn’t saturated either (200/2047 buffers used), so netdev_max_backlog wasn’t our bottleneck.

What actually fixed our real problem was two separate changes. First, switching TCP congestion control from bbr to cubic on the firewall. BBR was building a huge congestion window and overrunning a small buffer in the send path, causing massive packet loss. That alone took a fully collapsed tunnel (about 10-12 Mbit/s) up to around 230 Mbit/s. Second, adding MSS clamping on the WireGuard forward path (iptables TCPMSS clamp mss to pmtu), which fixed a separate issue where full size 1500 byte packets from the LAN didn’t fit into the 1420 MTU tunnel and got blackholed.

Once both were in place the total stalls disappeared. What’s left now is normal single stream TCP ramp up behavior over a high RTT path (about 213ms, Thailand to Switzerland), not a drop or backlog issue.

Appreciate the pointer though. Makes sense it would help on a high packet rate UDP daemon like NTP where the RX ring is the constraint. Just wasn’t our bottleneck here.

[janb (Jan Behrens)]

Hi,

maybe try to read this thread:

openVPN Roadwarriors slow throughput

Hi, since core 202 I am having a problem with openvpn. The ipfire I’m connecting to has 1Gig/1Gig Internet connection. And it worked as expected until last update. I made an iperf3 request through openvpn tunnel and one without. This one is through openvpn: [image] I already changed from tcp to udp, but there its even worse and unstable. I changed the MTU from 1400 to 1420. Enabled/disabled DCO. Rebooted the ipfire. Changed the cryptgraphy to something less secure. Maybe someone can…

I had a problem after upgrading to Core 202. Only thing which solved it for me was to replace the openvpn binaries with older ones.

Kind Regards