We have a very special interest setup and sometimes achieve strange results. I would like to ask the community for possible explanations or hints for improvements. Maybe the discussion will also help others with similar setups.
We have 2 locations connected via 3 OpenVPN net-2-net tunnels using 2 different WAN connections (1 x cable, 1 x DSL) on one side. This is working fine so far for long time. We actually try to increase data throughput for nightly backups ~60GB/night from location A to location B. In order to use multiple connections simultaneously (load balanced), we use a multipath ISCSI connection as described below.
Unfortunately the overall data throughput seems to be unstable and I do not clearly understand the circumstances.
I made tests with several different setups:
Location A (data centre):
- one powerful IPFire with red, green (192.168.16.0/24), blue (192.168.17.0/24), orange (192.168.18.0/24)
- this machine has direct internet access with 1 GBit/s using a public IPv4 address
- this machine acts as OpenVPN net-to-net client
Location B (home office)
- one medium/small sized IPFire Duo Box with red and green (192.168.0.0/24) network
- this machine is connected to the internet via an intermediate Zyxel dual WAN router which is connected to the internet “WAN1” via cable/fritzbox at 300/45 MBit/s down/upstream and “WAN2” via DSL/fritzbox at 95/35 MBit/s
- public IPs reacheable via dyndns
- this machine acts as OpenVPN net-to-net server
There are 3 OpenVPN net-to-net connections, called LINE G (green), B (blue) and O (orange) configured on both sides:
LINE G: loc. A (green) <=> loc. B (green), OpenVPN subnet 10.0.51.0/24 LINE B: loc. A (blue) <=> loc. B (green), OpenVPN subnet 10.0.52.0/24 LINE O: loc. A (orange) <=> loc. B (green), OpenVPN subnet 10.0.40.0/24
Firewall rules allow the green & blue network (loc. A) to access the green network (loc. B) and vice versa. Connections to orange (from loc. B to A) are allowed in this direction (not reverse). - This works fine so far.
A client on loc. A (connected to green & blue network) mounts an ISCSI target on a server in green (loc. B) via multipath (MPIO) connection:
- 4 paths from diff. green IPs (loc. A), supposed to use LINE G, connecting to one single green IP (loc. B)
- 4 paths from diff. blue IPs (loc. A), supposed to use LINE B, connecting to the same single green IP (loc. B)
These 8 parallel connections are each working. We made the overall experience, that increasing the number of connections (max. 32) helps to increase throughput.
We tested different combinations
LINEs G/B/O using .... between loc. A & B throughput MByte/s G B O loc. B->A (up) /A->B (down) cable cable cable 5,9/ 7,3 cable cable DSL 5,8/ 3,9 cable DSL DSL 10,2/ 4,5 upstream load balanced DSL DSL DSL 4,3/10,0 cable DSL cable 10,3/13,5 upstream load balanced DSL cable cable 10,3/14,8 upstream load balanced DSL cable DSL 10,2/ 9,4 upstream load balanced DSL DSL cable 4,3/20,0 => highest downstream rate DSL DSL -/- 4,3/10,1 DSL cable -/- 10,3/12,7 upstream load balanced cable DSL -/- 10,1/ 9,1 upstream load balanced
It seems amazing, that highest throughput rates downstream were achieved, when LINE O [loc. A (orange) to loc. B (green)] was established via cable connection, because there should be no data transferred from orange network AND firewall rules on loc. B deny incoming connections from loc. A (orange).
It seems to me that at least one of the two IPFire appliances is internally a little bit ‘confused’ and mixes up traffic between the 3 OpenVPN net-to-net tunnels (…)?
The OpenVPN net-to-net traffic charts show that in this case the whole downstream traffic uses LINE O and only the upstream traffic is equally distributed between LINE G and B.
- both IPFire appliances are far from reaching their CPU load limits (loc. A ~20%, loc. B ~60%)
- the pure bandwidth of the cable/DSL connections was successfully evaluated with IPERF
- the intermediate Zyxel dual WAN router is configured to respond to incoming OpenVPN traffic on the gateway that received the traffic, it is able to route traffic up to ~600 MBit/s
- I also tested throughput using a 4th OpenVPN net-to-net connection (fantasy network 192.168.19.0/24 on loc. A to green on loc. B), this led to decreased overall throughput although the connection was effectively not used
- I used a 2nd IPFire (a very small machine, APU Board) on loc. A, connected to the green & blue network with a different public IP-address. The OpenVPN net-to-net tunnel from blue (loc. A) to green (loc. B) was established using this machine => no substantial influence on overall throughput. This might indicate that the problem is located on IPFire on loc. B; unfortunately I cannot test a 2nd IPFire appliance in loc. B.
- I set up a second ALIAS-IP on green (loc. B) and routed all traffic from blue (loc. A) to green (loc. B) over this IP (routes on the ISCSI target server adjusted accordingly) => no substantial influence on overall throughput
My questions are:
- Why are the LINEs G & B only occipied with upstream traffic (loc. B → A) and why is LINE O only occupied with downstream traffic although there isn’t any data transfer between loc. A (orange) and loc. B (green)?
- Why seems upstream traffic to be distributed on 2 lines while downstream traffic only uses one single line?
- The above given test results tend to vary when both IPFires are rebooted; it seems that switching all OpenVPN tunnels on/off helps to reach the state again
- Does anyone have another idea how to cumulate the bandwidth of two internet connections with onboard resources using IPFire? I don’t intend to use additional commercial solutions.
I know that IPFire itself is neither able to use multi WAN, nor is it able to distribute traffic over multiple channels simultaneously. But I think it is (should be) able to transfer traffic between different IPs over different OpenVPN tunnels simultaneously. So if we use an ISCSI client as “load balancer” with a target that is able to manage multiple connections and an intermediate dual wan router, the setup described above could work - or doesn’t it?
Any help is welcome!