I have a need for routing of certain IP-destination from my GREEN-network through a dedicated interface (in my case a TUN device). That’s why I’m placing it under network, although it has got firewall components.
Until core update 165 this advanced routing use case was perfectly running on my IPFIRE instance. Since the migration to IPSET (for managing the relevant IPs) I wasn`t able to get it working again.
The following rule chain manages the marking and the connection tracking of the, via ta TUN device, routed packages:
iptables -t mangle -I PREROUTING -i green0 -m set --match-set IPSET src -j TOS --set-tos 0x10
iptables -t mangle -I PREROUTING -i green0 -m set --match-set IPSET dst --set-tos 0x10
iptables -t nat -A CUSTOMPOSTROUTING -s green0 -o tunX -j MASQUERADE
iptables -A CUSTOMFORWARD -s green0 -o tunX -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -A CUSTOMFORWARD -i tun2 -d blue0 -j ACCEPT
The routing itself is steered by the following rules:
0: from all lookup local
32763: from 172.24.222.0/24 tos 0x10 lookup XYZ
32764: from 172.24.222.0/24 fwmark 0xf lookup XYZ
32765: from all lookup static
32766: from all lookup main
32767: from all lookup default
And the custom routing table “XYZ” very standard, just for the TUN device on duty for this use case:
default via 10.36.0.1 dev tunX
10.36.0.0/16 dev tunX proto kernel scope link src 10.36.0.5
172.24.222.0/24 dev green0 proto kernel scope link src 172.24.222.1
192.168.87.0/24 dev red0 proto dhcp scope link src 192.168.87.64 metric 1002
in the course of trying to get it working again I figured out that the custom routing table “XYZ” isn’t present in /etc/iproute2/rt_tables, which I expected should have been the case. But I didn’t setup a 165er instance again to prove that it have been the case back then.
So I ran a test with an IP address from my IPSET by:
- ipset test IPSET 193.99.145.80 >> Warning: 193.99.145.80 is in set IPSET
- ip route show to match 193.99.145.80 >> default via 192.168.87.23 dev red0 proto dhcp src 192.168.87.64 metric 1002
- ip route get 193.99.145.80 >> 193.99.145.80 via 192.168.87.23 dev red0 src 192.168.87.64 uid 0 cache
Well, the traffic is still going via the standard red0 device. Now i Assumed the error should be in the firewall and the packages aren’t marked correctly.
But when I’m looking into the standard firewall log with my logging rules (see below) I can’t see any packages for requested IP. Which, to me, would mean, that the packages aren’t handled as intended by the rule chain above.
Logging rules:
iptables -t mangle -A PREROUTING -i green0 -m set --match-set IPSET DST -j LOG --log-prefix ’ VBE-PRR ’
iptables -A CUSTOMFORWARD -s 172.24.222.0/24 -o tunX -m conntrack --ctstate RELATED,ESTABLISHED -j LOG --log-prefix ’ VBE-CFo ’
iptables -t nat -A CUSTOMPOSTROUTING -s 172.24.222.0/24 -o tunX -j LOG --log-prefix ’ VBE-CPR ’
iptables -A CUSTOMFORWARD -i tunX -d 172.24.22.0/24 -j LOG --log-prefix ’ VBE-CFi ’
As I read about rp_filtering for spoofing protection (Advanced routing with firewall marks and `rp_filter` – Server Fault – RotaDEV.com) in the context of advanced routing and I decided to go for tos-marking as a possible workaround and so I switched on the logging for this to see what I can get from there.
sysctl -w net.ipv4.conf.all.log_martians=1
echo 1 >/proc/sys/net/ipv4/conf/all/log_martians
And now I see the following logging for my requested test IP address:
- kernel: IPv4: martian source 172.24.222.72 from 193.99.144.80, on dev tun4
*- kernel: ** VBE-PRR *IN=green0 OUT= MAC=00:0d:b9:4b:da:4d:d8:ec:5e:74:e1:71:08:00 SRC=172.24.222.72 DST=193.99.144.80 LEN=64 TOS=0x10 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=52121 DPT=443 WINDOW=65535 RES=0x00 SYN URGP=0
*- kernel: ** VBE-CPR *IN=green0 OUT=tun4 MAC= SRC=172.24.222.72 DST=193.99.144.80 LEN=64 TOS=0x10 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=52121 DPT=443 WINDOW=65535 RES=0x00 SYN URGP=0
- kernel: IPv4: martian source 172.24.222.72 from 193.99.144.80, on dev tun4
*- kernel: ** VBE-PRR *IN=green0 OUT= MAC=00:0d:b9:4b:da:4d:d8:ec:5e:74:e1:71:08:00 SRC=172.24.222.72 DST=193.99.144.80 LEN=64 TOS=0x10 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=52122 DPT=443 WINDOW=65535 RES=0x00 SYN URGP=0
*- kernel: ** VBE-CPR *IN=green0 OUT=tun4 MAC= SRC=172.24.222.72 DST=193.99.144.80 LEN=64 TOS=0x10 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=52122 DPT=443 WINDOW=65535 RES=0x00 SYN URGP=0
- kernel: IPv4: martian source 172.24.222.72 from 193.99.144.80, on dev tun4
*- kernel: ** VBE-PRR *IN=green0 OUT= MAC=00:0d:b9:4b:da:4d:d8:ec:5e:74:e1:71:08:00 SRC=172.24.222.72 DST=193.99.144.80 LEN=64 TOS=0x10 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=52121 DPT=443 WINDOW=65535 RES=0x00 SYN URGP=0
*- kernel: ** VBE-CFo *IN=green0 OUT=tun4 MAC=00:0d:b9:4b:da:4d:d8:ec:5e:74:e1:71:08:00 SRC=172.24.222.72 DST=193.99.144.80 LEN=64 TOS=0x10 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=52121 DPT=443 WINDOW=65535 RES=0x00 SYN URGP=0
- kernel: IPv4: martian source 172.24.222.72 from 193.99.144.80, on dev tun4
*- kernel: ** VBE-PRR *IN=green0 OUT= MAC=00:0d:b9:4b:da:4d:d8:ec:5e:74:e1:71:08:00 SRC=172.24.222.72 DST=193.99.144.80 LEN=64 TOS=0x10 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=52122 DPT=443 WINDOW=65535 RES=0x00 SYN URGP=0
*- kernel: ** VBE-CFo *IN=green0 OUT=tun4 MAC=00:0d:b9:4b:da:4d:d8:ec:5e:74:e1:71:08:00 SRC=172.24.222.72 DST=193.99.144.80 LEN=64 TOS=0x10 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=52122 DPT=443 WINDOW=65535 RES=0x00 SYN URGP=0
- kernel: IPv4: martian source 172.24.222.72 from 193.99.144.80, on dev tun4
*- kernel: ** VBE-PRR *IN=green0 OUT= MAC=00:0d:b9:4b:da:4d:d8:ec:5e:74:e1:71:08:00 SRC=172.24.222.72 DST=193.99.144.80 LEN=64 TOS=0x10 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=52121 DPT=443 WINDOW=65535 RES=0x00 SYN URGP=0
*- kernel: ** VBE-CFo *IN=green0 OUT=tun4 MAC=00:0d:b9:4b:da:4d:d8:ec:5e:74:e1:71:08:00 SRC=172.24.222.72 DST=193.99.144.80 LEN=64 TOS=0x10 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=52121 DPT=443 WINDOW=65535 RES=0x00 SYN URGP=0
- kernel: IPv4: martian source 172.24.222.72 from 193.99.144.80, on dev tun4
*- kernel: ** VBE-PRR *IN=green0 OUT= MAC=00:0d:b9:4b:da:4d:d8:ec:5e:74:e1:71:08:00 SRC=172.24.222.72 DST=193.99.144.80 LEN=64 TOS=0x10 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=52122 DPT=443 WINDOW=65535 RES=0x00 SYN URGP=0
*- kernel: ** VBE-CFo *IN=green0 OUT=tun4 MAC=00:0d:b9:4b:da:4d:d8:ec:5e:74:e1:71:08:00 SRC=172.24.222.72 DST=193.99.144.80 LEN=64 TOS=0x10 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=52122 DPT=443 WINDOW=65535 RES=0x00 SYN URGP=0
- kernel: IPv4: martian source 172.24.222.72 from 193.99.144.80, on dev tun4
*- kernel: ** VBE-PRR *IN=green0 OUT= MAC=00:0d:b9:4b:da:4d:d8:ec:5e:74:e1:71:08:00 SRC=172.24.222.72 DST=193.99.144.80 LEN=64 TOS=0x10 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=52121 DPT=443 WINDOW=65535 RES=0x00 SYN URGP=0
*- kernel: ** VBE-CFo *IN=green0 OUT=tun4 MAC=00:0d:b9:4b:da:4d:d8:ec:5e:74:e1:71:08:00 SRC=172.24.222.72 DST=193.99.144.80 LEN=64 TOS=0x10 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=52121 DPT=443 WINDOW=65535 RES=0x00 SYN URGP=0
*- kernel: ** VBE-PRR *IN=green0 OUT= MAC=00:0d:b9:4b:da:4d:d8:ec:5e:74:e1:71:08:00 SRC=172.24.222.72 DST=193.99.144.80 LEN=64 TOS=0x10 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=52126 DPT=80 WINDOW=65535 RES=0x00 SYN URGP=0
*- kernel: ** VBE-CPR *IN=green0 OUT=tun4 MAC= SRC=172.24.222.72 DST=193.99.144.80 LEN=64 TOS=0x10 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=52126 DPT=80 WINDOW=65535 RES=0x00 SYN URGP=0
- kernel: IPv4: martian source 172.24.222.72 from 193.99.144.80, on dev tun4
*- kernel: ** VBE-PRR *IN=green0 OUT= MAC=00:0d:b9:4b:da:4d:d8:ec:5e:74:e1:71:08:00 SRC=172.24.222.72 DST=193.99.144.80 LEN=64 TOS=0x10 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=52122 DPT=443 WINDOW=65535 RES=0x00 SYN URGP=0
*- kernel: ** VBE-CFo *IN=green0 OUT=tun4 MAC=00:0d:b9:4b:da:4d:d8:ec:5e:74:e1:71:08:00 SRC=172.24.222.72 DST=193.99.144.80 LEN=64 TOS=0x10 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=52122 DPT=443 WINDOW=65535 RES=0x00 SYN URGP=0
*- kernel: ** VBE-PRR *IN=green0 OUT= MAC=00:0d:b9:4b:da:4d:d8:ec:5e:74:e1:71:08:00 SRC=172.24.222.72 DST=193.99.144.80 LEN=64 TOS=0x10 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=52127 DPT=80 WINDOW=65535 RES=0x00 SYN URGP=0
*- kernel: ** VBE-CPR *IN=green0 OUT=tun4 MAC= SRC=172.24.222.72 DST=193.99.144.80 LEN=64 TOS=0x10 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=52127 DPT=80 WINDOW=65535 RES=0x00 SYN URGP=0
*- kernel: ** VBE-PRR *IN=green0 OUT= MAC=00:0d:b9:4b:da:4d:d8:ec:5e:74:e1:71:08:00 SRC=172.24.222.72 DST=193.99.144.80 LEN=64 TOS=0x10 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=52121 DPT=443 WINDOW=65535 RES=0x00 SYN URGP=0
*- kernel: ** VBE-CFo *IN=green0 OUT=tun4 MAC=00:0d:b9:4b:da:4d:d8:ec:5e:74:e1:71:08:00 SRC=172.24.222.72 DST=193.99.144.80 LEN=64 TOS=0x10 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=52121 DPT=443 WINDOW=65535 RES=0x00 SYN URGP=0
*- kernel: ** VBE-PRR *IN=green0 OUT= MAC=00:0d:b9:4b:da:4d:d8:ec:5e:74:e1:71:08:00 SRC=172.24.222.72 DST=193.99.144.80 LEN=64 TOS=0x10 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=52126 DPT=80 WINDOW=65535 RES=0x00 SYN URGP=0
*- kernel: ** VBE-CFo *IN=green0 OUT=tun4 MAC=00:0d:b9:4b:da:4d:d8:ec:5e:74:e1:71:08:00 SRC=172.24.222.72 DST=193.99.144.80 LEN=64 TOS=0x10 PREC=0x00 TTL=63 ID=0 DF PROTO=TCP SPT=52126 DPT=80 WINDOW=65535 RES=0x00 SYN URGP=0
My take from this log is:
- the packages seem to be marked correctly by the PREROUTING CHAIN
- the packages seem to be forwarded correctly by the CUSTOMFORWARD CHAIN
- the packages seem to be masqueraded correctly by the CUSTOMPOSTROUTING CHAIN
- but obviously I don’t get packages back via the tunX device – last rule in my chain (as the packages are going out via red0?)
- as I don’t see referring entries in the standard firewall log I assume somewhere here is the root cause for the problem. The packages seem to disappear - in the rp_filter? The filter value = 1
Any hints where to dig into?
Best summer greetings
ZierckeT
Remark: I also tried to use the fwmark method. Same result.