Connection Tracking dropping DNS TCP sessions

callifo · 3 March 2025 06:48

Hi,

I’ve been having some issues whereby DNS will stop working periodically (most notably, webpages will stop loading for a short period of time). My config is as follows:

Green Client ↔ Orange Pihole (ipfire libvirt VM) ↔ IPFire ↔ Internet

Clients point to Pihole, Pihole forwards upstream to IPfire, and then IPFire points to Cloudflare (TLS).

I see on the firewall logs, that sometimes the DNS TCP session between Pihole and IPfire is being dropped due to DROP_CTINVALID. (101.1 is IPfire, 101.2 is Pihole) I have inbound rules permitting DNS traffic from Pihole to the Orange FW interface.

These look to occur around the time the TCP connection in the tracking page expires. I tried increasing TIME_WAIT from 2 minutes to 5 minutes, but it didnt stop it occuring, just less often.

Pihole looks to keep retransmitting as it doesnt realise the TCP connection is being blocked.

Mar  3 17:06:13 ipfire kernel: DROP_CTINVALID IN=orange0 OUT= MAC=02:3b:48:cf:a4:eb:52:54:00:e8:bb:20:08:00 SRC=192.168.101.2 DST=192.168.101.1 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=39555 DF PROTO=TCP SPT=41474 DPT=53 WINDOW=501 RES=0x00 ACK FIN URGP=0
Mar  3 17:06:15 ipfire kernel: DROP_CTINVALID IN=orange0 OUT= MAC=02:3b:48:cf:a4:eb:52:54:00:e8:bb:20:08:00 SRC=192.168.101.2 DST=192.168.101.1 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=39556 DF PROTO=TCP SPT=41474 DPT=53 WINDOW=501 RES=0x00 ACK FIN URGP=0
Mar  3 17:06:19 ipfire kernel: DROP_CTINVALID IN=orange0 OUT= MAC=02:3b:48:cf:a4:eb:52:54:00:e8:bb:20:08:00 SRC=192.168.101.2 DST=192.168.101.1 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=39557 DF PROTO=TCP SPT=41474 DPT=53 WINDOW=501 RES=0x00 ACK FIN URGP=0
Mar  3 17:06:25 ipfire kernel: DROP_CTINVALID IN=orange0 OUT= MAC=02:3b:48:cf:a4:eb:52:54:00:e8:bb:20:08:00 SRC=192.168.101.2 DST=192.168.101.1 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=39558 DF PROTO=TCP SPT=41474 DPT=53 WINDOW=501 RES=0x00 ACK FIN URGP=0
Mar  3 17:06:39 ipfire kernel: DROP_CTINVALID IN=orange0 OUT= MAC=02:3b:48:cf:a4:eb:52:54:00:e8:bb:20:08:00 SRC=192.168.101.2 DST=192.168.101.1 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=39559 DF PROTO=TCP SPT=41474 DPT=53 WINDOW=501 RES=0x00 ACK FIN URGP=0
Mar  3 17:07:05 ipfire kernel: DROP_CTINVALID IN=orange0 OUT= MAC=02:3b:48:cf:a4:eb:52:54:00:e8:bb:20:08:00 SRC=192.168.101.2 DST=192.168.101.1 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=39560 DF PROTO=TCP SPT=41474 DPT=53 WINDOW=501 RES=0x00 ACK FIN URGP=0

Anyone assist with what might be causing this? Does DNS over TCP hold the socket open, but not send keepalives, so eventually the firewall is closing the connection?

Is there a fix for this behaviour, e.g. permanently allow the connections without tracking them in the firewall?

I tried Google as to whether Pihole/dnsmasq can configure TCP session timelimits or keepalives but I’ve not yet found a solution.

callifo · 3 March 2025 06:55

Another separate issue, which is actually what lead me to find the above issue, looks to be the almost identical.

This seems to occur for DNS over TLS (DoT) for my mobile devices which use the IPfire red public IP permanently for DoT - even when at home (thereby using NAT loopback).

The response (from the DoT server, different IP from Pihole), gets dropped when heading back to the IPfire Green interface.

These drops are much more evident then the first issue. Private DNS (DoT) will stop getting responses permanently until you disable, and then reenable it in Android. Everything on the phone grinds to a halt.

arne_f · 3 March 2025 09:02

Check /var/log/messsages for the detailed output.

I have seen this with some strange packets that has ACK and RST flags set.

callifo · 3 March 2025 22:00

I got this from /var/log/messages, is this what you are after, or is there something more detailed?

Nicholas Perin:

Mar  3 17:06:13 ipfire kernel: DROP_CTINVALID IN=orange0 OUT= MAC=02:3b:48:cf:a4:eb:52:54:00:e8:bb:20:08:00 SRC=192.168.101.2 DST=192.168.101.1 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=39555 DF PROTO=TCP SPT=41474 DPT=53 WINDOW=501 RES=0x00 ACK FIN URGP=0
Mar  3 17:06:15 ipfire kernel: DROP_CTINVALID IN=orange0 OUT= MAC=02:3b:48:cf:a4:eb:52:54:00:e8:bb:20:08:00 SRC=192.168.101.2 DST=192.168.101.1 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=39556 DF PROTO=TCP SPT=41474 DPT=53 WINDOW=501 RES=0x00 ACK FIN URGP=0
Mar  3 17:06:19 ipfire kernel: DROP_CTINVALID IN=orange0 OUT= MAC=02:3b:48:cf:a4:eb:52:54:00:e8:bb:20:08:00 SRC=192.168.101.2 DST=192.168.101.1 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=39557 DF PROTO=TCP SPT=41474 DPT=53 WINDOW=501 RES=0x00 ACK FIN URGP=0
Mar  3 17:06:25 ipfire kernel: DROP_CTINVALID IN=orange0 OUT= MAC=02:3b:48:cf:a4:eb:52:54:00:e8:bb:20:08:00 SRC=192.168.101.2 DST=192.168.101.1 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=39558 DF PROTO=TCP SPT=41474 DPT=53 WINDOW=501 RES=0x00 ACK FIN URGP=0
Mar  3 17:06:39 ipfire kernel: DROP_CTINVALID IN=orange0 OUT= MAC=02:3b:48:cf:a4:eb:52:54:00:e8:bb:20:08:00 SRC=192.168.101.2 DST=192.168.101.1 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=39559 DF PROTO=TCP SPT=41474 DPT=53 WINDOW=501 RES=0x00 ACK FIN URGP=0
Mar  3 17:07:05 ipfire kernel: DROP_CTINVALID IN=orange0 OUT= MAC=02:3b:48:cf:a4:eb:52:54:00:e8:bb:20:08:00 SRC=192.168.101.2 DST=192.168.101.1 LEN=52 TOS=0x00 PREC=0x00 TTL=64 ID=39560 DF PROTO=TCP SPT=41474 DPT=53 WINDOW=501 RES=0x00 ACK FIN URGP=0

callifo · 14 March 2025 01:31

Is there anyway of disabling connection tracking for DNS specifically, and just allow it permanently through?

dr_techno · 17 March 2025 14:32

Sorry I’ve been busy, but the issue comes from the pihole wanting to be a DNS resolver while ipfire has a resolver built in. Most routers/networks don’t have a local DNS resolver and the Pihole was designed for that, but for it to work with IPFire, you have to change the pihole into a forwarding dns to ipfire, which should be the resolver for all dns queries and not cloudflare or any other dns outside the network.

callifo · 18 March 2025 11:37

My Pihole is forwarding to IPfire, which then IPfire forwards to Cloudflare (TLS + DNSSEC). There is no forwarding direct to cloudflare from Pihole.

arne_f · 18 March 2025 14:08

I still not fully understand why this is marked as CT_INVALID but FIN is set so this paket will close the tcp connection. Maybee it was send to late or twice because the reciever has not aknowledge the last message. The reason for this is the server or client, not the IPFire.

callifo · 18 March 2025 23:30

Hi Arne, in this case, IPfire is the server. Pihole is sending client DNS requests to the IPfire DNS server on the Orange interface. I guess TCP is being used due to the use of DNSSEC.

tphz · 18 March 2025 23:43

callifo · 18 March 2025 23:50

I had created a forwarding rule allowing DNS into the orange interface,

This page does say,

There is no DNS server in the IPFire DMZ.

However this does not appear to be the case in the unbound config,
/etc/unbound/unbound.conf
# Listen on all interfaces
interface-automatic: yes
interface: 0.0.0.0

callifo · 19 March 2025 04:00

I think I found it, I’m trying to block internal hosts using DNS to the internet directly, as so much stuff will go around local DNS servers (Google + IOT devices), so I put this rule in place:

First rule allows specific hosts to the internet DNS (for testing, etc)

Second rule is Group-Hosts-Internal (Group consisting of GREEN network, BLUE network, and OpenVPN network) to 8.8.8.8, 8.8.8.4, 1.0.0.1 and 1.1.1.1 drop - this addresses DNS (udp/tcp), DoH and DoT. Seen some sneaky devices use DoT/DoH to get around standard blocks. Hard to block DoH as I would need to block all HTTPS websites

Third rule is Group-Hosts-Internal (Group consisting of GREEN network, BLUE network, and OpenVPN network) to RED network with TCP/UDP DNS.

This however appears to be leaking. Here is what is happening with one of my devices, and then the DROP_CTINVALID

> 14:09:19.572504 IP 192.168.100.145.64401 > 8.8.8.8.53: Flags [S], seq 3255670883, win 32768, options [mss 1464,nop,wscale 0,sackOK,nop,nop,nop,nop,TS val 0 ecr 0], length 0
> 14:09:19.572857 IP 8.8.8.8.53 > 192.168.100.145.64401: Flags [S.], seq 1653372222, ack 3255670884, win 0, options [mss 1460], length 0
> 14:09:19.612541 IP 192.168.100.145.64401 > 8.8.8.8.53: Flags [.], ack 1, win 33672, length 0
> 14:09:19.612541 IP 192.168.100.145.64401 > 8.8.8.8.53: Flags [F.], seq 1, ack 1, win 33672, length 0
> 14:09:22.289508 IP 192.168.100.145.64403 > 8.8.8.8.53: Flags [F.], seq 0, ack 1, win 33672, length 0
> 14:09:22.330981 IP 192.168.100.145.64455 > 8.8.8.8.53: Flags [R.], seq 1157608426, ack 344094030, win 33672, length 0
> 14:09:23.252971 IP 192.168.100.145.64449 > 8.8.8.8.53: Flags [F.], seq 3294787064, ack 1343020989, win 33672, length 0
> 14:09:24.142859 IP 192.168.100.145.64443 > 8.8.8.8.53: Flags [F.], seq 1139200915, ack 196475112, win 33672, length 0

The TCP Start packet gets passed, then all other packets get dropped.

14:27:00  DROP_CTINVALID  IN=blue0 OUT=red0 MAC=02:da:b6:2d:9e:fe:50:33:8b:73:60:f3:08:00 SRC=192.168.100.145 DST=8.8.8.8 LEN=40 TOS=0x00 PREC=0x00 TTL=63 ID=33641 DF PROTO=TCP SPT=64330 DPT=53 WINDOW=33672 RES=0x00 ACK FIN URGP=0 
14:26:59  DROP_CTINVALID  IN=blue0 OUT=red0 MAC=02:da:b6:2d:9e:fe:50:33:8b:73:60:f3:08:00 SRC=192.168.100.145 DST=8.8.8.8 LEN=40 TOS=0x00 PREC=0x00 TTL=63 ID=33640 DF PROTO=TCP SPT=64294 DPT=53 WINDOW=33672 RES=0x00 ACK FIN URGP=0 
14:26:59  DROP_CTINVALID  IN=blue0 OUT=red0 MAC=02:da:b6:2d:9e:fe:50:33:8b:73:60:f3:08:00 SRC=192.168.100.145 DST=8.8.8.8 LEN=40 TOS=0x00 PREC=0x00 TTL=63 ID=33639 DF PROTO=TCP SPT=64295 DPT=53 WINDOW=33672 RES=0x00 ACK FIN URGP=0 
14:26:59  DROP_CTINVALID  IN=blue0 OUT=red0 MAC=02:da:b6:2d:9e:fe:50:33:8b:73:60:f3:08:00 SRC=192.168.100.145 DST=8.8.8.8 LEN=40 TOS=0x00 PREC=0x00 TTL=63 ID=33638 DF PROTO=TCP SPT=64336 DPT=53 WINDOW=33672 RES=0x00 ACK FIN URGP=0 
14:26:58  DROP_CTINVALID  IN=blue0 OUT=red0 MAC=02:da:b6:2d:9e:fe:50:33:8b:73:60:f3:08:00 SRC=192.168.100.145 DST=8.8.8.8 LEN=40 TOS=0x00 PREC=0x00 TTL=63 ID=33637 DF PROTO=TCP SPT=64342 DPT=53 WINDOW=33672 RES=0x00 ACK FIN URGP=0 
14:26:57  DROP_CTINVALID  IN=blue0 OUT=red0 MAC=02:da:b6:2d:9e:fe:50:33:8b:73:60:f3:08:00 SRC=192.168.100.145 DST=8.8.8.8 LEN=40 TOS=0x00 PREC=0x00 TTL=63 ID=33636 DF PROTO=TCP SPT=64348 DPT=53 WINDOW=33672 RES=0x00 ACK RST URGP=0 
14:26:57  DROP_CTINVALID  IN=blue0 OUT=red0 MAC=02:da:b6:2d:9e:fe:50:33:8b:73:60:f3:08:00 SRC=192.168.100.145 DST=8.8.8.8 LEN=40 TOS=0x00 PREC=0x00 TTL=63 ID=33635 DF PROTO=TCP SPT=64296 DPT=53 WINDOW=33672 RES=0x00 ACK FIN URGP=0 
14:26:54  DROP_CTINVALID  IN=blue0 OUT=red0 MAC=02:da:b6:2d:9e:fe:50:33:8b:73:60:f3:08:00 SRC=192.168.100.145 DST=8.8.8.8 LEN=40 TOS=0x00 PREC=0x00 TTL=63 ID=33633 DF PROTO=TCP SPT=64294 DPT=53 WINDOW=33672 RES=0x00 ACK FIN URGP=0 
14:26:54  DROP_CTINVALID  IN=blue0 OUT=red0 MAC=02:da:b6:2d:9e:fe:50:33:8b:73:60:f3:08:00 SRC=192.168.100.145 DST=8.8.8.8 LEN=40 TOS=0x00 PREC=0x00 TTL=63 ID=33630 DF PROTO=TCP SPT=64301 DPT=53 WINDOW=33672 RES=0x00 ACK FIN URGP=0 
14:26:54  DROP_CTINVALID  IN=blue0 OUT=red0 MAC=02:da:b6:2d:9e:fe:50:33:8b:73:60:f3:08:00 SRC=192.168.100.145 DST=8.8.8.8 LEN=40 TOS=0x00 PREC=0x00 TTL=63 ID=33629 DF PROTO=TCP SPT=64295 DPT=53 WINDOW=33672 RES=0x00 ACK FIN URGP=0

Any suggestions why this is occuring? These error messages are the same and could be related to the DNS server issue too?

callifo · 19 March 2025 06:11

Disabling these two block rules seem to fix both issues above. I can see the connections arent hanging open in TIME_WAIT until they expire, drop off the Connection Tracking table, and then start to hit the DROP_CTINVALID rule.

I can see here, the DoT connection, was opened by my Android phone, answered, and then closed normally.

Any suggestions how to redo the rules so that I can block GREEN/BLUE from accessing internet DNS, without catching other DNS traffic moving between GREEN/BLUE and ORANGE or ORANGE → ORANGE FW?

I’m not sure why they are catching other traffic, but they seem to…

EDIT: Looks to be that the rules I created, whilst they work in the GUI, don’t create valid rules in IPTABLES.

Group to Group (internal network to specific internet DNS), does this:

DROP       tcp  --  192.168.100.0/24     dns.google
DROP       tcp  --  192.168.100.0/24     dns.google
DROP       tcp  --  192.168.100.0/24     one.one.one.one
DROP       tcp  --  192.168.100.0/24     one.one.one.one
DROP       tcp  --  192.168.10.0/24      dns.google
DROP       tcp  --  192.168.10.0/24      dns.google
DROP       tcp  --  192.168.10.0/24      one.one.one.one
DROP       tcp  --  192.168.10.0/24      one.one.one.one

Looks ok

However Group to Standard Network (RED) + Service Group, does this:

DROP       tcp  --  192.168.100.0/24     anywhere             multiport dports domain,domain-s
DROP       tcp  --  192.168.10.0/24      anywhere             multiport dports domain,domain-s
DROP       udp  --  192.168.100.0/24     anywhere             udp dpt:domain
DROP       udp  --  192.168.10.0/24      anywhere             udp dpt:domain

The source group, and service group survive, but the Standard Network (RED) group is not added to the rule, and it results in any traffic from those addresses being dropped.

hvacguy · 19 March 2025 19:58

Perhaps this would be a better way round.

callifo · 20 March 2025 06:22

Given every dns request will be green/blue to orange it mean every request would be NATd not just the ones attempting to evade local DNS.

Would prefer not to have to do that, blocking is preferable as it forces those devices to just use local DNS.

Might still play around with a permit rule internally then block all public address space by range.

On a side note will experiencing the DROP_CTINVALID messages with the DNS request from pihole to orange IPfire with the DNS rules disabled

callifo · 23 March 2025 11:33

As best as I can tell this seemed to be caused by the use of ‘Enable SYN Flood Protection (TCP only)’ on a rule.

I didn’t intentionally enable it, but this seems to kill any TCP traffic that matches the rule. I was using forwarding rules to allow forwarding of traffic from any network to the Pihole for DNS, and traffic from Pihole to the Orange IPfire interface. Using wireshark I could see that the client would send a SYN, server respond with a SYN ACK, then client would just sent unanswered ACK’s.