UDP leakage from Green to red after red router swapout ?!?

Networking
1

Hi All,

Running IPFire 2.29 (x86_64) - Core-Update 184 here as an internal network.

the high level setup is depicted below:

image
image911×973 49.2 KB

We cutover from legacy DOCSIS cable last week to Direct Fibre. The new ISP router was seriously AWFUL! so we replaced it with a Draytek which seems to have fixed all of the immediate problems like the DMZ not working in the ISP provided router.

I noticed that UDP traffic from green to red was working for several services, but not for 2 others.

Tcpdump’ing red I could see the traffic outbound but the issue didn’t dawn on me at the time as in my mind, I was still inside the firewall and before iptables but with hindsight it should have been obvious (my bad).

For the one service, I contacted the server owner who said nothing but mysteriously 10 minutes later everything started working again there so I suspect the problem was their end…

But I couldn’t initially resolve the second one.

I enabled a monitor port on the draytek and fired up wireshark.on a laptop connected directly to that draytek monitor port

To my surprise, I saw 192.168 traffic being pushed from green out into the red subnet without NATing - but only for this one service…

The upstream Draytek behaved correctly and dropped the traffic as the source IP was not in the 10.x.x.x network

I rebooted IPFire and the problem went away, so I dont believe it to be repeatable without multiple reboots /swapouts of the ISP router.

I can only conclude that this is a feature / bug in IPfire ?

Anyone seen this before?

if so , is there a workaround to prevent it occurring again please ?

Regards

“The relieved to have found it but perplexed that it happened”

BB

2

Could the UDP traffic have come from
DNS clients in your Green not using your Green DNS?
Reboot would force clients to update connections. Hopefully using IPFire DNS.

1 Like
3

Hi Shaun

No the 2 streams were UDP connections sending positioning data to tracking servers (one AIS [shipping] and one ADS-B [Aircraft] ) both sessions were established and were running 24/7 prior to the swap-out of the router.

I had anticipated that the dropping of the red interface on disconnection from the ISP supplied router and reconnection to the new router (different IP subnets but IPFIre had DHCP reservations in both boxes) would simply suspend traffic resulting in dropped UDP packets until the new IP address was assigned.

Instead, what happened was that source traffic from the sending hosts in the green subnet leaked out into the red external DMZ unNATed, and continued to do so, until such time as IPFire was rebooted.

fortunately the upstream router (new one) had a default setting whereby any traffic entering the Red DMZ that did not have a valid source IP (10.x.x.x./24) would be blocked.

other hosts in green and one in blue was also forwarding tracking and other data to external servers (a mix of TCP and UDP) , but remained unaffected.

its a weird one for sure .

Regards

BB

4

if it happens again, you could use iptables trace to trace the packet, see https://youtu.be/9HNKRP7x57M?si=eC9lC1TehILwqFyS, if I recall correct, ipfire seems already enabled packet logging/tracing

5

It would be helpful to post the hint in text form than as video link.
I’m not willing to veriy those clips, and I fear I’m speaking for other mods also.

2 Likes
6

How to trace IPTables in RHEL7 / CENTOS7 – Open Sourcerers has the text form tips that the video is based on

7

Well I just upgraded to 185…and now it doing it all the time :-/

You can see from connections.cgi its just one connection that
doesn’t NAT:

image

If I stop the source client process manually and restart it normality is resumed:

image

(Admittedly In the second pic I have done a “cut and shut” as line 3 was at the top of the table and lines 1+2 were not.)

The issue is clearly repeatable by breaking the ethernet connection on the red interface and moving it between routers.

Whilst in Ipfire 184 this could be fixed by a reboot of IPFire, in 185 that no longer seems to be the case? and I have to manually stop the source from transmitting to let IPFire Catch up.

Some kind of UDP input buffer overflow when red goes down ?

I would expect UDP packets to simply be dropped

I intended to raise it on bugzilla but I havent got the password reset email after 12 hours

Always the same source process I note, but I see no reason for IPFire to treat it any differently to the two parallel clients also running.

Regards

BB