There are two android phones that one works one does not. Both the same model and the same age. What does not work is the VPN connections that they use to an external service. MS teams sharing does not seem to work. Though i have not tried thoroughly. I will say that everything was working fine after the upgrade and power cycling my router until the second phone came into the network about 3 hours later and then the cable modem needed to be power cycled but the new phone never came back online.
I wouldn’t know where to start to getting anonomized logs that would help with this. As Brian Pruss mentioned i cannot find any reference to mtu 576 in anything even after grepping the whole file system. It does seem clear that setting was tied to the red port connecting and disconnecting. But with my testing today it was not clear that that setting did much.
first, I am sorry to bother you testing something again.
A while ago, @ms found a long-standing bug in Suricata (see this patch and this PR for technical details), which especially caused trouble for some TCP connections on machines having the IDS enabled and being connected to the internet via a slow line (mostly latency, but bandwidth seems to matter as well).
Core Update 162 (testing) comes with this fix. Since the troubles here are related to UDP, it might not actually help much, but please give it that update a test to ensure it at least does not make things worse.
I believe I may have a fix (or at least a workaround) for this problem, and it seems to be related to dhcpcd. After making a manual change to /var/ipfire/dhcpc/dhcpcd.conf , I was able to upgrade to Core 161 and use Google Meet successfully. I am no longer seeing fragmentation on large UDP or ICMP packets.
The change I made was to comment out the “option interface_mtu” line in dhcpcd.conf under /var/ipfire/dhcpc/. After that, I rebooted, and verified that the “mtu 576” option was no longer present in the output of “ip route”. I was able to perform large pings from both the Green subnet and IPFire itself.
At this point I upgraded from Core 152 to Core 159. After the upgrade and reboot, I saw that the dhcpcd.conf change had been reverted, so I once again commented out “option interface_mtu”, rebooted, and tested both pings and Google Meet. All tests were successful. I then upgraded to Core 161. This time the dhcpcd.conf change was not clobbered, and I was able to once again run my tests successfully.
The odd thing about this is, after capturing a DHCP exchange between IPFire and my ISP, I don’t see the MTU option (26) being requested or sent at all. It’s obviously dhcpcd that’s passing the setting to the rest of the system, but I can’t see why it’s doing that. It appears to have been doing this since at least Core 152 if not earlier, it’s just that some other change in the packet flow appears to have made the route MTU setting affect Green<->Red packets starting in Core 153.
At this point I’m finally able to get current on my router software, so I’m satisfied for the moment. I don’t know what the best way would be to fix this problem in the IPFire core, but I’m willing to be a tester for any changes the developers may propose. I’ll copy the above to the bugzilla thread. Thanks again to everyone that helped.
I have not tried CU 162 yet. (It’s a long story and I’ll spare you the details, but I haven’t been able to get time for it.) Based on what I’ve seen and what @pmueller wrote, it doesn’t look like the Suricata update will affect the issue for me, although it may make for a general improvement in performance.
What I’ve found is that the problem goes away (for me at least) when the MTU configuration in the routing rules is removed. That appears to be getting set by dhcpcd for reasons that I have yet to understand, even though I’ve verified that the DHCP server isn’t sending MTU configuration. Manually changing the dhcpcd configuration to ignore the MTU clears up the problem for me, but I’d still like to find a way that the problem could be solved in mainline IPFire.
Wow, thank you for this, I just got my rpi CM4 setup and once again had issues with Google Meets and Microsoft Teams Meetings and this solved my issue.
Oddly enough my point-to-point calls worked fine on Teams but joining scheduled meetings I was unable to share a screen. Commenting out “option interface_mtu” in /var/ipfire/dhcpc/dhcpcd.conf and a reboot fixed the issue.
just for the records, and to have everyone informed:
Upcoming Core Update 165 finally fixes this bug, which is sometimes caused by some cable modems or ISP equipment propagating a very low MTU. It would be great if those of you who are still experiencing trouble could test Core Update 165, and report if it fixes the issue for you as well.
I finally upgraded today after I saw the notification that 165 fixes this bug, but unfortunately it did not.
The workaround suggested by @prussbw worked for me as well! See below.
The change I made was to comment out the “option interface_mtu” line in dhcpcd.conf under /var/ipfire/dhcpc/. After that, I rebooted, and verified that the “mtu 576” option was no longer present in the output of “ip route”. I was able to perform large pings from both the Green subnet and IPFire itself.
@pmueller and @ms: I could not get two Meet clients to fuly work with 166 (audio and video). The only difference I noticed was that Meet via Chrome could get audio this time around whereas back on 153 it could not and only Firefox could get audio. So either the client-side code in Chrome improved to handle what was happening or IPFire improved. However, video was still not working… I tried the workaround above and it worked.
Please let me know if there is anything I can do to assist in cornering and squashing this bug.
Below the test of downloading 10GB file from ftp://ftp. task.gda.pl/test/10giga
Below a fragment of the ip route command result
default via 8x.xx.xx.xx dev red0 proto dhcp src 8x.xx.xx.xx metric 1003 mtu 1500
8x.xx.xx.xx/22 dev red0 proto dhcp scope link src 8x.xx.xx.xx metric 1003 mtu 1500
Below the test of downloading 10GB file from ftp://ftp. task.gda.pl/test/10giga
Below a fragment of the ip route command result
default via 8x.xx.xx.xx dev red0 proto dhcp src 8x.xx.xx.xx metric 1002 mtu 576
8x.xx.xx.xx/22 dev red0 proto dhcp scope link src 8x.xx.xx.xx metric 1002 mtu 576
Best regards.
Tomas
edit:
Tested on version:
IPFire 2.27 (x86_64) - Core Update 166 Development Build: master/8f696f60
Could you please post some more details about this? The output of “ip route” for example?
Hi @ms, sorry for the delay. Reverted the workaround, rebooted, and then ran ip route.
Here is the output of ip route:
default via 7x.xx.xx.xx dev red0 proto dhcp src 7x.xx.xx.xx metric 1003 mtu 576
7x.xx.xx.0/22 dev red0 proto dhcp scope link src 7x.xx.xx.xx metric 1003 mtu 576
192.168.xx.0/24 dev green0 proto kernel scope link src 192.168.xx.xx
192.168.xx.0/24 dev blue0 proto kernel scope link src 192.168.xx.xx
No, this happens just on plain installs and updates (post-152). As long as option interface_mtu is left on then this is what happens.
576 is not a random number either as 576 bytes is the minimum IPv4 datagram size that every host shall be capable of receiving. Although that is more of a historical footnote as most modern hosts can handle larger sizes.
I saw this GitHub issue for OPNsense (different as FreeBSD-based) where they were seeing similar issues suddenly on some of their userbase after an upgrade back in 2019: GitHub Issue Link.
well, to the best of my understanding, this is the way things are designed here: Since IPFire cannot know what MTU to set instead of the crappy 576 bytes propagated by the ISP modem, that has to be properly reconfigured manually.
Some connections work fine with an MTU of 1500 bytes. Some others, such as PPPoE dial-up ones, probably won’t. I don’t really think this is a task we can shed from our users’ shoulders.
The odd thing about this is, after capturing a DHCP exchange between IPFire and my ISP, I don’t see the MTU option (26) being requested or sent at all. It’s obviously dhcpcd that’s passing the setting to the rest of the system, but I can’t see why it’s doing that. It appears to have been doing this since at least Core 152 if not earlier, it’s just that some other change in the packet flow appears to have made the route MTU setting affect Green<->Red packets starting in Core 153.
Does the team have a recommended way to grab DHCP traffic running between IPFire and ISP DHCP Server?
If all of our ISPs end up being at fault, then two questions remain:
Why Post-152 - What exactly changed? Knowing the root cause is always comforting.
May we have a simple method in the IPFire Web Admin to override the default in a configuration setting. This workaround gets clobbered after some updates and a simple way to easily setup and persist this change would be helpful to those of us that are impacted.
I’m inclined to believe that this is incorrect behavior on the part of dhcpcd: If the DHCP server doesn’t specify an MTU (as is happening in my case), I would expect that the route MTU should be left unspecified, thereby letting the interface use its default. Instead, dhcpcd appears to be overruling the interface and setting MTU to the lowest possible setting, which seems over-conservative to me (given the application problems we’ve seen). The only current way to keep it from not specifying an MTU on the route appears to be to disable option interface_mtu completely, which of course has problems of its own from the POV of a widely deployed, general-purpose firewall like IPFire.
I’m considering raising this as a question on the dhcpcd Issues board. Perhaps they have a way around this that we haven’t found yet. However, I don’t want to give the impression that I’m speaking for the IPFire development team, at least without discussing it first. Any suggestions?
At some point the mechanism changed that dhcpcd uses to enforce the MTU. Instead of changing the MTU of the interface, they add it to the route. This is much better.
And before, we caught 576 as an invalid value (because it is unreasonably small) which we cannot do now because dhcpcd sets the MTU internally.
dhcpcd does not temper with the MTU at all unless it is told to. The modem sends those 576 bytes. That is why it needs to manually be overwritten now.
This makes dhcpcd ignore what the modem is sending, but generally we don’t want this. There are setups where people have a slightly smaller than usual MTU (1492, 1464) in their local networks; and some people are brave and run jumbo frames.
closing this thread for the sake of completeness and transparency, since the issue has been fixed in Core Update 165, although it requires manual interaction (i.e. configuring the appropriate MTU in IPFire).
