Google Meet Stopped Working After Core Update 152->153

Hi @mstriani

Then could you please add that information to the bugzilla bug. You can login with your IPFire Community password and email.

I tried to login but unsuccessfully

Hmm. I just tried it and it worked fine for me.

You did use the email that you are registered in IPFire Community with and not your IPFire Community username.

Update: Iā€™ve captured a success and failure case, with traces taken from both the Green and Red subnets each time. Iā€™ve found that there is a difference in the fragmentation of outgoing packets between 152 and 156.

Iā€™ve recently managed to get a Red/Green capture of a failed Google Meet session running through Core Update 156, and then rolled back to 152 to get a Red/Green capture of a successful session. Some notes:

  • In all cases, I set up a Meet session on my phone connected to LTE (not connected to WiFi and not going through IPFire). I then connected to that session from a PC connected to IPFire via wired Ethernet.
  • The fail-case symptoms I noted were slightly different from my original issue. Previously, I was unable to send or receive audio or video on the PC. This time, I was able to receive video and send audio, but was still unable to send video. This is similar to the issue I saw with Discord video chat.
  • I also no longer see the DTLS handshake failure. I donā€™t have an explanation for that - perhaps something changed on Googleā€™s side that affected it.
  • The difference Iā€™ve found between 152 and 156, is that in 156, UDP packets over a certain size (>580 bytes) are getting fragmented when traversing the firewall, whereas in 152 they are passed whole. In both cases the packets are smaller than most common MTUs, so I donā€™t have an explanation for the change.
  • I theorize that these larger packets are part of the outgoing video stream, and Google is rejecting them when fragmented. The smaller packets are likely the audio stream and are not fragmenting, thus why I can get audio out but not video.

The captures attached have again been trimmed and anonymized. Please reach out to me directly if you need the originals. I will also update the Bugzilla record.

google_meet_pcaps_anonimized.zip (1.9 MB)

2 Likes

Brian - thank you for testing and adding your notes here (and to bugzilla!). Hope this helps!

Jon

Thanks Brian for looking at this. I too have been waiting for this to be fixed before initiating any other upgrade. Iā€™m curious if you took a look at the adapter MTU? Did you at all run your app through a VPN and if that changed behaviour at all?

If anyone has to contribute any more data, I would be happy to hear it. So far the assumption is that this is a kernel bug, but the question is where is the bug and why is it thereā€¦

Hi Michael, IRacing was another application affected by the bug. According to ā€œthe webā€, they strictly use UDP. The behavior appeared that the client lost connectivity with the server and timed out. I tried using an OpenVPN connection and the connection no longer timed out. I cannot remember if I ran the connection as UDP or TCP (last I checked the app was set to UDP).

Interesting. Obviously it would not matter what application is being used, because the kernel does not care for what is in the packets.

However, literally everyone is using video conferencing, VoIP or a VPN, and they generally work. So something must be different on the systems of the affected users.

Do you have access to our server profiles so that they could be compared?

What do you want to compare to what?

Our different hardwaresā€¦ NICs etc

I do not think this is hardware-relatedā€¦

I was not thinking hardware - I was thinking driversā€¦ how often do you pick up driver updates and then roll them into your builds.

Hmm, about once a week? We test them and roll them out with about every second core update.

As you suggested in the Bugzilla entry, I am planning to repeat this on 157, but Iā€™m not sure when Iā€™ll get a chance. If thereā€™s anything else specific youā€™d like me to check while 157 is running, please let me know ahead of time if possible.

Hello,

Core Update 157 is a good starting point, however trying the development kernel is a good chance as well. You can install both and boot into one or the other.

Hi all,

for the records: Since Core Update 153, I suffered from VoIP call oddities every now and then, which I blamed broken connection tracking or another Linux kernel bug for:

https://lists.ipfire.org/pipermail/development/2021-March/009656.html

Having read about the fragmentation issue of very small UDP packages, this might or might not be related. In this case, I thought it might be interesting for you to know that I am unable to reproduce the VoIP problems after having installed Core Update 157 and rebooted afterwards.

I would be very happy if someone could confirm the Google Meet issue being gone after this update as well. :slight_smile:

Thanks, and best regards,
Peter MĆ¼ller

4 Likes

Hi all,

can anyone confirm or deny this problem to persist on Core Update 157?

In addition, could someone please install Core Update 158 (testing) and tell us if that update changed the behaviour again?

Thanks, and best regards,
Peter MĆ¼ller

Iā€™ve just tried 157, and have confirmed that I am still seeing the issue.

Iā€™m attaching a new set of PCAPs, but they just show the same fragmentation issue as before. Iā€™ve also thrown in the output from ā€˜ip linkā€™, ā€˜netstat -sā€™, ā€˜iptables -Lā€™ and ā€˜iptables -Sā€™ in case you can find anything useful in them. All Iā€™ve found is that the MTU is still set to 1500 as expected, so I canā€™t figure out why fragmentation is being applied. (I also did a ā€œgrep -IR /var/log/ā€ and verified that nothing showed up.)

Iā€™ll leave 157 in place until tomorrow evening local time (US), so if thereā€™s anything you want me to try, please let me know. Iā€™ll need to roll it back to 152 by Monday morning because I need Meet for work.

google_meet_pcaps_anon_157.zip (1.8 MB)

1 Like