During the last week of December I upgraded our firewall from Core 152 to 153.
My wife returned to work today. She works in the schools and is mandated by her employer to use the GSuite which includes Google Meet for interacting with students (distance learning).
On Google Chrome, you could see that the browser was picking up audio and video, but her coworkers were not hearing audio and not seeing video from her.
After a bit of experimentation we found:
This outage affected everything connected to LAN and WLAN regardless of OS (Ubuntu, iOS, Android, Windows).
Firefox performed better where audio would be transmitted across the internet, but no video would make it out.
I reset my modem, reset IPFire. Still no dice. I verified that my ISP had no maintenance or outage on going in my area. I also verified that Google Meet wasn’t having some sort of outage today. I was hesitant to point the finger at IPFire as I have never had an issue, but after reading “After Core 153, Discord Cam no longer working” and a few others I suspected that Core 153 might be the cause of our video conferencing woes.
So I took the unprecedented step of downgrading from Core 153 to 152. After restoring everything from backup and reinstalling add-ons (I only use hostapd), I fired up a Google Meet room and regardless of browser or OS everything worked again (audio and video streamed between clients flawlessly) so the issue appears to be in Core 153 somewhere.
I would love to help out in solving this and if there are any other details that I can provide, please let me know.
I am also seeing the same issues after a recent update to 153 - both with Google Meet and with camera on Discord. With Discord, I was able to see the other people on the call, and audio worked both ways, but I could not get video out. With Google Meet, I am able to see the other participants status, but I can neither send nor receive audio and video. I’ve tested this on both Windows and an Android device. I can confirm that Meet was working correctly the last time I used it before the update and did not work the first time after I applied the update.
I’ve only been able to dig into this a little bit (so this may be a red herring), but it may be related to DTLS. Looking at a Wireshark trace, I see a repeating cycle where Google sends me a DTLS Client Hello, and my device (on the IPFire Green subnet) responds with the Server Hello, but the handshake never completes - nothing comes back from Google afterwards and the cycle starts over. I’ve checked the firewall logs (and /var/log/messages), but I don’t see any dropped packets to/from the Google server.
Other things I’ve tried:
Disabling QOS
Disabling IPS (after checking for and not finding the Google server’s IP in the logs)
Update: After rolling back to 152, Meet is working for me once again. I confirmed on Wireshark that I can see the full DTLS handshake, and then see application data start flowing for the call.
My outgoing rules are default. Actually, I only have two firewall rules which are “DMZ Pinholes” to allow clients on Blue to contact a server on Green.
I don’t enable QoS, IPS, or Web Proxy. My firewall is quite close to default. For example, I only use a single add-on: hostapd.
I have happily used IPFire for years and I haven’t made any changes to my configuration since sometime in 2018, but then after updating to Core 153 - Google Meet suddenly doesn’t function. When I dropped back to Core 152 everything functions fully again. Nothing was in the firewall logs showing that anything was being dropped between the clients and the Google Meet servers.
I firmly believe there is a bug lurking somewhere in Core 153. It seems to have impacted at least three users at this point - @prussbw, @soundconjurer, and myself.
Could you please raise a bug then. Your IPFire People login credentials work on Bugzilla as well as IPFire Community. Following wiki link gives info about raising bugs. https://wiki.ipfire.org/devel/bugzilla
Have you blocked some coutries via location block or block p2pnetworks. (This are the 2 functions that blocks connections without logging, all other should add log entries.)
For example blocking “A3 Worldwide Anycast Instance” may block some goolge servers.
I have no idea wich part of core153 can block a single service.
It is huge but this is because the kernel and microcodes are shipped with this update.
Here a list of all changed parts:
It isn’t a single service - Multiple others have said they have issues with Discord as well after Core 153. At least one user (@prussbw) has downgraded back to Core 152 and they have confirmed Discord functionality was restored.
I dont think so because I have installed core153 and use Nextcloud talk which also based on WebRTC without problems.
WebRTC use simple DTLS and SRTP streams which IPFire only sees as UDP connections.
But there is a problem with UDP behind NAT. Some protocols need a special NAT helper if there is more than one client that use the same protocoll behind the nat. Im not sure if there is something changed with this helpers in core153.
One thing I can confirm is that in both the Meet and Discord cases, STUN was in use for NAT traversal - I can see the session messaging in the trace.
I can also confirm that my firewall rules are also near-default. The only rules I’ve created are to block access to the IPFire Web UI from my Blue subnet.
It seems possible that the NAT traversal might be affecting the outgoing DTLS handshake message in an unsupported manner, causing it to fail integrity/authentication checking at the receiver’s side. I’ve seen similar cases in things I’ve worked on in the past, but I haven’t had time to create a test bed to verify that theory. (Meet is non-negotiable for me for work reasons right now, so I have to stay on 152 until there’s either a fix or a workaround.) Any test bed I setup would have to be VM-based, so hopefully that would be sufficient to prove this out.
I agree that there should be a bug logged for this, but I’m not sure how well-characterized you need it to be. (We’ve already seen that Meet is working for some others, so I don’t want it to become closed as could-not-reproduce.) Let me know if you have any recommendations along those lines.
In our LAN, this problem with core update 153 also affected communications with zoom meetings, fb messenger and whatsapp eversince it wath applied on dec. 22d. When client connected from inside LAN, people outside lan would only see a black screen, but audio did however work.
Resetting modems, switches, did not help.
Stopping Guardian, IPS and Location block (the only services that we use) did not help either.
We noticed however that videoconferencing through web browser did work, using jitsi or whereby
Reverting to core 152 last monday solved the problem and now all videoconferencing clients of various types work again…
I have core 153. I just signed up for a Zoom account, installed the Zoom app on my iMac, and did a 5 minute video call with my son (he is at work). All worked A-OK on both sides. I did not try the Zoom browser version.
Jon iMac → IPFire → Cable Modem → Internet ->> Son at work
So maybe there is something configured differently in the IPFire box??
My side:
Guardian - No
IPS - RUNNING on red only.
Location Block - enabled
Firewall Rules - all off except for DNS/NTP redirect (testing)
P2P networks - all disabled (not checked)
Default firewall behaviour - FORWARD & OUTGOING is Allowed
Web Proxy - Transparent
URL Filter - not enabled (not checked) EDIT: Quality of Service: RUNNING
@jon: FWIW Microsoft Teams and Zoom worked for us via the desktop on Core 153.
For us it was that Google Meet was not running fully via the browser. On Chrome, it was not transmitting Video and Audio and Firefox would transmit Audio to other users, but no Video. Before anyone suggests it was a browser setting, we ran our tests on default profiles (no add-ons, factory settings, etc.) to ensure nothing funky was going on with browser settng or add-on. It was only when returning to Core 152 where functionality was restored.
My wife is working from home and using MS Teams (and sometime Zoom) during the entire day. All is working A-OK with core 153. That is why I mentioned this:
We may need to look at the differences in setup to figure out what went wrong…