Hi -
I am a long time user and supporter of IPFIRE. When upgrading to CORE 187, I encountered a particularly nasty bug which left my entire network exposed to the internet. Here is what happened:
TLDR: When the system’s out-of-memory killer was triggered (by a bug in Suricata?), my IPFIRE firewall was suddenly left completely open, exposing all ipfire service ports to the internet and disabling all firewall rules across GREEN/ORANGE/BLUE/RED. This happened several hours after upgrading to CORE 187, and I am able to replicate the bug on a fresh install of CORE 187 after enabling only Suricata and the emergingthreats community rules.
This is a particularly scary bug to me, because I would reasonably expect a firewall that runs out of memory to behave in a FAIL-SAFE manner (blocking all connections, for example), which is NOT what happened. Instead, my IPFIRE system (and the networks connected to it) were left exposed to the internet.
Long version:
On the morning of August 9, I upgraded to CORE 187 (from 184), did a quick test where everything seemed fine, and walked away to do other things. Several hours later (while watching TV), I got a strange alert from one of my monitoring systems telling me that two machines that should NOT be able to talk to each other across the firewall (one on BLUE, one on GREEN) were in-fact suddenly communicating. Upon looking at this, I immediately saw that the BLUE to GREEN connections were no longer being blocked by the ipfire firewall.
I then went to grc.com ShieldsUP, where it showed that my IPFIRE machine was now completely exposed to the internet, with all running services exposed (SSH, HTTPS, etc.)
At the time, I did not understand how this happened, so, in a bit of a panic, I quickly shut the machine down and disconnected the internet and then proceeded to rebuild a new firewall machine from scratch, leaving the old one for future forensic analysis.
Today, I had the time to go back and forensically analyze the logs and the system, to try to find out what happened and whether I had been actively hacked.
I found the following two events which seemed correlated:
-
I was alerted about the “connectivity issue” at 2024-08-09 18:18:15 -0400, and shortly after that (as described above), I discovered that my firewall had effectively been disabled.
-
/var/log/messages shows the oom-killer kicking in about 1 minute earlier:
Aug 9 18:17:03 ipfire kernel: Suricata-Main invoked oom-killer: gfp_mask=0x140dca(GFP_HIGHUSER_MOVABLE|__GFP_COMP|__GFP_ZERO), order=0, oom_score_adj=0
The fact that they were time-correlated made me suspicious, so I wanted to see if I could reproduce the failure.
I then booted up the old/failed ipfire system in a lab environment to see how it behaved.
The good news is upon booting, everything looked fine and nothing was incorrectly exposed to the internet.
I googled how to simulate an out-of-memory oom-failure and came across the following suggestion, which I tried:
swapoff -a && tail /dev/zero
Upon doing this, I found the ipfire machine was now in the same state that I had seen on August 9. All IPFIRE service ports were EXPOSED on the RED interface (!!)
I then tried re-installing CORE 184 and restoring from a backup. Through limited testing, I was NOT able to reproduce the failure on CORE 184.
I then upgraded the system to CORE 187. Once I did that, I was able to again replicate the failure with the same commands listed above.
I then spent several hours trying to create a simplified and reproducible test case on a fresh install of CORE 187. Here are the steps that I came up with:
-
Fresh install CORE 187.
-
Configure all 4 interfaces.
-
In Intrusion Prevention System, choose Provider “Emergingthreats.net Community Rules” and click ADD
-
In Intrusion Prevention System, click to enable RED, GREEN, BLUE, and ORANGE as “Monitored Interfaces”
-
In Intrusion Prevention System, click on “Enable Intrusion Prevention System” and then click SAVE
-
Verify that it now says “RUNNING”
-
On a seperate linux machine, connected to the RED interface, perform an “nmap -v address of RED INTERFACE” - this should show normal operation, all ports inaccessible.
-
Log in to the IPFIRE console as root.
-
Type “killall -9 /usr/bin/suricata” to simulate Suricata being killed
(alternately, you can try “swapoff -a && tail /dev/zero” to simulate out of memory, but I find this only sometimes results in Suricata being killed, and the bug seems to only appear when suricata actually dies) -
On a seperate linux machine, connected to the RED interface, perform an “nmap -v address of RED INTERFACE” - this should now show that all ports with running services are EXPOSED. This is the bug.
I hope this is helpful. Please let me know if you need any additional information.
Thanks
-Dan
