IPS seems to drop legitimate blue to orange HTTPS traffic

Getting Started with IPFire Updates Trouble
1

After the update to 189 I had to disable IPS on Orange as it suddenly blocked a Orange host to contact a MySQL database in Green through the defined firewall pinhole as I reported here: (CU189 - IPS enable / disable on Interfaces changed - #4 by robinr1)
Meanwhile I found that IPS indeed actively blocks that traffic as Potentially Bad Traffic:

[Drop] [**] [1:2010937:3] ET SCAN Suspicious inbound to mySQL port 3306 [**] [Classification: Potentially Bad Traffic] [Priority: 2] {TCP} 192.168.2.2:58228 -> 192.168.0.9:3306

I’m not sure on how to prevent this from happening except disabling IPS completely on Orange. If anyone could guide me on this one, please do, I would like to have IPS enabled on Orange. You never know…

However, now after upgrading to 190, it became worse: suddenly Blue devices are no longer communicating over HTTPS with an Orange host despite a pinhole existing and no changes to the firewall. Connections now time out.
Again I have been searching long for the problem. When I restart the firewall (/etc/init.d/firewall restart) it works again, for some time. This can be for a few minutes up to half a day. And then it stops working again.
On a hunch, I have now tried disabling IPS on Blue, and suddenly it just works again, without explicit restarting of the firewall. So I did some digging in the IPS logging to check what rule would block Blue HTTPS to Orange. But I find no logging of it dropping those connections.
I have been live watching the IPS and firewall logging now, enable IPS on Blue, see HTTPS traffic timing out again, but nothing logged about it.
Disabling IPS on Blue again, and HTTPS traffic from Blue to Orange instantly works again.

I have this Orange HTTPS server also NATed over RED, and IPS is also active on RED ofcourse. But I have no problem contacting the Orange host from outside. The only difference between the RED rule and the Blue rule is that RED is using NAT and had SYN Flood protection ON.

Is anyone else experiencing similar problems or is able to simulate this problem in their end ?

2

Hi Robin,

Simple problem first. Disable the rule that is causing this.

In the Emerging threats rulesets there is one called

emerging-scan.rules

Inside that there is that rule that is enabled by default. Just un-check it.

Screenshot_2025-01-11_22-41-15
Screenshot_2025-01-11_22-41-15968×418 89.8 KB

I haven’t seen anything like that but I only use my Blue for guests or visitors, so I wouldn’t give them access to Orange anyway. My Green to Orange is working fine.

I find it very strange that you can fix the problem by disabling IPS on Blue but there are no log messages when the IPS block occurs.

It might be worth trying to disable that emerging-scan.rules ruleset completely as there are a variety of HTTP related rules in there.

At least if the problem still exists then you know it is not related to that ruleset.

The only other suggestion I have is to disable each selected ruleset, one by one and see if that helps or not.

If you identify a ruleset that is linked to the problem then you could look at the enabled rules in that set to see if any of them make any sense in terms of blocking your traffic.

Do you have any of the following rulesets enabled?

emerging-web_client.rules
emerging-web_server.rules
emerging-web_specific_apps.rules
1 Like
3

Thanks! That indeed did the trick. IPS is now enabled again on Orange.

I also tried disabling emerging-scan.rules as you suggested. But blue HTTPS traffic to orange started timing out again.

Next I tried disabling each selected ruleset, waited for suricata to stop using 100% of a CPU as it reloads the selected rulesets, test blue HTTPS traffic, disable the next ruleset and repeat.
Until no rulesets where selected anymore, then suricata stopped. And blue HTTPS worked again (as IPS is down).
Then I selected the first ruleset, waited, tested… and blue HTTPS traffic times out again. So it times out with only the last ruleset selected and it times out with only the first ruleset selected.

So it seems to stop working no matter which ruleset(s) suricata has loaded.
I’m starting to assume this is a bug? Should I open one in bugzilla ?

4

when I had my Amcrest doorbell cam on blue and the camera system’s NVR on orange, The only way I could get it to connect to the NVR was to reserve an address on blue and then set the firewall rules blue ip address to orange network (not firewall) and the other way around… I also blocked the camera from red too. Here is a screen shot of that. Currently, I moved that access point to the camera side of the NVR to get it off the network and repaired a router that I am using as the new access point in blue.

I also have orange blocked from the internet because I use it as a device only network for the printers, NVR, NAS, etc.

You’ll also notice I had to authorised green to orange and the reverse so I can contact the web servers in the devices on orange. I could print from green w/o the rules, but it would give me an error because the printer could not return ink level, and job finished. Nor I could log into the printer until I established those rules.

20241215_07h18m02s_grim
20241215_07h18m02s_grim1920×1080 172 KB

5

I have indeed firewall rules allowing BLUE → ORANGE specific host: HTTPS
The point is that it actually always has worked correctly and now suddenly no longer works. And when IPS is disabled on Blue, it works again.
The only recent change I have done is upgrading to CU 190.

Orange here is effectively DMZ and hosts in it have very restricted access to Green, alien hosts would have no access to Green or Blue.
The orange host serving a webserver on https is accessible from RED through NAT, but on the local network the host resolves to it’s internal orange IP so when a client is connected on Blue it should go straight to orange instead of going through RED (firewall) and then NATed to the orange host.

6

it should when the source and destination are ip host and color client networks like the example below. But if you use FQDN you must put that public web address in hosts and add that entry into the rules with a source to/from destination.

20241215_07h17m20s_grim
20241215_07h17m20s_grim1920×1080 140 KB

7

It might be a bug but if it was me, what I would do first would be to do a fresh install of CU189 and restore your backup and then confirm that the problem is not present. Then upgrade to CU190 and see if the problem is still present.

If yes, then I would say it was a bug in CU190 and that you can reproduce it with your setup.
If no then it was likely some corruption during the update, although what is difficult to tell.

Before doing the fresh install I would scp out the
/var/log/pakfire/update-core-upgrade-190.log
and all the
/var/log/messages
files that cover the period that you have been having the problem.
Also worth keeping a copy of the
/var/log/suricata/fast.log
files.

That way you have a copy of the files most likely to contain any key info.

8

I will also try and see if I can get some time to set up an HTTPS server on my vm testbed system, Orange network and set up a firewall rule to access it from my vm Blue Network and see if I can reproduce what you are experiencing.

9

By doing that you remove that rule from all interfaces. If you still want that rule to be used for all other interfaces and the problem you had is with a specific host on Orange that is the only one that needs to access that MySQL database on Green, you could always whitelist that Orange Host in the IPS whitelist section.

That way the Orange host can access the MySQL server but all other hosts on all other enabled interfaces will still be IPS scanned with that rule.

10

I understand, but then that host is completely excluded from IPS scanning, I presume ? And since that host is a webserver accessible from RED, it is probably one of the highest risks on my network so I would like to have it as secure as possible. I think disabling only the mySQL rule for all hosts then is more secure over disabling IPS completely for that host alone.

Meanwhile, now that IPS is enabled again on Orange, I found that blue to Orange HTTPS is not the only thing no longer working:
The host in Orange also serves as webmail client, so it has IMAP(143) pinholes to my mailserver in Green. And now that I enabled IPS again on Orange, it is no longer able to communicate with IMAP in Green. Again without logging anything in messages or fast.log…
I tried enabling IPS on Blue again, and also there, when IPS is enabled, any mail client in Blue fails to connect to IMAP on Green.

But here it gets even weirder as then I tried again disabling ruleset after ruleset each time waiting for suricata to reload the rules and then testing again, and after disabling ThreatFox, Blue devices suddenly where able again to access IMAP on green. Orange however still couldn’t.
I then enabled all rules again except ThreatFox and it kept on working in blue but not orange. Finally I re-enabled ThreatFox… and it kept on working in blue but not orange… :face_with_raised_eyebrow:
Blue HTTPS to orange still did not work in any of the tried scenarios with IPS enabled on Blue.
For now I have not been able to simulate the problem on blue → green (IMAP) again anymore. That just went away by disabling rules one by one and then re-enabling them…

Another strange thing: in an attempt to understand what is going on with IMAP traffic, I enabled firewall logging for all IMAP rules I have. But the only things logged by the firewall in messages about port 143 is when it is accessed through RED using DNAT. I see no FORWARDFW logging when IMAP on green is accessed from Blue or Orange (with IPS disabled on both) despite logging is enabled for all those firewall rules (and firewall was restarted)

Reinstallation is not that easy, as it is an IPFire Mini appliance and there is never a good moment to put the household without internet. I’ve already irritated the wife and children by having the above problems. (Over time I tried to de-Google as much as I could in our household, so our mail, contacts, calendar, phone tracking, ‘cloud’-photo albums, shared file-storage, etc is all handled by this orange host… making all members heavily dependent on this server instead of Google… But I have to watch my steps to keep them committed to it and not try to shadow-IT back to Google behind my back, compromising the privacy-friendly environment I try to have here.)
So I’m not sure I can do this any time soon, so for now I have indeed whitelisted my orange host like you suggested, which currently seems to solve all problems. But I’m a bit concerned about what if that host gets compromised. (granted, it has already pinholes to green to access my most critical infrastructure and data, so maybe IPS won’t be able to protect me much in such a case)

11

That is true. Better some scanning than none, especially when a server is also accessed from RED.

Okay, I can understand that. I only have myself so I only irritate myself if I cause a problem or disruption :smiley:

I have got a web server set up on one of my vm hosts running on my development vm network.

I then set up a firewall rule to allow a specific blue host to be able to access the web server host on orange via https protocol only.

It has now been running for a couple of hours and the blue host can still access the web server on orange. So currently I have not been able to duplicate the issue you are experiencing.

The IPS is running with the Emergingthreats.net Community Rules and the Abuse.ch SSLBL Blacklist Rules providers.

I will leave it running for some time and see if I see any problems after a longer time.

12

I have all 4 enabled. Not that it seems to be much of a difference.
I will try a reboot soon and check if that changes anything in the behavior.

13

One would like to know what the rule really does.

Because it might be a rule designed to block all traffic on the port if it is not using encryption on port 3306

14

So my vm network has been running for 18.5 hours now and the blue host can still access the web server host in the orange network.

So it doesn’t look like a systemic type of issue. There must be some interaction type effects between different packages or there must be some corruption somewhere.

What is also strange is that you indicate that after restarting the firewall it works fine but then after some time has passed ( can be between minutes and hours) it stops again.
I can’t eaisly think of a bug that would cause that sort of issue and also not show anything at all in the logs.

When the blue host can not access the web server host in orange, can a host in the green network still access things in the orange zone?

If you have two hosts in blue set up to access the web server host in orange do they both suffer the same problem at the same time?

15

problem is since they are using older versions, they could be running into the bugs that popped up in Kernels 6.6.xx. I’m still waiting on news of ipfire adopting 6.13rc since it was added to main-stable last week.

16

When it becomes a release version and not a release candidate (rc).

Currently it is on rc7 and if no major issues are found it is being said that the final version of 6.13 will get released in around a week or so’s time.

1 Like
17

A Linux rc is just like the testing tree of IPFire.
Not recommended for all use cases.

18

excelllent. I will by happy that will be release so I can go back to the stable branch. The machine that is running ipfire 192 with kernel 6.12 runs great but my motherboard is failing. Its now a four core machine and my 3V on board has a lot of noise on it. So this afternoon, I’ll be setting up that 10Gb supermicro server in its place.

I’m still investigating a real bug in the system, that was caused by someone modifying fontconfig for X11. This has been broken for a long time and some who went to systemd use a different system while others like redhat and ubuntu made their own solution. Its still broke in Alpine and other init desktop distros. But since we don’t use X11, we should be able to use the standard version of fontconfig from freedesk.org. The solution points to installing this in the master copy to replace the broken one that exist.

19

Yes, all hosts in blue suffer the problem at the same time.

Yes Green (where IPS has always been disabled) never has a problem accessing Orange.
The Orange host also is still accessible from RED (DNAT) (where IPS has always been enabled) without problems.

Indeed… well… the problem in this observation is probably that I depended on feedback from other household members, as normally I always have an openVPN session running on my android phone, hence I never experienced the problems myself. And whenever someone complained, I checked to see that it indeed is no longer working and I restarted the firewall and it all started working again… until someone complained again.
Meanwhile I found out that Suricata easily takes 10 - 15 minutes to load al rules (IPFire Mini (rev1)) when restarted or when it is enabled on an additional interface.
I’m not sure how suricata drops suspicious activity, as I don’t see much of it in iptables; but I assume it works closely with the firewall, so reloading the firewall may cause some time to pass for suricata to effectively start dropping again ?
I will do some more tests with this soon. Currently suricata is disabled on blue and orange for keeping the peace in the house. I will have to try this at a moment when no-one is home, or everyone is asleep :slight_smile:

20

Should be In logs IPS logs