Memory usage climbing

I am not very concerned about this but considering I have 16GB of RAM and having read that should be more than enough for IPFire, I am not really seeing that it is “more than enough”.

I should be able to up the RAM to 32GB ( this is the motherboard ) but I am reluctant to spend money on this Atom C2000 board due to well known issues .

Any obvious culprits I could look in to for unusual memory usage?

I will be replacing this server eventually, just not sure how to prioritize it.

Link to system: www.ipfire.org - Profile 28344dbcc071d0274eb162cb7edc576d929ec225

Edit: I do have a potential culprit though:

IPS/IDS is a powerhog that scale sometimes exponentially with connection speed and number of published service. A big amount of security budget is spent on optimized rules for a “legally safe” set of rules which deliver the most protection/legally defending/cpu power efficient set of rules.
Your really nice embedded motherboard is a wonderful piece of hardware… 8 years old.
And more kids are around the low power block
https://www.cpubenchmark.net/singleCompare.php?remove=3144

Tweaking and optimizing rules, on a pure results/time ratio might be an announced fail, however might help reduce useless memory overhead and improve your knowledge.
If you don’t have time, maybe is worth design an upgrade.

Well, the fun fact is that I have not tweaked or optimized anything related to IDS.

Added a few rule sets, that is all.

image

I would add more if freely available

I just enabled the same rulesets you have (originally I just had Abuse.ch and Emergingthreats.net.

My memory did not change with the additional rules sets. I am running at around 580MB whereas you seem to be running at 9.8GB!!

I only have 4GB total memory on my system and I am running consistently at around 22% memory used.

Might be worth disabling your IPS and then enabling it again and see if the memory level changes or not.
Nearly 10GB for IPS looks like something is definitely wrong.

Have you just added your rulesets in or did you then go and turn on every rule in each of the rulesets because that would use a lot of memory and also would be likely to give you potential false positive problems as sometimes the rules may block stuff you need to use.

As an additional item, note that the PT Attack ruleset has ben made a Read Only ruleset on its github site since Sep 2022 and therefore is no longer getting any updates. Last significant updates were in 2021 so the rule is around 5 years out of date.

This status was updated in the wiki in Dec 2023 and the PT Attack ruleset will be removed in CU185, together with some other no longer existing rulesets.

The OISF Traffic just identifies the type of traffic you are having on your network. So it will flag up for you if you have Twitter traffic, Facebook traffic, WhatsApp traffic etc.
That list was last updated in Nov 2022. The items already on the list will mostly still be current, although Twitter is now X, however all the new traffic types that have originated since end 2022 will not be flagged, such as Mastodon, etc.

EDIT:

I just disabled my IPS with all of the same rulesets as you have defined and then re-nabled it again (pressing Save in between).

My IPS memory has gone down from 508MB to 370MB.

Definitely worth trying the disable/enable option.

1 Like

Will try that when back home, Adolf.

I wonder, is Location Block part of the IPS or the Firewall?

Because I would expect that to consume a bit of memory since I have everything blocked.

It is not part of the IPS. Location Block is comparatively a very simple task so it does not consume a lot of memory or cpu.

The IPS has to check the details of the actual traffic against each of the rules that have been selected and this consumes memory and cpu. Hence why it is better to use things like the IP Blocklist for certain lists that are also available on the IPS. The IPS will consume much more resource.

For example the Emerging Threats rules set has one entry named
emerging-dshield.rules

It is better to not have that selected in the IPS but to have it selected in the IP Address Blocklists
Dshield.org Recommended Block List

as the IP Address Blocklist will, consume much less resource to block those IP’s than the IPS would need to use.

Disabling and re enabling the IPS made a huge difference in memory usage. Will check that again in a few hours.

To be honest I did not really pay much attention to IP Address Blocklists but now I enabled all recommended. Can’t believe that oversight.

Also read up a bit here for some caveats.

1 Like

Memory is better and keeping better.

I have a question though…

What time span is reflected in the listed Memory numbers in Services?

There is no way to alter the period for that that I can see.

How is your memory usage after restarting IPS and waiting about a day?

I noticed as well that Suricata rules, use a lot of RAM.

although in your case having 16 GB is plenty

I would remove at least 3 of your rules
OISF and PT have been EOL, for a while and SNORT might be a duplicate of ET

On my IPFIre, I added Threatfox and URLHaus rules, Threatfox is a real memory hog as mentioned before.

If you are interested in IP Blocklists, or maybe RPZ blocklists, I am listing a bunch here,

Avoid the DoH list, :frowning: and let me know if you need more help.

1 Like

It is a snapshot of the memory. Also on the IPS page where it also shows memory, it shows a snapshot of the memory when the page is refreshed.

1 Like

Well, one would appreciate a notification if some of those lists are outdated and not maintained.

I do understand that is perhaps not anything IPFire can keep track of, since there are many lists, but still a bit of a nuisance because I really really want to keep my network as safe as possible.

After all, IPFire do provide us with this selection.

image

Should not some be removed as per your statement?

A user on this forum flagged up that the PT Attack ruleset was no longer being updated. Based on that I had a search through the various lists and found that the three Secureworks subscription rulesets no longer existed.

The OISF ruleset looks like it is not being actively updated but as it just flags types of traffic on your network, virtually all of those named traffic types such as Facebook, Instagram, WhatsApp, etc are still valid. You just won’t get any new types of traffic added such as Mastodon or Twitter changing to X etc.
The OISF ruleset does not block anything. It is there so that you can look at your IPS logs and see if any traffic types that you don’t want to have on your network are being used and by which client. Then you can decide what other action you want to take.

Based on the above ruleset status that I found, I updated the IPS Ruleset wiki page in Dec 2023.
https://www.ipfire.org/docs/configuration/firewall/ips/rulesets

What will become CU185 already has the PT Attack and the three Secureworks subscription rulesets removed from it.

When CU185 goes to Testing I will then update the IPS Rulesets wiki page again with further information.

Make sure that you are not just blindly selecting every ruleset provider you can and then selecting all rules.

You need to research the rulesets to see what they are intended to do and whether their intention is aligned with what you need.
If you are using Ubuntu or Debian with apt-get update systems without a centralised update server then you don’t want to turn on the rules that block all apt-get traffic. That is especially true if you activate any of the rules that are not enabled by the provider by default. Those may result in unexpected false positives depending on your network.

There is a page in the wiki that describes how to approach the ruleset selection.
https://www.ipfire.org/docs/configuration/firewall/ips/rule-selection

5 Likes

I might have done just that. :stuck_out_tongue:

To me those are just padlocks that I close and throw away the key until I notice something is not working.