Pi3b not fully updating blocklist due to lack of memory?

pi3b isnot fully updating blocklist due to lack of memory? 600mb is free. Pi3B has 1GB ram, 32MB Micro sd April 20 update

Settings:

Month: Day:

IP Blocklist

 The following block lists were updated:
    BLOCKLIST_DE: 28 Time(s)
    BOGON_FULL: 4 Time(s)
    CIARMY: 14 Time(s)
    DSHIELD: 13 Time(s)
    SHODAN: 1 Time(s)
    TOR_ALL: 6 Time(s)
    TOR_EXIT: 6 Time(s)

 The following errors were detected:
    <ERROR> Could not update THREATVIEW_IO_IP blocklist - Download error! : 4 Time(s)
    <ERROR> Could not update EMERGING_COMPROMISED blocklist - Download error! : 42 Time(s)
    <ERROR> Could not update BLOCKLIST_DE blocklist - Download error! : 42 Time(s)
    <ERROR> Could not update BOGON_FULL blocklist - Download error! : 29 Time(s)
    <ERROR> Could not update TOR_EXIT blocklist - Download error! : 42 Time(s)
    <ERROR> Could not update DSHIELD blocklist - Download error! : 42 Time(s)
    <ERROR> Could not update BOGON blocklist - Download error! : 42 Time(s)
    <ERROR> Could not update SPAMHAUS_DROP blocklist - Download error! : 42 Time(s)
    <ERROR> Could not update CIARMY blocklist - Download error! : 42 Time(s)
    <ERROR> Could not update TOR_ALL blocklist - Download error! : 42 Time(s)
    <ERROR> Could not update FEODO_RECOMMENDED blocklist - Download error! : 42 Time(s)
    <ERROR> Could not update EMERGING_FWRULE blocklist - Download error! : 42 Time(s)


Kernel and Firewall:

 WARNING:  Out of memory killer killed these executables
    unbound :  1 Time(s)


1GB ram is the minimum you need. With that you will not be able to use IPS or DNS Firewall. Make sure both of those are disabled.

It might be that the IP Blocklist downloads any updated list to a temp location first and if the download is successful then it will replace the existing file so it might be that you need more ram than you think during the download and checking phase.
To confirm that I would need to check the code but i don’t have time to do that right now.
Will see if i can check that tomorrow morning.

also, may 1 update added this :

WARNING:  Kernel Errors Present
    Error: Driver 'efi-fra ...:  1 Time(s)
    brcmfmac mmc1:0001:1: Direct firmware load for brcm/brcmfmac43430-sdio.raspberrypi,3-model-b.bin failed with error -2 ...:  1 Time(s)

I checked the code and the update is done in memory.

Each blocklist is checked before downloading to see if it has been updated. If yes then the list is downloaded into a perl variable and checked that the download was successful before moving to the next step.

The next step is that the downloaded list is then parsed and the output placed into the ip set compatible file which is then updated in the firewall rules.

as the ip list is downloaded into a variable then it is in memory until it has been implemented into the firewall rules or ignored if the download was unsuccessful.

So the update will use some extra memory but I don’t know how much.

This is something that will have occurred during a reboot. The firmware loading had a failure.

If this has just happened once and does not repeat when doing a reboot again, then some hiccup happened during the boot that prevented the firmware from loading.

This post seems to address the issue and MAY solve it, but I won’t be trying it anytime soon. I am posting it in case someone has the same problem. Thanks for your help.

https://forums.raspberrypi.com/viewtopic.php?t=314208

It looks like I will have to upgrade. How much memory would you suggests to do dns server malware and some other stuff. Pi4 or something else? thanks

The developers recommend at least 1GB. If various built-in/add-on features are used, the amount of memory required will be much higher. Memory consuming features/add-ons are the web proxy and URL filter as well as the Intrusion Detection/Prevention System and the DNS Firewall. Using all of these could consume closer to 5GB or 6GB depending on the number of categories that are selected in each feature.
Another ( temporarily ) memory consuming function is the upgrading process. Download / decrypt are done in memory.
An IPFire where all the available functions are used would therefore be better with 8GB of memory.

I just tested it on my test Raspberry Pi 4 with 8GB of RAM:

  • Firewall enabled with a few rules
  • IP Address BlockLists
  • GeoIP Block
  • Enabled all DNS firewall lists
  • Enabled IPS with a few rules
  • Loaded the Toulouse URL filter list

The Raspberry Pi handled it with respectable bandwidth.



It seems that this driver error happens when there isn’t enough memory

A power outage scrooged the PI3B SSD. I had to reinstall and since then , the blocklist isn’t updating fully. So is !GB not really enough?

IP Blocklist

 The following block lists were updated:
    BLOCKLIST_DE: 57 Time(s)
    BOGON_FULL: 6 Time(s)
    CIARMY: 24 Time(s)
    DSHIELD: 23 Time(s)
    EMERGING_COMPROMISED: 1 Time(s)
    EMERGING_FWRULE: 1 Time(s)
    SHODAN: 1 Time(s)
    SPAMHAUS_DROP: 1 Time(s)
    THREATVIEW_IO_IP: 1 Time(s)
    TOR_ALL: 12 Time(s)

 The following errors were detected:
    <ERROR> Could not update DSHIELD blocklist - Download error! : 3 Time(s)
    <ERROR> Could not update TOR_ALL blocklist - Download error! : 4 Time(s)
    <ERROR> Could not update FEODO_RECOMMENDED blocklist - Download error! : 3 Time(s)
Disk space:

 Filesystem      Size  Used Avail Use% Mounted on
 /dev/mmcblk0p3   29G  2.2G   25G   8% /
 /dev/mmcblk0p1  496M  200M  297M  41% /boot
 /dev/mmcblk0p2   32M  162K   32M   1% /boot/efi


Status information

CRON Server RUNNING 1.90 MB RSS
D-Bus Daemon RUNNING 1.74 MB RSS
DHCP Server STOPPED
DNS Proxy Server RUNNING 41.72 MB RSS
Intrusion Prevention System STOPPED
Link Layer Discovery Protocol STOPPED
Logging Server RUNNING 1.68 MB RSS
NTP Server RUNNING 4.16 MB RSS
OpenVPN Roadwarrior Server STOPPED
Secure Shell Server STOPPED
VPN STOPPED
Web Proxy STOPPED
Web Server RUNNING 28.00 MB RSS

None of the errors listed are to do with your memory.
As it says the code had an error trying to download the list.
The code will try to download the list 5 times and if those 5 all failed then you will get that message.

Either there was a problem with internet access, or the route to the list server had an issue or there was some problem with the dns resolving of the fqdn for the list server.

FEODO_RECOMMENDED checks the status every 5 mins so 3 failures in a 24 hour period is not bade.

DSHIELD and TOR_ALL check every hour. So 3 and 4 failures respectively in 24 hours is not a bad result.

You can check when those lists were actually updated by looking in /var/ipfire/ipblocklist/modified. That file has the Epoch time for the last modification update.

You can convert that Epoch time into your normal time via a convertor such as

https://unixtime.org/

With the services you have got listed then you will probably be okay with that amount of memory. I have a system with 2GiB of memory and it is running at around 13% of memory without IPS or DNS Firewall enabled.
On that system if I turn on IPS then I know that after a while I will use all the remaining memory up and the system will start triggering the OOM (Out Of Memory) killer to kill some running programs. That was a message you showed in your first post. So with only 1GiB of memory don’t try and enable IPS and DNS Firewall.

1 GiB is the minimum you need to run the basic functions of IPFire. Enabling additional functions without increasing the memory on your system will increase the likelihood of using all the memory up and the system having to run the OOM which will likely stop something running correctly.

I had to use an eero for a while, which forced me to use ipfire DHCP.

I am assuming the log summary below for the DHCP server is the blocklist updates not being received? Thanks

DHCP Server:

 Unknown Entries:
    execute_statement argv[0] = /usr/sbin/unbound-dhcp-leases-client: 56 Time(s)
    execute_statement argv[1] = commit: 56 Time(s)
    execute_statement argv[2] = ADDRESS=192.168.1.158: 56 Time(s)
    execute_statement argv[3] = NAME=eero: 56 Time(s)


No, this are just the DHCP messages for the unbound-dhcp-leases-client. dhcpd signals (name, IP) relation to the DNS resolver unbound.
Blocklists are part of iptables or RPZ lists for the DNS firewall ( part of unbound ).