I created a test version of a RPZ add-on and I am looking for feedback

[hjkl (H&M)]

Final test for DOH.rpz list: using DNS over HTTPS (netblocks.org) to query for ipfire.org

RPZ metrics grew instantly:

rpz-metrics
name                 hits     active   lines    hits/line%  last update
---------------------------------
allow                0        enabled  0                 0   2024-08-01
block                0        enabled  0                 0   2024-08-01
BlockDOH_jpgpi250    17       enabled  3651              0   2024-08-04
SSLblAbuseCh         0        enabled  52                0   2024-08-04
URLHausAbuseCh       0        enabled  259               0   2024-08-04
                     =======           ========
Totals -->           17                3962

And here is the detail:

So DOH.rpz list does its job!

Tests completed

[himurae (himurae battosai)]

How can a noobie use this to load hagezi lists to block ads mainly

[jon]

I wrote this quick & dirty grep search to find blocked domains.

so doh.opendns.com is blocked, but opendns.com is not in the list. So the website is accessible.

I’ll add this to the list of rpz-scripts when I am done experimenting.

grep --color --recursive --line-number --fixed-strings --ignore-case "DOMAINtoSearch" '/etc/unbound/zonefiles/'

[root@ipfire ~] # grep --color --recursive --line-number --fixed-strings --ignore-case "opendns" '/etc/unbound/zonefiles/'
/etc/unbound/zonefiles/dohJPG.rpz:47:2.dnscrypt-cert.opendns.com CNAME .
/etc/unbound/zonefiles/dohJPG.rpz:1485:dns.opendns.com CNAME .
/etc/unbound/zonefiles/dohJPG.rpz:1900:doh.familyshield.opendns.com CNAME .
/etc/unbound/zonefiles/dohJPG.rpz:1969:doh.opendns.com CNAME .
/etc/unbound/zonefiles/dohJPG.rpz:1996:doh.sandbox.opendns.com CNAME .
/etc/unbound/zonefiles/dohJPG.rpz:2170:familyshield.opendns.com CNAME .
/etc/unbound/zonefiles/dohJPG.rpz:3028:resolver1-fs.opendns.com CNAME .
/etc/unbound/zonefiles/dohJPG.rpz:3030:resolver1.ipv6-sandbox.opendns.com CNAME .
/etc/unbound/zonefiles/dohJPG.rpz:3031:resolver1.opendns.com CNAME .
/etc/unbound/zonefiles/dohJPG.rpz:3034:resolver2-fs.opendns.com CNAME .
/etc/unbound/zonefiles/dohJPG.rpz:3036:resolver2.ipv6-sandbox.opendns.com CNAME .
/etc/unbound/zonefiles/dohJPG.rpz:3037:resolver2.opendns.com CNAME .
/etc/unbound/zonefiles/dohJPG.rpz:3111:sandbox.opendns.com CNAME .
[root@ipfire ~] # 

[jon]

@himurae

How can a noobie use this to load hagezi lists to block ads mainly

Since you are a noobie, I would suggest waiting until the add-on had been approved by IPFire Developers.

If you want to experiment, then there are instructions above in Post #52. But I would not recommend this today unless you are very comfortable with Linux and with IPFire.

[hjkl (H&M)]

@peppetech

I added the tmpfs line to /etc/fstab to prevent flash memory wear
it is in Chapter 8

May I ask how?

What is in chapter 8 does not work for ipfire…

#edit fstab - 10M tmpfs
tmpfs /etc/unbound/zonefiles tmpfs nodev,nosuid,gid=unbound,uid=unbound,mode=0755,size=10M 0 0

#then attempt to mount
mount /etc/unbound/zonefiles
mount: /etc/unbound/zonefiles: failed to parse mount options 'rw,nodev,nosuid,gid=unbound,uid=unbound,mode=0755,size=10M': Invalid argument.

Late edit: this works for ipfire

tmpfs  /etc/unbound/zonefiles tmpfs  nodev,nosuid,mode=0755,size=10M  0 0

[bbitsch (Bernhard Bitsch)]

To block popup ads just do
rpz-config add popup https://cdn.jsdelivr.net/gh/hagezi/dns-blocklists@latest/rpz/popupads.txt

For the composed lists, f.e. MutliLight, use
rpz-config add MultiLight https://cdn.jsdelivr.net/gh/hagezi/dns-blocklists@latest/rpz/light.txt

General:
the command is rpz-config add with

• can be chosen freely to distinguish the lists, will show up in the logs as ... unbound: [31113:0] info: rpz: applied [doh] ...
• can be found on the hagezi page tables , type RPZ.

EDIT: @himurae , this all depends on your definition of ‘noobie’. For really beginners to the firewalling/… topic, adher to @jon’s advice.

[bbitsch (Bernhard Bitsch)]

Further I think it isn’t a good idea to use RAM for storage of permanent files ( the contents may change ). Logs are also not stored in tmpfs anymore.
The effect can be seen at the graphs after a power loss. The data are stored in /var/log/rrd, which is located in tmpfs. To minimise the loss, periodically the ram disk is stored to HDD.

[hjkl (H&M)]

Acknowledged!
There is no power loss for this IPFIRE box - is on UPS and if UPS battery reach 30 min or 50% apcupsd initiate a clean shutdown…

[hjkl (H&M)]

Another test this time focusing on Huawei tracking(s)

RPZ list:

rpz-config add HagheziHuaweiNative https://raw.githubusercontent.com/hagezi/dns-blocklists/main/rpz/native.huawei.txt
unbound: info: rpz: add config file "HagheziHuaweiNative.rpz.conf"
unbound: rpz: running "unbound-control reload"
ok

Then powering up MyHuawei app from my Huawei phone and watching the rpz-metrics going up

 rpz-metrics
name                   hits     active   lines    hits/line%  last update
----------------------------------
allow                  0        enabled  0                 0   2024-08-01
block                  0        enabled  0                 0   2024-08-01
BlockDOH_jpgpi250      17       enabled  3651              0   2024-08-04
HagheziHuaweiNative    26       enabled  0                 0   2024-08-04
SSLblAbuseCh           0        enabled  52                0   2024-08-04
URLHausAbuseCh         0        enabled  276               0   2024-08-04
                       =======           ========
Totals -->             43                3979

This RPZ mechanism is really neat!
Thank you @jon !

[hjkl (H&M)]

I discovered an unwanted side effect of using tmpfs for storing /etc/unbound/zonefiles

rpz.tmpxxx files can’t be stored there but unbound does use those rpz lists (like huawei native!) so I had to stop using tmpfs for that!

log excerpt while /etc/unbound/zonefiles was stored in RAM:

Aug  4 21:38:52 black-x86-64 unbound: [630:0] info: generate keytag query _ta-4a5c-4f66. NULL IN
Aug  4 21:38:52 black-x86-64 unbound: [630:0] error: could not open /etc/unbound/zonefiles/HagheziXiaomiNative.rpz.tmp630: Permission denied
Aug  4 21:38:52 black-x86-64 unbound: [630:0] error: could not open /etc/unbound/zonefiles/HagheziDOH.rpz.tmp630: Permission denied
Aug  4 21:38:52 black-x86-64 unbound: [630:0] error: could not open /etc/unbound/zonefiles/HagheziHuaweiNative.rpz.tmp630: Permission denied
Aug  4 21:38:52 black-x86-64 unbound: [630:0] error: could not open /etc/unbound/zonefiles/HagheziLgTVWebOS.rpz.tmp630: Permission denied
Aug  4 21:38:53 black-x86-64 unbound: [630:0] error: could not open /etc/unbound/zonefiles/HagheziPopupAds.rpz.tmp630: Permission denied
Aug  4 21:38:55 black-x86-64 unbound: [630:0] error: could not open /etc/unbound/zonefiles/HagheziMulti.rpz.tmp630: Permission denied

[jon]

@hjkl

I discovered an unwanted side effect of using tmpfs for storing /etc/unbound/zonefiles

I do not use tmpfs but I have seen these errors before. unbound expects the RPZ files to be owned by nobody like this:

[root@ipfire ~] # ls -al /etc/unbound/zonefiles/
total 10788
drwxr-xr-x 2 nobody nobody     4096 Aug  4 14:26 .
drwxr-xr-x 4 root   root       4096 Aug  4 13:15 ..
-rw-r--r-- 1 root   root        815 Jul 14 13:42 allow.rpz
-rw-r--r-- 1 nobody nobody    26835 Aug  4 07:05 AmazonTrkrHZ.rpz
-rw-r--r-- 1 nobody nobody     7231 Aug  4 07:05 AppleTrkrHZ.rpz
. . .

so I do this in the rpz-config code to set up the RPZ files correctly:

		#	set-up zone file
		/usr/bin/touch "${rpzFile}"
		#	unbound requires these settings for rpz files
		/bin/chown nobody:nobody "${rpzFile}"
		/bin/chmod 644 "${rpzFile}"

This may be needed for the tmpfs area also.

[himurae (himurae battosai)]

Thanks a ton Love your energy!!! i have successfully accomplished adblocking feature in ipfire @jon thanks for the AWESOME program.

Program works good ,will it survive a reboot and how to delete/change a dnsbl list in case

[root@ipfire tmp]# rpz-metrics
name          hits     active   lines    hits/line%  last update
-------------------------------
allow         0        enabled  0                 0   2024-08-01
block         0        enabled  0                 0   2024-08-01
MultiLight    264      enabled  134985            0   2024-08-05
              =======           ========
Totals -->    264               134985

[jon]

@himurae

will it survive a reboot

yes. when the rpz-config add MultiLight ... script ran, it created a config file at /etc/unbound/local.d/MultiLight.rpz.conf.

And after a restart/reboot/power-off-on that config file will get reloaded and used.

does this help?

[peppetech]

Another suggestion for testing an RPZ list:

NSFW by OISD - this list blocks P—rn / Adult / Shock / Gore
I have been following Stephan’s subReddit for couple of years, before he switched to www.oisd.nl

License GNU General Public License v3.0
Allowed Update frequency - 1 hour

A word of Caution: THIS IS A LARGE LIST - 23MB -
URL:
https://nsfw.oisd.nl/rpz

[peppetech]

I’ve been experimenting some more, and looks like I managed to disable 2 of the zonefiles ")

# rpz-metrics
name         hits     active   lines    hits/line%  last update
-----------------------------------
allow        0        enabled  0                 0   2024-07-31
block        0        enabled  0                 0   2024-07-31
light        0        enabled  134985            0   2024-08-05
nsfw         0        enabled  839024            0   2024-08-05
urlhaus      0        enabled  286               0   2024-08-05
doh          0        disabled 3672              0   2024-08-05
threatfox    2        disabled n/a               0          n/a
             =======           ========
Totals -->   2                 977967

[jon]

@peppetech

nsfw         0        enabled  839024            0   2024-08-05
. . .
             =======           ========
Totals -->   2                 977967

keep in mind DNS will be noticeable slower with such big files.

• Large RPZ files will slow down the unbound reload time and slow down a DNS lookup. Over 500,000 lines of RPZ files (total lines for all RPZ files) is discouraged. Over 1,000,000 lines of RPZ files (total lines for all RPZ files) is NOT recommended.

from: https://www.ipfire.org/docs/addons/rpz#known-issues

[hjkl (H&M)]

Hi Jon,
I also have a custom script that collects some tens of thousands IP addresses in the blocklist.conf.
Sources for those are variate: from AD-block sources to IP Block lists.

Question: does RPZ accept sources that are not designed for RPZ?

Why I ask: the block list part attached to unbound does not have the metrics like RPZ so tracking down a false positive is quite hard and involves tcpdump monitoring for NXDOMAIN answers and then whitelisting that domain.
With RPZ I hope this part to be much easier with the help of metrics plus the good loging (a grep -i RPZ does provide good insights in real time)

Thanks for guidelines!

[jon]

@hjkl

Why I ask: the block list part attached to unbound does not have the metrics

I am not sure I understand what you are using. Is it this?

IP Address Blocklists

Or something else?

If it is IP Address Blocklists, then you can try:

tail -f /var/log/messages | grep -i --color BLKLST_

@hjkl

Question: does RPZ accept sources that are not designed for RPZ?

No.
I once found a “hosts to RPZ” type conversion script while Giggle searching, and I tried writing something similar, but once someone recommended the hagezi lists I set that effort aside. hagezi seems to have everything I needed!

[hjkl (H&M)]

@jon

I am not sure I understand what you are using. Is it this?

No,
Is a custom PiHole script sfeakes created many years ago

Here it is that PiHole (or whatever it is)

GitHub - sfeakes/ipfire-scripts: Scripts for ipfire

[jon]

Got it!

Depending on the list you picked from here:

https://github.com/sfeakes/ipfire-scripts?tab=readme-ov-file#below-are-a-list-of-the-sources-that-can-be-configured-turned-on-or-off-with--s-parameter

Some of those lists may have an RPZ list available on a different website.

Does anything appear in the message logs when an IP is blocked by the sfeakes/ipfire-scripts script?