I created a test version of a RPZ add-on and I am looking for feedback

roberto · 20 October 2024 12:34

Do you know piHole? Well, the same, saving appearances.

Please correct me if I’m confused.

Regards.

jon · 20 October 2024 14:43

Basically RPZ blocks domain names.

Example: I have a RPZ list for blocking ads and tracking. When I go to a website with ads/tracking, RPZ will block those ads from appearing in my web browser.

Does this help?

Real world example:
I added this RPZ list to the RPZ add-on:
PopUpAdsHZ https://raw.githubusercontent.com/hagezi/dns-blocklists/main/rpz/popupads.txt

And just for testing I disabled all of my other RPZ lists.

I went to cnn.com on my web browser.

and these items were BLOCKED by RPZ:

Oct 20 09:58:07 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.adnxs.com. rpz-nxdomain 192.168.60.196@62730 ib.adnxs.com. HTTPS IN
Oct 20 09:58:07 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.adnxs.com. rpz-nxdomain 192.168.60.196@57201 ib.adnxs.com. A IN
Oct 20 09:58:07 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.outbrain.com. rpz-nxdomain 192.168.60.196@63175 widgets.outbrain.com. HTTPS IN
Oct 20 09:58:07 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.outbrain.com. rpz-nxdomain 192.168.60.196@60622 widgets.outbrain.com. A IN
Oct 20 09:58:07 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.outbrain.com. rpz-nxdomain 192.168.60.196@62232 odb.outbrain.com. HTTPS IN
Oct 20 09:58:07 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.outbrain.com. rpz-nxdomain 192.168.60.196@55236 odb.outbrain.com. A IN
Oct 20 09:58:07 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.adsafeprotected.com. rpz-nxdomain 192.168.60.196@60421 cdn.adsafeprotected.com. HTTPS IN
Oct 20 09:58:07 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.adsafeprotected.com. rpz-nxdomain 192.168.60.196@52111 cdn.adsafeprotected.com. A IN
Oct 20 09:58:07 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.proximic.com. rpz-nxdomain 192.168.60.196@52986 linode-api.us-east.proximic.com. A IN
Oct 20 09:58:07 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.proximic.com. rpz-nxdomain 192.168.60.196@58042 linode-api.us-east.proximic.com. A IN
Oct 20 09:58:07 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.proximic.com. rpz-nxdomain 192.168.60.196@50943 linode-api.us-east.proximic.com. HTTPS IN
Oct 20 09:58:07 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.proximic.com. rpz-nxdomain 192.168.60.196@56030 linode-api.us-east.proximic.com. HTTPS IN
Oct 20 09:58:08 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.ml314.com. rpz-nxdomain 192.168.60.196@49501 cdn.ml314.com. HTTPS IN
Oct 20 09:58:08 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.ml314.com. rpz-nxdomain 192.168.60.196@59568 cdn.ml314.com. A IN
Oct 20 09:58:08 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.ml314.com. rpz-nxdomain 192.168.60.196@50236 vi.ml314.com. HTTPS IN
Oct 20 09:58:08 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.ml314.com. rpz-nxdomain 192.168.60.196@64268 vi.ml314.com. A IN
Oct 20 09:58:08 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.stickyadstv.com. rpz-nxdomain 192.168.60.196@58768 ads.stickyadstv.com. HTTPS IN
Oct 20 09:58:08 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.stickyadstv.com. rpz-nxdomain 192.168.60.196@60372 ads.stickyadstv.com. A IN
Oct 20 09:58:08 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.pubmatic.com. rpz-nxdomain 192.168.60.196@53939 ads.pubmatic.com. HTTPS IN
Oct 20 09:58:08 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.pubmatic.com. rpz-nxdomain 192.168.60.196@62920 ads.pubmatic.com. A IN
Oct 20 09:58:08 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.postrelease.com. rpz-nxdomain 192.168.60.196@58568 jadserve.postrelease.com. HTTPS IN
Oct 20 09:58:08 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.postrelease.com. rpz-nxdomain 192.168.60.196@51917 jadserve.postrelease.com. A IN
Oct 20 09:58:08 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.clean.gg. rpz-nxdomain 192.168.60.196@53675 i.clean.gg. HTTPS IN
Oct 20 09:58:08 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.clean.gg. rpz-nxdomain 192.168.60.196@52693 i.clean.gg. A IN
Oct 20 09:58:08 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.rubiconproject.com. rpz-nxdomain 192.168.60.196@54451 fastlane.rubiconproject.com. HTTPS IN
Oct 20 09:58:08 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.rubiconproject.com. rpz-nxdomain 192.168.60.196@52859 fastlane.rubiconproject.com. A IN
Oct 20 09:58:08 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.casalemedia.com. rpz-nxdomain 192.168.60.196@49300 htlb.casalemedia.com. HTTPS IN
Oct 20 09:58:08 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.casalemedia.com. rpz-nxdomain 192.168.60.196@58074 htlb.casalemedia.com. A IN
Oct 20 09:58:08 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.pubmatic.com. rpz-nxdomain 192.168.60.196@54787 hbopenbid.pubmatic.com. HTTPS IN
Oct 20 09:58:08 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.pubmatic.com. rpz-nxdomain 192.168.60.196@60599 hbopenbid.pubmatic.com. A IN
Oct 20 09:58:08 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] prebid.media.net. rpz-nxdomain 192.168.60.196@63711 prebid.media.net. HTTPS IN
Oct 20 09:58:08 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] prebid.media.net. rpz-nxdomain 192.168.60.196@50893 prebid.media.net. A IN
Oct 20 09:58:08 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.3lift.com. rpz-nxdomain 192.168.60.196@52665 tlx.3lift.com. HTTPS IN
Oct 20 09:58:08 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.3lift.com. rpz-nxdomain 192.168.60.196@56733 tlx.3lift.com. A IN
Oct 20 09:58:08 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] ad-delivery.net. rpz-nxdomain 192.168.60.196@64407 ad-delivery.net. HTTPS IN
Oct 20 09:58:08 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] ad-delivery.net. rpz-nxdomain 192.168.60.196@57252 ad-delivery.net. A IN
Oct 20 09:58:09 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.rubiconproject.com. rpz-nxdomain 192.168.60.196@60285 eus.rubiconproject.com. HTTPS IN
Oct 20 09:58:09 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.rubiconproject.com. rpz-nxdomain 192.168.60.196@58737 eus.rubiconproject.com. A IN
Oct 20 09:58:09 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.pubmatic.com. rpz-nxdomain 192.168.60.196@52002 image8.pubmatic.com. HTTPS IN
Oct 20 09:58:09 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.pubmatic.com. rpz-nxdomain 192.168.60.196@63738 image8.pubmatic.com. A IN
Oct 20 09:58:09 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.rubiconproject.com. rpz-nxdomain 192.168.60.196@50650 pixel-us-east.rubiconproject.com. HTTPS IN
Oct 20 09:58:09 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.rubiconproject.com. rpz-nxdomain 192.168.60.196@49701 pixel-us-east.rubiconproject.com. A IN
Oct 20 09:58:09 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.tremorhub.com. rpz-nxdomain 192.168.60.196@49508 eq97f.publishers.tremorhub.com. HTTPS IN
Oct 20 09:58:09 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.tremorhub.com. rpz-nxdomain 192.168.60.196@53720 eq97f.publishers.tremorhub.com. A IN
Oct 20 09:58:09 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.casalemedia.com. rpz-nxdomain 192.168.60.196@57321 dsum-sec.casalemedia.com. HTTPS IN
Oct 20 09:58:09 ipfire unbound: [2129:0] info: rpz: applied [PopUpAdsHZ] *.casalemedia.com. rpz-nxdomain 192.168.60.196@52953 dsum-sec.casalemedia.com. A IN

This one RPZ blocks many items, but it did not block everything.

jon · 20 October 2024 15:56

starting a build with new language file!

Thank you!!!

markadewet · 20 October 2024 19:10

@roberto No, nerver heard of PiHole until I started with IPFire. To be honest I am still learning a lot.

bbitsch · 20 October 2024 20:54

To filter web access, blocking unwanted destinations, there are three ways, in principle ( IPFire modules):

analyse the traffic going through the proxy (URLFilter)
filtering in the firewall (IPblocklists)
filtering the name resolution ( Jon’s addon)

Topic 1.'s flaws: you must assure, that all web traffic is going through the proxy; HTTPS traffic can’t be analysed ( it is encrypted ); onla web traffic can be handled
Topic 2.'s flaws: the IPs of ‘suspect’ desinations tend to change very often, the update effort is not low.
Topic 3. uses the usual process of internet communication to host mydestination.domain. First resolve the server name to an IP using DNS, then communicate with this IP. If the name resolution is blocked in the DNS ‘server’, the client can’t communicate to this end point. Flaws: DNS resolution is slower; the collection of possible lists isn’t researched enough,yet

Hope I could clarify a bit the mystery of RPZ.

markadewet · 21 October 2024 06:43

@bbitsch Yes, thank you, your explanation does make it a bit easier to understand, thank you.

jon · 21 October 2024 15:24

Here is the latest!

rpz-beta-0.1.12-12 on 2024-10-21
rpz.cgi:

feature: added new language file for French (thank you gw-ipfire)

rpz-beta-0.1.12-12.ipfire.tar (40 KB)

@gw-ipfire - Can you test this? I had trouble when building the rpz.fr.pl file.

I changed each ' to \' . Example:

'rpz exitcode 105' => 'L\'URL n\'est pas valide"',

Hopefully this is correct for French!

bbitsch · 21 October 2024 15:52

Yes, this is correct, do didn’t change french.
But you must escape characters with a meaning in the language, here Perl.
’ starts or ends a string.
Looking at the source of the translation file, I found some extranous ", f.e. in your example. They can be deleted, I think.
rpz exitcode 105 should read L'URL n'est pas valide

EDIT: looking at the original rpz.en.pl, I found these characters also.

jon · 21 October 2024 16:07

nice find! Thank you!

gw-ipfire · 21 October 2024 18:42

hi

Hopefully this is correct for French!

yes is correct for French!
ty

gw-ipfire · 21 October 2024 18:51

in rpz.cgi:
please add an indication for the Title to download following the title
list to download
ty

jon · 21 October 2024 19:25

A post was merged into an existing topic: Rpz in whitelist mode

jon · 21 October 2024 19:33

I am not sure I understand what you are looking for.

Are you looking for a visual indicator (icon?) that displays when the RPZ is downloaded? Or something else?

FYI - The RPZ.cgi (or the RPS scripts) do not download anything. All of the downloading and background timers are all controlled by the unbound code.

gw-ipfire · 22 October 2024 09:21

hi
sorry for my unclear request, I will show you with an image

to indicate that the lites are downloaded from the web
ty

bbitsch · 22 October 2024 09:35

I don’t think, this necessary.
If a user configures this module, he should have read the wiki. There is described the main mechanism of the RPZ functionality - RPZ lists from the internet and local adjustments with white and black lists.

tphz · 29 October 2024 17:48

After adding the list https://hole.cert.pl/domains/v2/domains_rpz.db from https://cert.pl/en/warning-list/
an error message is visible in the logs.

unbound: [3699:0] error: rpz: name of record (test-rpz.hole.cert.pl.hole.cert.pl.) to insert into RPZ is not a subdomain of the configured name of the RPZ zone (certpl.rpz.)

Below is the beginning of the contents of the domains_rpz.db file

; RPZ file from hole.cert.pl
$TTL 300 ; default TTL
$ORIGIN hole.cert.pl.
@  IN SOA  hole.cert.pl. hostmaster.hole.cert.pl. (
           1730213786 ; serial
           300      ; refresh [5m]
           60       ; retry [1m]
           604800   ; expire [7d]
           300      ; minimum TTL [5m]
           )
@  IN NS   localhost.
test-rpz.hole.cert.pl CNAME hole.cert.pl. ; Test entry for testing hole.cert.pl RPZ
0-1x.8632152.xyz CNAME hole.cert.pl.
...

Is the problem in the domains_rpz.db file or in the IPFire rpz configuration?

tphz · 29 October 2024 18:09

Other problem after adding https://threatfox.abuse.ch/downloads/threatfox.rpz list

When you click save, it disappears/cuts the .rpz extension from the threatfox.rpz file.

Below logs after clicking save then apply.

|19:11:48|unbound: [1946:0]|info: generate keytag query _ta-4a5c-4f66. NULL IN|
|19:11:48|unbound: [1946:0]|info: start of service (unbound 1.21.0).|
|19:11:48|unbound: [1946:0]|notice: init module 2: iterator|
|19:11:48|unbound: [1946:0]|notice: init module 1: validator|
|19:11:48|unbound: [1946:0]|notice: init module 0: respip|
|19:11:48|unbound: [1946:0]|notice: Restart of unbound 1.21.0.|
|19:11:48|unbound: [1946:0]|info: server stats for thread 0: requestlist max 0 avg 0 exceeded 0 jostled 0|
|19:11:48|unbound: [1946:0]|info: server stats for thread 0: 0 queries, 0 answers from cache, 0 recursions, 0 prefetch, 0 rejected by ip ratelimiting|
|19:11:48|unbound: [1946:0]|info: service stopped (unbound 1.21.0).|
|19:06:25|unbound: [1946:0]|info: start of service (unbound 1.21.0).|
|19:06:25|unbound: [1946:0]|notice: init module 2: iterator|
|19:06:25|unbound: [1946:0]|notice: init module 1: validator|
|19:06:25|unbound: [1946:0]|notice: init module 0: respip|
|19:06:25|unbound: [1946:0]|notice: Restart of unbound 1.21.0.|
|19:06:25|unbound: [1946:0]|info: 0.262144 0.524288 2|
|19:06:25|unbound: [1946:0]|info: 0.131072 0.262144 3|
|19:06:25|unbound: [1946:0]|info: 0.065536 0.131072 1|
|19:06:25|unbound: [1946:0]|info: lower(secs) upper(secs) recursions|
|19:06:25|unbound: [1946:0]|info: [25%]=0.152917 median[50%]=0.218453 [75%]=0.32768|
|19:06:25|unbound: [1946:0]|info: histogram of recursion processing times|
|19:06:25|unbound: [1946:0]|info: average recursion processing time 0.248840 sec|
|19:06:25|unbound: [1946:0]|info: server stats for thread 0: requestlist max 0 avg 0 exceeded 0 jostled 0|
|19:06:25|unbound: [1946:0]|info: server stats for thread 0: 9 queries, 3 answers from cache, 6 recursions, 0 prefetch, 0 rejected by ip ratelimiting|
|19:06:25|unbound: [1946:0]|info: service stopped (unbound 1.21.0).|
|18:58:11|unbound: [1946:0]|info: generate keytag query _ta-4a5c-4f66. NULL IN|
|18:58:11|unbound: [1946:0]|info: start of service (unbound 1.21.0).|
|18:58:11|unbound: [1946:0]|notice: init module 2: iterator|
|18:58:11|unbound: [1946:0]|notice: init module 1: validator|
|18:58:11|unbound: [1946:0]|notice: init module 0: respip|
|18:58:11|unbound: [1946:0]|notice: Restart of unbound 1.21.0.|
|18:58:11|unbound: [1946:0]|info: 0.262144 0.524288 4|
|18:58:11|unbound: [1946:0]|info: 0.131072 0.262144 5|
|18:58:11|unbound: [1946:0]|info: 0.065536 0.131072 3|
|18:58:11|unbound: [1946:0]|info: lower(secs) upper(secs) recursions|
|18:58:11|unbound: [1946:0]|info: [25%]=0.131072 median[50%]=0.209715 [75%]=0.32768|
|18:58:11|unbound: [1946:0]|info: histogram of recursion processing times|
|18:58:11|unbound: [1946:0]|info: average recursion processing time 0.229358 sec|
|18:58:11|unbound: [1946:0]|info: server stats for thread 0: requestlist max 0 avg 0 exceeded 0 jostled 0|
|18:58:11|unbound: [1946:0]|info: server stats for thread 0: 20 queries, 8 answers from cache, 12 recursion s, 0 prefetch, 0 rejected by ip ratelimiting|
|18:58:11|unbound: [1946:0]|info: service stopped (unbound 1.21.0).|

edit

After disable list

When you click the pencil to edit, you can add the file extension .rpz then it does not disappear when you click the save button.

However, the extension .rpz disappears after enabling the list.

bbitsch · 29 October 2024 19:42

This is a problem already found. The WUI gets the active RPZ list from rpz-config list. There is problem in the output of this command. You can try to correct this problem by

awk -F':' '/^\s*name:/{ gsub(/[[:blank:]]|\.rpz/, "",$2) ; NAME=$2 } \
/^\s*url:/{ gsub(/[[:blank:]]/, "") ; print NAME"="$2":"$3} '  \
/etc/unbound/local.d/*rpz.conf

in lines 104-06 in /usr/sbin/rpz-config ( the list command ).

I think @jon 's next release will contain this patch.

tphz · 29 October 2024 20:02

Thank you for your quick response.
I’m afraid that at my place it is 103-105

jon · 29 October 2024 20:02

Bernhard found this also! Thank you!!

Yes, this is in the next release. It is building right now.

I do not believe this causes any errors. It seems to be a display only issue.
if you look at the threatfox.rpz.conf file then all should be OK.

Can you send a screenshot of what was entered on the RPZ.cgi webgui?