URL Filter and self updating blacklists?

mutley · 13 January 2022 19:24

I guess one other thing is to check your unbound config file against ipFire distribution, make sure there are no changes.
So your file /etc/unbound/unbound.conf against this one.

github.com

ipfire/ipfire-2.x/blob/master/config/unbound/unbound.conf

#
# Unbound configuration file for IPFire
#
# The full documentation is available at:
# https://nlnetlabs.nl/documentation/unbound/unbound.conf/
#

server:
	# Common Server Options
	chroot: ""
	directory: "/etc/unbound"
	username: "nobody"
	do-ip6: no

	# System Tuning
	include: "/etc/unbound/tuning.conf"

	# Logging Options
	use-syslog: yes
	log-time-ascii: yes

This file has been truncated. show original

cbrown · 13 January 2022 19:56

The ‘/etc/unbound/local.d/blocklist.conf’ file is still intact with 143282 entries and looks normal and zeusclicks.com is in the file.
diffed my /etc/unbound/unbound.conf and found no diffs
stand by for more …

Thanks for the help

cbrown · 13 January 2022 20:10

Okay ran again. Started out working fine for the first three minutes. then started getting the ip address resolved by dns as previously reported. Still waiting for 'Starting Unbound DNS Proxy … ’ to finish (as previously reported – this takes a really really long time), Oh well it just completed with
‘./dns_blocklist.sh: Blocked Hosts Update, 143274 hosts blocked’
Can’t say I see anything different in the log grep but here it is …

Jan 13 13:58:55 ipfire unbound: [29681:0] info: service stopped (unbound 1.13.2).
Jan 13 13:58:55 ipfire unbound: [29681:0] info: server stats for thread 0: 3499 queries, 1211 answers from cache, 2288 recursions, 46 prefetch, 0 rejected by ip ratelimiting
Jan 13 13:58:55 ipfire unbound: [29681:0] info: server stats for thread 0: requestlist max 20 avg 1.25321 exceeded 0 jostled 0
Jan 13 13:58:55 ipfire unbound: [29681:0] info: average recursion processing time 0.422614 sec
Jan 13 13:58:55 ipfire unbound: [29681:0] info: histogram of recursion processing times
Jan 13 13:58:55 ipfire unbound: [29681:0] info: [25%]=0.156622 median[50%]=0.344193 [75%]=0.688251
Jan 13 13:58:55 ipfire unbound: [29681:0] info: lower(secs) upper(secs) recursions
Jan 13 13:58:55 ipfire unbound: [29681:0] info: 0.000000 0.000001 226
Jan 13 13:58:55 ipfire unbound: [29681:0] info: 0.001024 0.002048 4
Jan 13 13:58:55 ipfire unbound: [29681:0] info: 0.002048 0.004096 2
Jan 13 13:58:55 ipfire unbound: [29681:0] info: 0.008192 0.016384 6
Jan 13 13:58:55 ipfire unbound: [29681:0] info: 0.016384 0.032768 8
Jan 13 13:58:55 ipfire unbound: [29681:0] info: 0.032768 0.065536 72
Jan 13 13:58:55 ipfire unbound: [29681:0] info: 0.065536 0.131072 154
Jan 13 13:58:55 ipfire unbound: [29681:0] info: 0.131072 0.262144 513
Jan 13 13:58:55 ipfire unbound: [29681:0] info: 0.262144 0.524288 508
Jan 13 13:58:55 ipfire unbound: [29681:0] info: 0.524288 1.000000 647
Jan 13 13:58:55 ipfire unbound: [29681:0] info: 1.000000 2.000000 132
Jan 13 13:58:55 ipfire unbound: [29681:0] info: 2.000000 4.000000 15
Jan 13 13:58:55 ipfire unbound: [29681:0] info: 4.000000 8.000000 1
Jan 13 13:59:13 ipfire unbound: [3672:0] notice: init module 0: validator
Jan 13 13:59:13 ipfire unbound: [3672:0] notice: init module 1: iterator
Jan 13 13:59:13 ipfire unbound: [3672:0] info: start of service (unbound 1.13.2).
Jan 13 13:59:14 ipfire unbound: [3672:0] info: generate keytag query _ta-4a5c-4f66. NULL IN

mutley · 13 January 2022 20:49

3 minutes is a long time to restart unbound. (but you’d also expect to see some errors in the log if it was having problems). Mine takes about 5 seconds, but that’s on Intel I5 cpu with 8gig ram. Before we get into verbose unbound logging, let’s see if it’s anything to do with the size or contents of the blocklist file. I you run dns-blocklist against a smallest list without any custom white or black lists, let’s see if it works.
So if you run ~/bin/dns_blocklist.sh -s 1 that will create a list from only the adaway.org source, which is 7522 entries. (FYI that’s been running for the last 5 mins on my system). Then test with nslookup against any entries in the `/etc/unbound/local.d/blocklist.conf’ file.

cbrown · 13 January 2022 21:06

it seems to start functioning right away (started your script at 13:58:55, unbound restarts at 13:59:13) at that point I could do nslookup and get desired behavior. It was 3-minutes in that behavior reverted to getting real ip address; and another minute or so before the script finished showing ‘./dns_blocklist.sh: Blocked Hosts Update, 143274 hosts blocked’
FWIW: I am running DNS-over-TLS. Do you think that could be relevant to the issue?

cbrown · 13 January 2022 21:13

Running ~/bin/dns_blocklist.sh -s 1 ran much much faster However running nslookup on entries in ‘/etc/unbound/local.d/blocklist.conf’ resulted in ip address resolved … not the desired 0.0.0.0

Thanks again for the continued help

mutley · 13 January 2022 21:19

I can’t see how TLS should effect it, as this should be picked up by unbound before it even tried to make an outbound TLS connection. I’ve never found any decent / reliable TLS servers, so gave up on it. But I’ll try to find some and test. The other thing we could try is query unbound without DNS see what it thinks. you can do that with this command unbound-control list_local_data that will give you a lot of stuff specific to your local network, so you might want to try unbound-control list_local_data | grep 0.0.0.0, and that will filter out a lot of your local stuff. (or grep 127.0.0.1) depending on how you are using the -r switch with dns-blocklist. if you see all the entries from /etc/unbound/local.d/blocklist.conf then at least unbound is still using that file and holding it in memory.

I’ll start plating with TLS.

cbrown · 13 January 2022 21:29

So the ‘unbound-control list_local_data’ nicely listed the 7522 entries from adaway.org

[root@ipfire ~]# unbound-control list_local_data | egrep “0.0.0.0$” | wc -l
7522

So far, so good

cbrown · 13 January 2022 21:48

Aha, it would seem the “Enable Safe Search” seems sort of suspect

Edit: After unchecking the “Enable Safe Search” feature on the " Domain Name System" page, things seem to be working as expected … even with DNS-over-TLS

Edit2: Even after running with your defaults – getting 143274 unique entries things seem to be working … even after 5 minutes

Thanks again

mutley · 13 January 2022 22:01

I’m so sorry this is taking so long. The fact that it’s obviously cached the file means its good, so it’s not that, also seems it’s accepting the include local.d/* and picking up the server: stanza I use (which I know was issue in some unbound versions, none that I’ve seen ipfire use though). I know you can overcome the local-data, but your config it the same as ipfire source, so shouldn’t be that. I guess let’s check your config.

make sure their are no other files are in /etc/unbound/local.d other that black list. If there is they might have something in them to overwrite the default unbound config.
check /etc/unbound/forward.conf make sure nothing strange is overwriting. should look like

forward-zone:
        name: "."
        forward-addr: <your dns IP>
        <repeat of above>

If that all checks out, let’s see if local-zone is also being ignored in some way. if your run ~/bin/dns_blocklist.sh -s 1 -r always_nxdomain the script will now create a different style unbound config. it will use a local-zone (not local-data) and DNS lookup should now return NXDOMAIN rather than an IP. It will have the same result of blocking just done it a different manor. You will now need to use unbound-control list_local_zones to check unbound has the file in memory. Again obviously use nslookup with a domain in the file, like nslookup tru.am

mutley · 13 January 2022 22:03

@cbrown GREAT. Glad you found it. Let me play with safe search and see what that does to unbound config and how I can make sure it’s all working. Thanks so much for your time and patience on solving this.

cbrown · 13 January 2022 22:07

Another FWIW … Here’s the set of TLS servers I’ve been successfully using for quite a while:

140.238.215.192 dot.post-factum.tk
199.58.81.218 dns.cmrg.net
91.239.100.100 anycast.uncensoreddns.org
5.1.66.255 anycast01.ffmuc.net
185.222.222.222 dns.sb
185.184.222.222 dns.sb

… from https://wiki.ipfire.org/dns/public-servers

hjkl · 17 January 2022 20:18

@mutley
I am using for… I forgot how many years.

Gakd to see is stil maintained that script : I closed my Raspberry Pi 1 (oh yeah!) that was providing PiHole service when discovered your script.

Happy to report it works also in core 162… Or at least I did not spot any error (yet).

At one moment in time, just for fun, I reached a dew millions lines in blocklist.conf by using 10 or more sources that aggregate other hundreds of sources…

Thank you,
Great script!
H&M

cbrown · 17 January 2022 20:26

Hi @hjkl would you mind sharing your set of additional blocklists?
Thanks,
@cbrown

mutley · 18 January 2022 02:02

@hjkl Really glad its been useful to you, and also glad you posted about it. it’s been running for years on my system without issues. There are a few small updates that will make managing your own lists easier now.

Still need to figure out why the ipfire safe search unbound configuration effects it though. But after looking at how that works, I’m kinda surprised it was even implemented, seems to defeat the purpose of configuring your own DNS servers.

mutley · 18 January 2022 02:08

@cbrown you can use any blocklist that uses a known format, so the majority of pi-hole block lists. There is a good list in the below url.

anon87475738 · 19 January 2022 07:26

@mut ley:

I guess that this is not possible with your script?

Deep CNAME inspection

“This will allow Pi-hole to find whether any domain in the CNAME chain is known to be blocked. If one is found, Pi-hole can now block the original query. The feature defaults to being enabled but can be disabled with an FTL config option ( CNAME_DEEP_INSPECT=false ).”

mutley · 19 January 2022 14:05

Yes it can. Read the NXDOMAIN part of the information.

Unbound can’t return an IP for these cases, it has to reject at the domain level, so you turn this on by setting the return to be something other than an IP address. Valid returns are listed on the page, but something like -r always_nxdomain will do what you want.

Word of caution, I found this to be too restrictive. Many of the blocklists that are maintained by individuals or small teams are not the most accurate, and once a domain added, usually it’s their for ever. So when you start blocking blindly at the next level higher you tend to block valid stuff. So be careful with the blocklists you pick.
But on the positive side, it will reduce the size of the blocklist it passes to unbound, so less overhead their.

anon87475738 · 19 January 2022 14:12

Thank you very much. With your Script Pi-Hole is then no longer necessary. Disadvantage would be less comfort compared to a GUI with statistics (especially if you want to find out why something causes problems).

Edit:
I tried the script:

dns_blocklist.sh -s 1,2,3,4,“https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/spam.mails",“https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/easylist”,“https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/crypto”,“https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/gambling”,“https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/malware”,“https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/notserious”,“https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/pornblock1”,“https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/pornblock2”,“https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/pornblock3”,“https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/pornblock4”,“https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts”,“https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt”,“https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt”,“https://raw.githubusercontent.com/ookangzheng/dbl-oisd-nl/master/dbl.txt”,“https://raw.githubusercontent.com/notracking/hosts-blocklists/master/dnscrypt-proxy/dnscrypt-proxy.blacklist.txt”,“https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/Fake-Science”,“https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/Phishing-Angriffe”,“https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/Streaming”,“https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/Win10Telemetry”,“https://easylist.to/easylistgermany/easylistgermany.txt”,“https://easylist.to/easylist/easyprivacy.txt”,“https://easylist.to/easylist/fanboy-social.txt”,“https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/extra.txt”,“https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt”,“https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt”,“https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt”,“https://raw.githubusercontent.com/AdguardTeam/AdguardFilters/master/MobileFilter/sections/adservers.txt”,"https://raw.githubusercontent.com/AdguardTeam/AdguardFilters/master/MobileFilter/sections/spyware.txt” -r 0.0.0.0
Retreived 0 domain names from local blacklist file
Retreived 7528 domain names from https://adaway.org/hosts.txt
Retreived 8730 domain names from https://winhelp2002.mvps.org/hosts.txt
Retreived 97346 domain names from https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
Retreived 76779 domain names from https://raw.githubusercontent.com/notracking/hosts-blocklists/master/hostnames.txt
Retreived 0 domain names from https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/spam.mails
Retreived 0 domain names from https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/easylist
Retreived 0 domain names from https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/crypto
Retreived 0 domain names from https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/gambling
Retreived 0 domain names from https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/malware
Retreived 0 domain names from https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/notserious
Retreived 0 domain names from https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/pornblock1
Retreived 0 domain names from https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/pornblock2
Retreived 0 domain names from https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/pornblock3
Retreived 0 domain names from https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/pornblock4
Retreived 97346 domain names from https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
Retreived 0 domain names from https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt
Retreived 0 domain names from https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
Retreived 0 domain names from https://raw.githubusercontent.com/ookangzheng/dbl-oisd-nl/master/dbl.txt
Retreived 0 domain names from https://raw.githubusercontent.com/notracking/hosts-blocklists/master/dnscrypt-proxy/dnscrypt-proxy.blacklist.txt
Retreived 0 domain names from https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/Fake-Science
Retreived 0 domain names from https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/Phishing-Angriffe
Retreived 0 domain names from https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/Streaming
Retreived 0 domain names from https://raw.githubusercontent.com/RPiList/specials/master/Blocklisten/Win10Telemetry
Retreived 33 domain names from https://easylist.to/easylistgermany/easylistgermany.txt
Retreived 12893 domain names from https://easylist.to/easylist/easyprivacy.txt
Retreived 4 domain names from https://easylist.to/easylist/fanboy-social.txt
Retreived 295 domain names from https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/extra.txt
Retreived 378 domain names from https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt
Retreived 0 domain names from https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt
Retreived 0 domain names from https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
Retreived 967 domain names from https://raw.githubusercontent.com/AdguardTeam/AdguardFilters/master/MobileFilter/sections/adservers.txt
Retreived 0 domain names from https://raw.githubusercontent.com/AdguardTeam/AdguardFilters/master/MobileFilter/sections/spyware.txt
Cleaning & Sorting list of 302299 entries
Writing list of 148965 entries to unbound configuration
Stopping Unbound DNS Proxy… [ OK ]
Starting Unbound DNS Proxy… [ OK ]
/root/bin/dns_blocklist.sh: Blocked Hosts Update, 148965 hosts blocked

Retreived 0 => this probably means that the list is incompatible. Similiar issue: https://forum.kuketz-blog.de/viewtopic.php?p=76319#p76319

mutley · 26 January 2022 15:59

That is correct. When I wrote this script all block-lists followed one of 3 standards, now that Pi-Hole has become popular it looks like another “standard” or non-standard has emerged, that is simply a list of domains with nothing else.
I have been thinking of if (and how) to incorporate those. But I also really haven’t seen anything in those lists that you can’t get in the better maintained ones.