URL Filter and self updating blacklists?

Networking Web Proxy
8

hmm… my skills in programming is on the html level :smiley: so just nothing.
what do you mean with unbound filtering? Just filter via DNS-Server?

9

Unbound drives the DNS service in IPFire. Adding a block list in the /etc/unbound/local.d directory will do the job.

10

Does it need a special formating of the list?

11
12

I haven’t looked into Squidguard in detail yet. Only cursorily a few months ago I used it once in IPFire. My impression was that the result was significantly worse compared to Pi-Hole or AdGuard Home. That was only a short trial. However, I’m afraid that my impression would be confirmed if I used it more intensively.

The pre-built blacklists for Squidguard are probably not nearly as extensive as all the lists you can use together in Pi-Hole, etc.

Trackers are the biggest problem, for which I would additionally like to use something like Pi-Hole.
I know that Pi-Hole and AdGuard can be bypassed. Nevertheless, you can clearly notice the difference.

Apparently there are ways to convert the lists. Seems to involve effort though. There are some converters, but I don’t know how good they work (and if they still work).
Would be nice to have such functionality in IPFfire.

13

Hi all,
not sure if this interesting but we have had a little similar discussion in → https://community.ipfire.org/t/protect-users-who-get-spammy-phishing-links-in-emails in here → Protect users who get spammy phishing links in emails - #40 by ummeegge (more specific) with an older idea of how to get lists updated in SquidGuard with a little bashing around but more important are IMHO some good lists.

Best,

Erik

1 Like
14

Blackweb is a project that collects and unifies public blocklists of domains (porn, downloads, drugs, malware, spyware, trackers, bots, social networks, warez, weapons, etc.) to make them compatible with Squid-Cache

Not tried yet but sounds interesting.

edit:
Doesn’t work with IPFIre.

15

@anon87475738 This is my script that I created long before Pi Hole, and you are correct, it makes ipfire have the same functionality and can consume many different formats. I just noticed that it does need a small update to be fully compatible with the version of unbound in core update 162, but I didn’t think anyone was using it so didn’t post the update to GitHub. I’ll post that in the next few days. But yes, it’s been on my ipfire box for over 5 years and works very effectivly.

3 Likes
16

I updated the script that was referenced above. It uses DNS to block sites, you can use the default lists (just realized it uses the same as PiHole), or create your own, also add custom whitelists & blacklists, and many other options if you want to get into unbound tagging.

2 Likes
17

Hmm, starts out like it is working but then reverts to giving resolved addresses after a bit. Note the script hangs for an extended period of time at “Starting Unbound DNS Proxy …”. During the first few minutes, I get 0.0.0.0 for blacklisted items. Then after a few minutes I start getting real resolved ip addresses again.

Writing list of 143035 entries to unbound configuration
Stopping Unbound DNS Proxy…
Starting Unbound DNS Proxy…
/root/bin/dns_blocklist.sh: Blocked Hosts Update, 143035 hosts blocked

Could you point me where to look for where things are going amiss?
Thanks

18

The slow “Starting Unbound…” is simply the script waiting for unbound to start before exiting. Unbound is slow to start so not hanging, just normal behavior.

The output of the script is simply /etc/unbound/local.d/blocklist.conf so if that file it their (and in correct format), something else is probably going on. What are you using and where are you using it to check the DNS? Below is some info that may help, but need to know a little more to help you further.

If you monitor /var/log/messages that will tell you any unbound error / warning. so
cat /var/log/messages | grep -I unbound to look at historical errors, and tail -f /var/log/messages | grep -I unbound to monitor current messages.

To check ipfire DNS for unbound. Simply use nslookup blocked.host.domainname on ipfire machine, if that returns 0.0.0.0 but other hosts are reverting back. This it’s not unbound/ipfire/this scrip.

As for getting 0.0.0.0 for first few minutes then reverting back if this is on a machine other than ipfire, your source machine may be overriding the DNS lookup from ipfire and reverting to a different DNS lookup. (There are so many options here depending on OS and configurations) that would need to know a little more about the system you are using.
But a simple option would be to block any DNS queries in ipfire firewall rules. You could also try returning a valid (but incorrect) ip like 127.0.0.1 rather than 0.0.0.0. That would be run the script with the -r 127.0.0.1 option.

19

Okay, ran the script with -r 127.0.0.1 and doing nslookup on ipfire box started getting

[root@ipfire local.d]# nslookup zeusclicks.com
Server: 127.0.0.1
Address: 127.0.0.1#53
Name: zeusclicks.com
Address: 127.0.0.1

after a minute or so …

[root@ipfire local.d]# nslookup zeusclicks.com
Server: 127.0.0.1
Address: 127.0.0.1#53
Non-authoritative answer:
Name: zeusclicks.com
Address: 65.175.104.93

Here’s the grep from /var/log/messages … do you see anything suspect ?

Jan 13 11:50:20 ipfire unbound: [4700:0] info: service stopped (unbound 1.13.2).
Jan 13 11:50:20 ipfire unbound: [4700:0] info: server stats for thread 0: 26768 queries, 8455 answers from cache, 18313 recursions, 584 prefetch, 0 rejected by ip ratelimiting
Jan 13 11:50:20 ipfire unbound: [4700:0] info: server stats for thread 0: requestlist max 24 avg 0.753876 exceeded 0 jostled 0
Jan 13 11:50:20 ipfire unbound: [4700:0] info: average recursion processing time 0.295940 sec
Jan 13 11:50:20 ipfire unbound: [4700:0] info: histogram of recursion processing times
Jan 13 11:50:20 ipfire unbound: [4700:0] info: [25%]=0.0464152 median[50%]=0.190709 [75%]=0.464073
Jan 13 11:50:20 ipfire unbound: [4700:0] info: lower(secs) upper(secs) recursions
Jan 13 11:50:20 ipfire unbound: [4700:0] info: 0.000000 0.000001 3689
Jan 13 11:50:20 ipfire unbound: [4700:0] info: 0.000064 0.000128 1
Jan 13 11:50:20 ipfire unbound: [4700:0] info: 0.000128 0.000256 1
Jan 13 11:50:20 ipfire unbound: [4700:0] info: 0.000256 0.000512 1
Jan 13 11:50:20 ipfire unbound: [4700:0] info: 0.000512 0.001024 5
Jan 13 11:50:20 ipfire unbound: [4700:0] info: 0.001024 0.002048 10
Jan 13 11:50:20 ipfire unbound: [4700:0] info: 0.002048 0.004096 18
Jan 13 11:50:20 ipfire unbound: [4700:0] info: 0.004096 0.008192 62
Jan 13 11:50:20 ipfire unbound: [4700:0] info: 0.008192 0.016384 71
Jan 13 11:50:20 ipfire unbound: [4700:0] info: 0.016384 0.032768 168
Jan 13 11:50:20 ipfire unbound: [4700:0] info: 0.032768 0.065536 1326
Jan 13 11:50:20 ipfire unbound: [4700:0] info: 0.065536 0.131072 2010
Jan 13 11:50:20 ipfire unbound: [4700:0] info: 0.131072 0.262144 3944
Jan 13 11:50:20 ipfire unbound: [4700:0] info: 0.262144 0.524288 3153
Jan 13 11:50:20 ipfire unbound: [4700:0] info: 0.524288 1.000000 3142
Jan 13 11:50:20 ipfire unbound: [4700:0] info: 1.000000 2.000000 642
Jan 13 11:50:20 ipfire unbound: [4700:0] info: 2.000000 4.000000 54
Jan 13 11:50:20 ipfire unbound: [4700:0] info: 4.000000 8.000000 12
Jan 13 11:50:20 ipfire unbound: [4700:0] info: 8.000000 16.000000 4
Jan 13 11:50:38 ipfire unbound: [29681:0] notice: init module 0: validator
Jan 13 11:50:38 ipfire unbound: [29681:0] notice: init module 1: iterator
Jan 13 11:50:38 ipfire unbound: [29681:0] info: start of service (unbound 1.13.2).
Jan 13 11:50:39 ipfire unbound: [29681:0] info: generate keytag query _ta-4a5c-4f66. NULL IN

20

Very sorry you are having problems. That’s really strange behavior, it’s as if the file ‘/etc/unbound/local.d/blocklist.conf’ is being remove. Can you check that file still exists and is NOT empty once it reverts back to giving you correct nslookup info?
something like cat /etc/unbound/local.d/blocklist.conf | grep zeusclicks.com and make sure you get some output.

if you do get output, can you please try keeping the entire log of unbound until you see it revert back. Then post that, see if there is anything. I’m trying not to get you to put unbound into verbose mode as that a ton of stuff to post & sift through. So something like the below.

  1. In one terminal use tail -f /var/log/messages | grep -I unbound and keep that running.
  2. Then in another terminal, run the dns-blocklist.sh script, and keep doing nslookup zeusclicks.com until it fails.
  3. Then post the output from the first terminal with tail command output.
21

I guess one other thing is to check your unbound config file against ipFire distribution, make sure there are no changes.
So your file /etc/unbound/unbound.conf against this one.

22

  1. The ‘/etc/unbound/local.d/blocklist.conf’ file is still intact with 143282 entries and looks normal and zeusclicks.com is in the file.

  2. diffed my /etc/unbound/unbound.conf and found no diffs

  3. stand by for more … :grinning:

Thanks for the help

23

Okay ran again. Started out working fine for the first three minutes. then started getting the ip address resolved by dns as previously reported. Still waiting for 'Starting Unbound DNS Proxy … ’ to finish (as previously reported – this takes a really really long time), Oh well it just completed with
‘./dns_blocklist.sh: Blocked Hosts Update, 143274 hosts blocked’
Can’t say I see anything different in the log grep but here it is …

Jan 13 13:58:55 ipfire unbound: [29681:0] info: service stopped (unbound 1.13.2).
Jan 13 13:58:55 ipfire unbound: [29681:0] info: server stats for thread 0: 3499 queries, 1211 answers from cache, 2288 recursions, 46 prefetch, 0 rejected by ip ratelimiting
Jan 13 13:58:55 ipfire unbound: [29681:0] info: server stats for thread 0: requestlist max 20 avg 1.25321 exceeded 0 jostled 0
Jan 13 13:58:55 ipfire unbound: [29681:0] info: average recursion processing time 0.422614 sec
Jan 13 13:58:55 ipfire unbound: [29681:0] info: histogram of recursion processing times
Jan 13 13:58:55 ipfire unbound: [29681:0] info: [25%]=0.156622 median[50%]=0.344193 [75%]=0.688251
Jan 13 13:58:55 ipfire unbound: [29681:0] info: lower(secs) upper(secs) recursions
Jan 13 13:58:55 ipfire unbound: [29681:0] info: 0.000000 0.000001 226
Jan 13 13:58:55 ipfire unbound: [29681:0] info: 0.001024 0.002048 4
Jan 13 13:58:55 ipfire unbound: [29681:0] info: 0.002048 0.004096 2
Jan 13 13:58:55 ipfire unbound: [29681:0] info: 0.008192 0.016384 6
Jan 13 13:58:55 ipfire unbound: [29681:0] info: 0.016384 0.032768 8
Jan 13 13:58:55 ipfire unbound: [29681:0] info: 0.032768 0.065536 72
Jan 13 13:58:55 ipfire unbound: [29681:0] info: 0.065536 0.131072 154
Jan 13 13:58:55 ipfire unbound: [29681:0] info: 0.131072 0.262144 513
Jan 13 13:58:55 ipfire unbound: [29681:0] info: 0.262144 0.524288 508
Jan 13 13:58:55 ipfire unbound: [29681:0] info: 0.524288 1.000000 647
Jan 13 13:58:55 ipfire unbound: [29681:0] info: 1.000000 2.000000 132
Jan 13 13:58:55 ipfire unbound: [29681:0] info: 2.000000 4.000000 15
Jan 13 13:58:55 ipfire unbound: [29681:0] info: 4.000000 8.000000 1
Jan 13 13:59:13 ipfire unbound: [3672:0] notice: init module 0: validator
Jan 13 13:59:13 ipfire unbound: [3672:0] notice: init module 1: iterator
Jan 13 13:59:13 ipfire unbound: [3672:0] info: start of service (unbound 1.13.2).
Jan 13 13:59:14 ipfire unbound: [3672:0] info: generate keytag query _ta-4a5c-4f66. NULL IN

24

3 minutes is a long time to restart unbound. (but you’d also expect to see some errors in the log if it was having problems). Mine takes about 5 seconds, but that’s on Intel I5 cpu with 8gig ram. Before we get into verbose unbound logging, let’s see if it’s anything to do with the size or contents of the blocklist file. I you run dns-blocklist against a smallest list without any custom white or black lists, let’s see if it works.
So if you run ~/bin/dns_blocklist.sh -s 1 that will create a list from only the adaway.org source, which is 7522 entries. (FYI that’s been running for the last 5 mins on my system). Then test with nslookup against any entries in the `/etc/unbound/local.d/blocklist.conf’ file.

25

it seems to start functioning right away (started your script at 13:58:55, unbound restarts at 13:59:13) at that point I could do nslookup and get desired behavior. It was 3-minutes in that behavior reverted to getting real ip address; and another minute or so before the script finished showing ‘./dns_blocklist.sh: Blocked Hosts Update, 143274 hosts blocked’
FWIW: I am running DNS-over-TLS. Do you think that could be relevant to the issue?

26

Running ~/bin/dns_blocklist.sh -s 1 ran much much faster :grinning: However running nslookup on entries in ‘/etc/unbound/local.d/blocklist.conf’ resulted in ip address resolved … not the desired 0.0.0.0

Thanks again for the continued help

27

I can’t see how TLS should effect it, as this should be picked up by unbound before it even tried to make an outbound TLS connection. I’ve never found any decent / reliable TLS servers, so gave up on it. But I’ll try to find some and test. The other thing we could try is query unbound without DNS see what it thinks. you can do that with this command unbound-control list_local_data that will give you a lot of stuff specific to your local network, so you might want to try unbound-control list_local_data | grep 0.0.0.0, and that will filter out a lot of your local stuff. (or grep 127.0.0.1) depending on how you are using the -r switch with dns-blocklist. if you see all the entries from /etc/unbound/local.d/blocklist.conf then at least unbound is still using that file and holding it in memory.

I’ll start plating with TLS.