URL Filter and self updating blacklists?

fstarter · 4 November 2021 23:26

Hi,
is there a possibility to set some self updating lists for google, facebook and co. tracking? Not by uploading a txt file, but by seting the url with such file.

best regards
fstarter

roberto · 5 November 2021 19:27

Hi @fstarter.

You can create your own blacklist using the “Blacklist editor”. It creates a category for you that you can enable/disable and put the URLs you want.

Too easy. You can Google countless lists to block the URLs you want. There are a lot of them and if they are not in the copy/paste format, you modify them with “notepad++” and you create your own filter.

Hope this idea helps you.

You will tell us.

Greetings.

fstarter · 6 November 2021 09:40

Hi @roberto,
thanks. I know that I can create and / or upload a blacklist. The thing is, that it’s better to have blacklists, which are maintained / updated by somebody. The world of tracker and co. is very dynamic. So there is maby a possibility to insert a url to the blacklist of some organisations. That would be nice.

Best regards
fstarter

mfischer · 6 November 2021 19:03

Hi,

This can be done:

HTH,
Matthias

fkienker · 6 November 2021 19:39

The problem is these external blacklist databases are formatted in a variety of ways. They generally have to be reformated and “cleaned up” before they can work with the URL filter. We wrote shell scripts that take care of downloading, reformating, merging into the URL Filter, and then restarting Squid to activate the updated blacklists on a scheduled basis. It was not a trivial task and they to be maintained on a fairly regular basis.

The IPFire developers have more than enough to keep themselves busy. Something like this would be a low-priority item.

Best regards,
Fred

fstarter · 8 November 2021 19:37

is there any list-url out there just with the right format?

fkienker · 9 November 2021 22:12

I don’t believe we ever found a useful list in “native” IPFire format. When everyone moved from HTTP to HTTPS the usefulness of the URLFilter part of squid decreased significantly. We switched to doing the filtering in UnBound, which performs better.

How is your curl, awk, sed, grep and bash scripting? It’s not hard to put together an automated script to do this. It just isn’t going to be a pretty web page in IPFire unless you are willing to invest a lot of time.

fstarter · 9 November 2021 22:28

hmm… my skills in programming is on the html level so just nothing.
what do you mean with unbound filtering? Just filter via DNS-Server?

fkienker · 16 November 2021 22:52

Unbound drives the DNS service in IPFire. Adding a block list in the /etc/unbound/local.d directory will do the job.

fstarter · 17 November 2021 10:46

Does it need a special formating of the list?

anon87475738 · 20 November 2021 13:20

anon87475738 · 20 November 2021 13:31

I haven’t looked into Squidguard in detail yet. Only cursorily a few months ago I used it once in IPFire. My impression was that the result was significantly worse compared to Pi-Hole or AdGuard Home. That was only a short trial. However, I’m afraid that my impression would be confirmed if I used it more intensively.

The pre-built blacklists for Squidguard are probably not nearly as extensive as all the lists you can use together in Pi-Hole, etc.

Trackers are the biggest problem, for which I would additionally like to use something like Pi-Hole.
I know that Pi-Hole and AdGuard can be bypassed. Nevertheless, you can clearly notice the difference.

Apparently there are ways to convert the lists. Seems to involve effort though. There are some converters, but I don’t know how good they work (and if they still work).
Would be nice to have such functionality in IPFfire.

ummeegge · 22 November 2021 19:16

Hi all,
not sure if this interesting but we have had a little similar discussion in → https://community.ipfire.org/t/protect-users-who-get-spammy-phishing-links-in-emails in here → Protect users who get spammy phishing links in emails - #40 by ummeegge (more specific) with an older idea of how to get lists updated in SquidGuard with a little bashing around but more important are IMHO some good lists.

Best,

Erik

anon87475738 · 3 January 2022 10:53

Blackweb is a project that collects and unifies public blocklists of domains (porn, downloads, drugs, malware, spyware, trackers, bots, social networks, warez, weapons, etc.) to make them compatible with Squid-Cache

Not tried yet but sounds interesting.

edit:
Doesn’t work with IPFIre.

mutley · 12 January 2022 02:39

@anon87475738 This is my script that I created long before Pi Hole, and you are correct, it makes ipfire have the same functionality and can consume many different formats. I just noticed that it does need a small update to be fully compatible with the version of unbound in core update 162, but I didn’t think anyone was using it so didn’t post the update to GitHub. I’ll post that in the next few days. But yes, it’s been on my ipfire box for over 5 years and works very effectivly.

mutley · 12 January 2022 23:32

I updated the script that was referenced above. It uses DNS to block sites, you can use the default lists (just realized it uses the same as PiHole), or create your own, also add custom whitelists & blacklists, and many other options if you want to get into unbound tagging.

cbrown · 13 January 2022 01:29

Hmm, starts out like it is working but then reverts to giving resolved addresses after a bit. Note the script hangs for an extended period of time at “Starting Unbound DNS Proxy …”. During the first few minutes, I get 0.0.0.0 for blacklisted items. Then after a few minutes I start getting real resolved ip addresses again.

Writing list of 143035 entries to unbound configuration
Stopping Unbound DNS Proxy…
Starting Unbound DNS Proxy…
/root/bin/dns_blocklist.sh: Blocked Hosts Update, 143035 hosts blocked

Could you point me where to look for where things are going amiss?
Thanks

mutley · 13 January 2022 16:27

The slow “Starting Unbound…” is simply the script waiting for unbound to start before exiting. Unbound is slow to start so not hanging, just normal behavior.

The output of the script is simply /etc/unbound/local.d/blocklist.conf so if that file it their (and in correct format), something else is probably going on. What are you using and where are you using it to check the DNS? Below is some info that may help, but need to know a little more to help you further.

If you monitor /var/log/messages that will tell you any unbound error / warning. so
cat /var/log/messages | grep -I unbound to look at historical errors, and tail -f /var/log/messages | grep -I unbound to monitor current messages.

To check ipfire DNS for unbound. Simply use nslookup blocked.host.domainname on ipfire machine, if that returns 0.0.0.0 but other hosts are reverting back. This it’s not unbound/ipfire/this scrip.

As for getting 0.0.0.0 for first few minutes then reverting back if this is on a machine other than ipfire, your source machine may be overriding the DNS lookup from ipfire and reverting to a different DNS lookup. (There are so many options here depending on OS and configurations) that would need to know a little more about the system you are using.
But a simple option would be to block any DNS queries in ipfire firewall rules. You could also try returning a valid (but incorrect) ip like 127.0.0.1 rather than 0.0.0.0. That would be run the script with the -r 127.0.0.1 option.

cbrown · 13 January 2022 18:02

Okay, ran the script with -r 127.0.0.1 and doing nslookup on ipfire box started getting

[root@ipfire local.d]# nslookup zeusclicks.com
Server: 127.0.0.1
Address: 127.0.0.1#53
Name: zeusclicks.com
Address: 127.0.0.1

after a minute or so …

[root@ipfire local.d]# nslookup zeusclicks.com
Server: 127.0.0.1
Address: 127.0.0.1#53
Non-authoritative answer:
Name: zeusclicks.com
Address: 65.175.104.93

Here’s the grep from /var/log/messages … do you see anything suspect ?

Jan 13 11:50:20 ipfire unbound: [4700:0] info: service stopped (unbound 1.13.2).
Jan 13 11:50:20 ipfire unbound: [4700:0] info: server stats for thread 0: 26768 queries, 8455 answers from cache, 18313 recursions, 584 prefetch, 0 rejected by ip ratelimiting
Jan 13 11:50:20 ipfire unbound: [4700:0] info: server stats for thread 0: requestlist max 24 avg 0.753876 exceeded 0 jostled 0
Jan 13 11:50:20 ipfire unbound: [4700:0] info: average recursion processing time 0.295940 sec
Jan 13 11:50:20 ipfire unbound: [4700:0] info: histogram of recursion processing times
Jan 13 11:50:20 ipfire unbound: [4700:0] info: [25%]=0.0464152 median[50%]=0.190709 [75%]=0.464073
Jan 13 11:50:20 ipfire unbound: [4700:0] info: lower(secs) upper(secs) recursions
Jan 13 11:50:20 ipfire unbound: [4700:0] info: 0.000000 0.000001 3689
Jan 13 11:50:20 ipfire unbound: [4700:0] info: 0.000064 0.000128 1
Jan 13 11:50:20 ipfire unbound: [4700:0] info: 0.000128 0.000256 1
Jan 13 11:50:20 ipfire unbound: [4700:0] info: 0.000256 0.000512 1
Jan 13 11:50:20 ipfire unbound: [4700:0] info: 0.000512 0.001024 5
Jan 13 11:50:20 ipfire unbound: [4700:0] info: 0.001024 0.002048 10
Jan 13 11:50:20 ipfire unbound: [4700:0] info: 0.002048 0.004096 18
Jan 13 11:50:20 ipfire unbound: [4700:0] info: 0.004096 0.008192 62
Jan 13 11:50:20 ipfire unbound: [4700:0] info: 0.008192 0.016384 71
Jan 13 11:50:20 ipfire unbound: [4700:0] info: 0.016384 0.032768 168
Jan 13 11:50:20 ipfire unbound: [4700:0] info: 0.032768 0.065536 1326
Jan 13 11:50:20 ipfire unbound: [4700:0] info: 0.065536 0.131072 2010
Jan 13 11:50:20 ipfire unbound: [4700:0] info: 0.131072 0.262144 3944
Jan 13 11:50:20 ipfire unbound: [4700:0] info: 0.262144 0.524288 3153
Jan 13 11:50:20 ipfire unbound: [4700:0] info: 0.524288 1.000000 3142
Jan 13 11:50:20 ipfire unbound: [4700:0] info: 1.000000 2.000000 642
Jan 13 11:50:20 ipfire unbound: [4700:0] info: 2.000000 4.000000 54
Jan 13 11:50:20 ipfire unbound: [4700:0] info: 4.000000 8.000000 12
Jan 13 11:50:20 ipfire unbound: [4700:0] info: 8.000000 16.000000 4
Jan 13 11:50:38 ipfire unbound: [29681:0] notice: init module 0: validator
Jan 13 11:50:38 ipfire unbound: [29681:0] notice: init module 1: iterator
Jan 13 11:50:38 ipfire unbound: [29681:0] info: start of service (unbound 1.13.2).
Jan 13 11:50:39 ipfire unbound: [29681:0] info: generate keytag query _ta-4a5c-4f66. NULL IN

mutley · 13 January 2022 19:19

Very sorry you are having problems. That’s really strange behavior, it’s as if the file ‘/etc/unbound/local.d/blocklist.conf’ is being remove. Can you check that file still exists and is NOT empty once it reverts back to giving you correct nslookup info?
something like cat /etc/unbound/local.d/blocklist.conf | grep zeusclicks.com and make sure you get some output.

if you do get output, can you please try keeping the entire log of unbound until you see it revert back. Then post that, see if there is anything. I’m trying not to get you to put unbound into verbose mode as that a ton of stuff to post & sift through. So something like the below.

In one terminal use tail -f /var/log/messages | grep -I unbound and keep that running.
Then in another terminal, run the dns-blocklist.sh script, and keep doing nslookup zeusclicks.com until it fails.
Then post the output from the first terminal with tail command output.