I use IPSet with the lists mentioned above which are also vast and for very different kinds of threads.
Is IPSet not filling your IP tables up with rules from the lists? Isn’t that the point of IPSet to manage lists of IPs in iptables?
@ummeegge
Hi Ian,
yes IPset uses IPs and CIDRs from Firehol. IPSet stores IPs and CIDRs in bitmap or hash format, so also big lists are not a problem.
Best,
Erik
@ummeegge Ok so I’m not sure what I’m confused about then?
My comment is same in that I do not like blocking at the IP level for the reason that bad actors are deliberately hiding amongst legitimate endpoints so that you can’t do exactly this. It’s also how services such as Signal (and some TOR bridges) bypass censorship as an example.
It’s called something but the term escapes me at present.
I see your concerns even i haven´t had such problems but i use IPSet also in may other ways.
Reading your first topic again you are also not that happy with the URL-Filter solution which i wanted to offer mainly. We have tested also some performance at that time whereby to download, sort and converting a list contained 380258 domains the script needed
real 1m9.621s
user 0m30.920s
sys 0m11.823s
and there was also no noticable performance losses. But again, this was only meant as another possible or additional way.
May you mean scrambler like obfs4proxy and such ?
Best,
Erik
May you mean scrambler like obfs4proxy and such ?
I think what I was thinking of is called “domain fronting” but looks like Google and AWS have cracked down on it, I think is still possible with Cloudflare, might be less of an issue now I guess, but still I don’t want to block by Ip for these reasons.
What Signal would do was host their service on the same endpoint as some other services, and perform DNS lookups to the other service to get their IP address because their domain was being filtered by ISP DNS filter, and then starting the TLS handshake on the endpoint that has a certificate named for a bunch of services not just Signal therefore it couldn’t be blocked at either the DNS or IP level without blocking legitimate services.
My major concern with URL-Filtering through a non-transparent proxy is that it doesn’t really support come and go devices. And also I have to actually push proxy settings to all devices. If the devices leave the office i.e. go to an employees home or offsite to a remote location, for them to use the proxy they have to VPN in etc. Super messy. Where as I can just set the DNS server to my DNS server when any device gets a lease on my network, this is 1000% better IMO and when the device leaves my network, they are using their own DNS.
When you start dealing with personal devices, people don’t want to use their personal laptop connected to the work proxy etc. so with the shift to work from home and more and more BYOD happening I don’t see URL-filtering as having a future for me.
As you can force your users to use the system DNS server, you can force them to the non-transparent proxy ( deny HTTP(S) to destinations other than IPFire ). You can publish the proxy settings with DHCP.
1 Like
You can publish the proxy settings with DHCP.
Are they unset when they get a new lease at home? I’m guessing yes but want to be sure, if so that’s more viable than I was thinking.
This looks interesting. It is to bad it is not more user friendly to do.
No WUI. and not available as a plugin.
Using scrips is not my thing.
+1 a wui for this would be nice
It is the defined habit for DHCP requests/answers that the attributes are set according to the environment.
The persistence is a property of the device/OS.
@anon65703081 nad @hvacguy ,
there is already a development out there which uses IPSet --> IP Address Blacklists --> https://lists.ipfire.org/pipermail/development/2020-February/007085.html but with fixed lists. Not sure when and if this will be released.
Best,
Erik
2 Likes
Right, so if it’s set in DHCP on my network, and a user goes home gets a lease on their home network, will it forget the proxy settings?
The user gets the information from the respective DHCP server. This means that when he comes home, what his dhcp specifies applies, and when he returns to the company, the settings there apply.
1 Like
Yea but the case when the DHCP server is not specifying any proxy settings, it assumes to clear the settings that are set from the last lease I guess you mean?
there is already a development out there which uses IPSet → IP Address Blacklists → IP Address Blacklists - Development - lists.ipfire.org 2 but with fixed lists. Not sure when and if this will be released.
Best,
Erik
would be nice to push, I did not know about these possibilities until today.
1 Like
I am assuming yes
1 Like
If the device is configured right. But who knows.
@anon65703081
I think the core developers are waiting for a actual push from Tim as you can read in the topic. IPset is now available since 2016 but it is not that often used at it seems (may i am wrong with this). I think i won´t use Tim´s development even it is really great what he have done but i simply use own lists and wanted to have a more liberal choice.
@anon45931963
To go one step further with your request, have modified now the script and integrated only your list.
Script looks like this:
#!/bin/bash -
set -x
#
# Test script for individual automation for custom blacklists and their updates
# ummeegge[at]ipfire.org, $date Wed Oct 01 17:35:42 CEST 2015
# Modified for community request -->
# https://community.ipfire.org/t/protect-users-who-get-spammy-phishing-links-in-emails/4111
# for a phising list. $date Sat Dez 19 10:42:23 CEST 2020
###########################################################################################
#
# Locations
INSTDIR="/tmp/customlists";
TMPLIST="/tmp/customlists/list";
SET="/var/ipfire/urlfilter/settings";
LIST="/var/ipfire/urlfilter/blacklists/custom/blocked/domains";
# Check for installation dir
if [ -d "${INSTDIR}" ]; then
rm -rf ${INSTDIR};
mkdir ${INSTDIR};
else
mkdir ${INSTDIR};
fi
# Set hock for 'Customblacklist' in URL-Filter CGI and activate 'Custom blacklist'
HOCK=$(grep -Fx 'ENABLE_CUSTOM_BLACKLIST=on' ${SET})
if [[ -z ${HOCK} ]]; then
sed -i -e 's/ENABLE_CUSTOM_BLACKLIST=.*/ENABLE_CUSTOM_BLACKLIST=on/' ${SET}
squidGuard -c /etc/squidGuard/squidGuard.conf &
sleep 2;
fi
#---------------------------------------------------------------------------------
# Change dir
cd ${INSTDIR};
## Download URLs and files for external Domain lists
URL1="https://phishing.army/download/phishing_army_blocklist_extended.txt";
FILE1="phishing_army_blocklist_extended.txt";
####################################################################################
## If you want to integrate your own local blacklist, delete the '#' in the USERLIST
## line and set the path to your list and the name.
#USERLIST="/path/to/list/nameoflist";
####################################################################################
## Download and process and/or integrate it
# Get all lists
wget ${URL1} >/dev/null 2>&1;
## Processing domains
#1
sed -e '/^#/d' -e '/^?/d' ${FILE1} > ${TMPLIST};
############################################################################################################
## if you want to integrate your own above defined local blacklist, delete the '#' in the '${USERLIST}' line
#cat ${USERLIST} >> ${TMPLIST};
############################################################################################################
# remove unwanted Carriage Returns
sed -i -e 's/^M//g' ${TMPLIST}
# sort and delete double entries and paste it to custom blacklist
sort ${TMPLIST} | uniq > ${LIST};
# Updating squidGuard database and restart and log it to syslog
squidGuard -C "${LIST}";
squidGuard -c /etc/squidGuard/squidGuard.conf &
sleep 2;
/etc/init.d/squid flush;
logger -t ipfire "URL filter: Custom blacklist updated";
# CleanUP
#rm -rf ${INSTDIR};
# EOF
URL-Filter and Squid are running including logging with a modified block page settings.
Your above posted list have:
$ wc -l customlists/phishing_army_blocklist_extended.txt
19814 customlists/phishing_army_blocklist_extended.txt
entries.
Have executed it in debugging mode and checked the needed time:
$ time ./domain-phising-list
+ INSTDIR=/tmp/customlists
+ TMPLIST=/tmp/customlists/list
+ SET=/var/ipfire/urlfilter/settings
+ LIST=/var/ipfire/urlfilter/blacklists/custom/blocked/domains
+ '[' -d /tmp/customlists ']'
+ rm -rf /tmp/customlists
+ mkdir /tmp/customlists
++ grep -Fx ENABLE_CUSTOM_BLACKLIST=on /var/ipfire/urlfilter/settings
+ HOCK=ENABLE_CUSTOM_BLACKLIST=on
+ [[ -z ENABLE_CUSTOM_BLACKLIST=on ]]
+ cd /tmp/customlists
+ URL1=https://phishing.army/download/phishing_army_blocklist_extended.txt
+ FILE1=phishing_army_blocklist_extended.txt
+ wget https://phishing.army/download/phishing_army_blocklist_extended.txt
+ sed -e '/^#/d' -e '/^?/d' phishing_army_blocklist_extended.txt
+ sed -i -e 's/^M//g' /tmp/customlists/list
+ sort /tmp/customlists/list
+ uniq
+ squidGuard -C /var/ipfire/urlfilter/blacklists/custom/blocked/domains
+ sleep 2
+ squidGuard -c /etc/squidGuard/squidGuard.conf
+ /etc/init.d/squid flush
Stopping Squid Proxy Server (this may take up to a few minutes).... [ OK ]
Creating Squid swap directories... [ OK ]
Starting Squid Proxy Server...
2020/12/19 10:37:18| WARNING: BCP 177 violation. Detected non-functional IPv6 loopback. [ OK ]
+ logger -t ipfire 'URL filter: Custom blacklist updated'
./domain-phising-list 1.80s user 1.54s system 25% cpu 13.315 total
The warning can be ignored --> Squid web proxy started with 2 errors .
URL-Filter custom-blacklist:
Just tested it with hxxx://102312.*.com/ which looks like this:
URL-Filter logging:
Needed resources with 5 users:
$ ./ps_mem.py | grep -E 'squid|redirect_|squidGuard'
5.0 MiB + 1.8 MiB = 6.8 MiB squidGuard (4)
21.7 MiB + 3.7 MiB = 25.4 MiB redirect_wrappe (4)
24.7 MiB + 3.0 MiB = 27.7 MiB squid (2)
To get a little more butter to the fish as we say it here.
Best,
Erik
EDIT: Wiki for “Web Proxy Auto-Discovery Protocol (WPAD) / Proxy Auto-Config (PAC)” --> https://wiki.ipfire.org/configuration/network/proxy/extend/wpad .
3 Likes
Thanks. I was more interested in the manageability per wui.
1 Like