URL Blacklist update from Univ. Toulouse doesn't work

jon · 25 April 2022 02:32

I just changed from Shalla to Univ. Toulouse and it worked OK for me.

Once the switch happened it did take a few minutes to download the new info. I’d guess 2-3 minutes (a guess!)

(I don’t know why there are four entries)

I am assuming you did see this on the Shalla site:

mumpitz · 25 April 2022 03:16

I have not changed anything, that must have happened automatically with an update, I think times exactly 104 days ago, since the update fails.
Through my search here in the forum I learned that a change took place because Shalla stopped its service.
How and where can I check the URL with which ipfire tries to retrieve the list, possibly there is an error in it.

jon · 25 April 2022 03:32

menu Logs → System Logs . Drop down menu to URL Filter Blacklist. Click Update

I am guessing the 104 days is since Shalla stopped service. (I seem to remember my day count being a similar number).

This is what I did to eliminate the Shalla categories on the URL Filter page.

And then I turned on the Univ. Toulouse service.

Then it all worked!

mumpitz · 25 April 2022 03:46

There is the log that the update fails, but there is not the URL of the new list.

I will try to remove the old list and then do the update again.

Small update:
I deleted the blacklists folder and tried to update the list, failed again.
Then I searched for the URL and found it in autoupdate.conf, then downloaded the list and uploaded it to the ipfire, that worked, an update still fails. No idea why.

mumpitz · 27 April 2022 16:18

The manual update via the custom source URL option now worked, the Toulouse list categories were also created correctly. I hope that now the daily update will be done automatically again.

06:19:40 installpackage[urlfilter]: URL filter blacklist - Update from Univ. Toulouse completed

mumpitz · 28 April 2022 22:21

So the automatic update still fails, no idea what is wrong, if anyone has an idea please shout here.

bonnietwin · 29 April 2022 08:10

Can you check if you have the following symlink created in the /etc/frcon.daily/ directory.

lrwxrwxrwx 1 root root 39 Apr 29 10:00 urlfilter → /var/ipfire/urlfilter/bin/autoupdate.pl

The link should be created when you select the update frequency and then press Save update settings.
If you choose weekly or monthly the link is made in the fcron.weekly or fcron.monthly directories.

I have found with suricata that sometimes if the symlink has been removed or not created properly then just pressing the Save update settings button while staying on the same frequency doesn’t fix the problem.

If you don’t have the link in the fcron.daily directory then you need to change the frequency to another period such as weekly, press Save update settings and then change back to daily and press Save update settings again. Then you can check if the symlink has been created.

mumpitz · 29 April 2022 08:47

Yes the link is set, it also tries to update every day, but it fails

installpackage[urlfilter]: URL filter blacklist - ERROR: Not a valid URL filter blacklist

I also checked if the path for the download is correct, everything is correct, manually it works, automatically it doesn’t. For whatever reason.

Edit:
Yes too bad, the manual update now also no longer works, it follows the error message

installpackage[urlfilter]: URL filter blacklist - ERROR: Unable to retrieve blacklist from Univ. Toulouse

mumpitz · 1 May 2022 00:54

Could someone give me a URL from Toulouse that works?
I have now tried 3 different ftp-clients except the ipfire to connect to this FTP server (ftp://ftp.univ-tlse1.fr/pub/reseau/cache/squidguard_contrib/blacklists.tar.gz).
Either a user with password is required and an anonymous login is not possible, or I get timeouts when connecting.
In this case it will never work.
Thanks!

trish · 1 May 2022 05:18

I am not sure what the exact URL is for a mirror, I am also getting errors connecting to the ftp

Here are some links I found regarding this UT1 blacklist
http://cri.univ-tlse1.fr/blacklists/

This supposed to be a mirror on Ghub:

Hope this helps

mumpitz · 1 May 2022 10:58

Thanks for the tip, I made out an address and it seems to work.
http://dsi.ut-capitole.fr/blacklists/download/blacklists.tar.gz
First manual attempt was a success.
Source:

github.com

ramonromancastro/Blacklists-UT1/blob/master/Blacklists-UT1.sh

#!/bin/bash

# Blacklists-UT1.sh is a bash function to download Blacklists UT1 lists
# (http://dsi.ut-capitole.fr/blacklists/index_en.php) and modify them
# to use in conjuntion with Squid or squidGuard.
#
# Copyright (C) 2018 Ramon Roman Castro <ramonromancastro@gmail.com>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.

This file has been truncated. show original

mumpitz · 3 May 2022 07:49

Unbelievable, the automatic update did not work again tonight and the manually executed this morning again with error message, what is going on there please? There were no such problems in the past, it worked smoothly. And now only cabbage and turnips, everywhere on strike the system and no solutions in sight.

07:03:19 installpackage[urlfilter]: URL filter blacklist - ERROR: Not a valid URL filter blacklist

01:33:14 installpackage[urlfilter]: URL filter blacklist - ERROR: Not a valid URL filter blacklist

xperimental · 3 May 2022 21:34

Same here. Since Shalla is gone → last update 240 days ago.

Firewall outgoing opened ports: FTP-control + FTP-data.

Edit: changed the firewall setting to allow all outgoing for testing purpose. Situation unchanged… but wait the RPI seems to be a little bit slow. It works if all outgoing connections from the firewall are allowed. In my case ftp is a really bad solution. I’m used to open only the ports I need. That’s also for the firewall itself, but with ftp the port for the data transfer is unknows for a static port firewall definition and can’t be definded/opened.

Toulouse must not be used with the firewall setting “outgoing” set to “blocked”.

If I trigger the update I can see established connections to france via ftp, but the url filter status doesn’t change / update.

This is to all of my 3 firewalls here.

23:27:10 installpackage[urlfilter]: URL filter blacklist - ERROR: Not a valid URL filter blacklist

trish · 4 May 2022 05:56

Yes I had the same error yesterday
installpackage[urlfilter]: URL filter blacklist - ERROR: Not a valid URL filter blacklist

but today the update completed

mumpitz · 25 June 2022 12:51

Yes from time to time the update works, but if you had some bad weeks the last update is 3 weeks old or longer, this isn’t optimal! Today I get an update, but before it takes several days.

trish · 23 July 2022 01:33

Yes it is bad, not working for a week for me.,
maybe we let it update at a different time, If the server is in France, try to schedule the update around 1-4AM CET.

I will try to find the file where to change it.

bonnietwin · 23 July 2022 08:06

Apparently the University of Toulouse ftp server is sometimes having problems and is unavailable. A bug was raised against this to change from the ftp server to the http server, which seemed to be more consistently available and this change will be in CU 170.

https://bugzilla.ipfire.org/show_bug.cgi?id=12891

Maybe this will help.

I would not expect changing the time when you download from them to make much of a difference as the servers should be operating 24/7 unless there is a problem.

The last update to the ftp server was done at 19:21 on 22nd July (likely to be local time)
The last update to the http server was done at 21:21 on 22nd July.

xperimental · 23 July 2022 17:39

This will be great, because I can set the firewall setting “outgoing” to “blocked” again.

mumpitz · 24 July 2022 23:08

Yes, great news!

trish · 17 August 2022 01:49

The U. Toulouse list is unreachable for long time.

Automatic blacklist update [ Last successful blacklist update was 32 days ago ]

I am trying to use a list with a good availability e.g.
https://urlhaus.abuse.ch/downloads/csv_online/

They allow update interval of 5 minutes or more. So once a day shouldn’t be an issue
But I am still getting the same error:

installpackage[urlfilter]: 	URL filter blacklist - ERROR: Not a valid URL filter blacklist

Or even a mirror of the University of Toulouse Blacklist:

installpackage[urlfilter]: 	URL filter blacklist - ERROR: Not a valid URL filter blacklist