IP Blocklist download errors

Security Intrusion Prevention
1

Hello,

I read the IP-Blocklist Error for TOR_EXIT thread but the problem I’m seeing doesn’t have the same cause.

The IPFire Log Summary regularly shows a high number of updates and often up to 10 failures:

The following block lists were updated:
    TOR_ALL: 94 Time(s)
    TOR_EXIT: 94 Time(s)

 The following errors were detected:
    <ERROR> Could not update TOR_ALL blocklist - Download error! : 2 Time(s)
    <ERROR> Could not update TOR_EXIT blocklist - Download error! : 2 Time(s)

(2 failures is rare, I used this example to show the 94 updates.)

When I attempt to curl the TOR list source I get:

 curl https://www.dan.me.uk/torlist/?exit
Umm... You can only fetch the data every 30 minutes - sorry.  It's pointless any faster as I only update every 30 minutes anyway.
If you keep trying to download this list too often, you may get blocked from accessing it completely.
(this is due to some people trying to download this list every minute!)

Dividing 24 hours by 94 I end up with 15 minutes (and change). Is my IPFire system trying to update blocklists too frequently? I’ve never touched this setting other than to enable it for a few blocklists (I recently re-added DSHIELD).

I see this in fcrontab:

# Update Lists for IP-based blocking every 15 minutes.
@ 15 [ -f "/var/ipfire/red/active" ] && /usr/local/bin/update-ipblocklists >/dev/null 2>&1

  • Could someone please confirm that this 15 minute update schedule is the default?
    EDIT: The other thread indicates it is - I understand why you’d want frequent updates, but it seems a bit frequent. Note the output from the curl command! I can’t think of anything else on my network which would be trying to pull the same lists.

  • Do you have any ideas for how I can troubleshoot this problem?
    It’s happening with all lists I enable. While I no longer monitor it, my internet connection has proven to be reliable so I have no reason to believe that is the cause.

Thank you!

2

see:

 cat /var/ipfire/ipblocklist/sources

it looks like it is set to one hour.

1 Like
3

Thanks for that reference, you’re right. I can confirm the same with mine.

Do I possibly have a stale fcrontab from some previous IPFire release?!
If I delete the fcrontab entry and then re-save the “IP Address Blocklists” page I imagine it should re-create any necessary entries.

4

right from the source:

5

Thanks. I’ve found it hard to find the source in the past.

Ok so this script runs every 15 minutes

# Update Lists for IP-based blocking every 15 minutes.
@ 15 [ -f "/var/ipfire/red/active" ] && /usr/local/bin/update-ipblocklists >/dev/null 2>&1

and presumably checks the sources file you listed earlier so that updates should only happen at the intervals set. I wonder why mine is actually attempting an update every 15 minutes for a list set to 30 minute updates.

6

That one I cannot answer… My 1st thought was curl https://www.dan.me.uk/torlist/?exit was run too many times and maybe you are on some sort of fail ban list. Try disabling the tor for an hour and then re-enable. see what happens.

7

Crontab (the source you provided) is set to run IP-Based blocklist every 15 min…

This should happen only every 1 hour…

8

fcron runs the IP Blocklist update script every 15 mins.

That script checks what the rate is from the blocklist info file for each list and then compares that with the last time that the url was accessed for that list.

If the rate time period (eg 1 hour) has not yest passed then IPFire will not try and access the url at all. You will then see the message

Skipping XXXXXX blocklist - Too frequent update attempts!

but nothing has been attempted.

If the rate for a blocklist is set at 1 day then you will get the skipping message every 15 mins until a full 24 hours has passed, and only then will the url be accessed.

An alternative skipping message that you might see is

Skipping XXXXXX blocklist - It has not been modified!

In that case it will have checked the url (after the appropriate rate time period has passed) and then it checks the downloaded list and does not replace the existing list if it has not been modified in the intervening period since the last actual update.

1 Like
9

If you are getting the download error with every blocklist that is enabled, that indicates some global issue causing the problem.
The update-ipblocklists perl code depends on the time being correct. Can you confirm that the time is correct on your IPFire?

What are the contents of the
/var/ipfire/ipblocklist/modified
file?

10

Hello,

Sorry it turns out that the problem is only with the TOR lists being blocked.

They appear to be running every 15 minutes, despite the way IPFire is configured to run them every hour based on /var/ipfire/ipblocklist/sources as Jon noted.

While I have a fairly old IPFire install, I’ve not modified any of these scripts or settings.

Here’s yesterday’s report:

IP Blocklist

 The following block lists were updated:
    DSHIELD: 24 Time(s)
    TOR_ALL: 91 Time(s)
    TOR_EXIT: 91 Time(s)

 The following errors were detected:
    <ERROR> Could not update TOR_ALL blocklist - Download error! : 5 Time(s)
    <ERROR> Could not update TOR_EXIT blocklist - Download error! : 5 Time(s)
11

Hello, to answer your question:

# cat /var/ipfire/ipblocklist/modified
TOR_ALL=1700958688
FEODO_RECOMMENDED=1694168707
CIARMY=1663923841
DSHIELD=1701987601
TOR_EXIT=1700958688

Currently only TOR and DSHIELD options are enabled.

12

The last time that TOR_ALL and TOR_EXIT were updated was 12 days ago on 26th November based on conversion of the epoch number 1700958688 to human readable date.

In your first post you show the message you get when trying with the curl command.

I would suggest that you disable the TOR_ALL & TOR_EXIT lists and then wait at least 30 minutes, say an hour or longer, and then try the same curl command again.

If you still get the same message then the likelihood is that too many attempts were made in too short a time and your IP has been blocked by the TOR blacklist site.

They are being run every 15 minutes because the last download occurred 12 days ago on 26th November.

Could you also show the results from running
less /var/log/messages | grep TOR_EXIT

1 Like
13

It would be good to also show the results of

zless /var/log/messages.12.gz | grep TOR_EXIT
and
zless /var/log/messages.13.gz | grep TOR_EXIT

These two should bracket the point in time at which your last successful download of the TOR_EXIT ipblocklist was achieved so we can see what was done just before and after the last successful update.

2 Likes
14

Thank you @bonnietwin !

I have had the TOR lists disabled since my last post, so I did a curl today and it succeeded. I then re-enabled the TOR lists, so we will see if the problem happens again. My IPFire instance is attempting to update the TOR list every 15 minutes, which is strange, so the problem will happen at some point.

Going back through the historic logs I see that the ipblocklist script correctly only attempted an update every hour:

Sep  9 22:13:14 gateway ipblocklist: <INFO> Successfully updated TOR_EXIT blocklist. 
Sep  9 22:28:08 gateway ipblocklist: <INFO> Skipping TOR_EXIT blocklist - Too frequent update attempts! 
Sep  9 22:43:08 gateway ipblocklist: <INFO> Skipping TOR_EXIT blocklist - Too frequent update attempts! 
Sep  9 22:58:08 gateway ipblocklist: <INFO> Skipping TOR_EXIT blocklist - Too frequent update attempts! 
Sep  9 23:13:08 gateway ipblocklist: <INFO> Skipping TOR_EXIT blocklist - Too frequent update attempts! 
Sep  9 23:28:14 gateway ipblocklist: <INFO> Successfully updated TOR_EXIT blocklist.

In more recent times things changed. A new pattern is now occurring where it is attempting too often (15 successes to every 1 failure):

Dec  2 02:54:52 gateway ipblocklist: <ERROR> Could not update TOR_EXIT blocklist - Download error! 
Dec  2 03:09:49 gateway ipblocklist: <INFO> Successfully updated TOR_EXIT blocklist. 
Dec  2 03:24:48 gateway ipblocklist: <INFO> Successfully updated TOR_EXIT blocklist. 
Dec  2 03:39:49 gateway ipblocklist: <INFO> Successfully updated TOR_EXIT blocklist. 
Dec  2 03:54:49 gateway ipblocklist: <INFO> Successfully updated TOR_EXIT blocklist. 
Dec  2 04:09:49 gateway ipblocklist: <INFO> Successfully updated TOR_EXIT blocklist. 
Dec  2 04:24:50 gateway ipblocklist: <INFO> Successfully updated TOR_EXIT blocklist. 
Dec  2 04:39:52 gateway ipblocklist: <INFO> Successfully updated TOR_EXIT blocklist. 
Dec  2 04:54:52 gateway ipblocklist: <INFO> Successfully updated TOR_EXIT blocklist. 
Dec  2 05:09:55 gateway ipblocklist: <INFO> Successfully updated TOR_EXIT blocklist. 
Dec  2 05:24:55 gateway ipblocklist: <INFO> Successfully updated TOR_EXIT blocklist. 
Dec  2 05:39:57 gateway ipblocklist: <INFO> Successfully updated TOR_EXIT blocklist. 
Dec  2 05:54:58 gateway ipblocklist: <INFO> Successfully updated TOR_EXIT blocklist. 
Dec  2 06:09:58 gateway ipblocklist: <INFO> Successfully updated TOR_EXIT blocklist. 
Dec  2 06:24:58 gateway ipblocklist: <INFO> Successfully updated TOR_EXIT blocklist. 
Dec  2 06:39:59 gateway ipblocklist: <INFO> Successfully updated TOR_EXIT blocklist. 
Dec  2 06:54:53 gateway ipblocklist: <ERROR> Could not update TOR_EXIT blocklist - Download error!

So perhaps the source site is more lenient about blocking people who are too frequent. My system is clearly trying to update more often than their 30 minute rate limit! I’m unsure why as I have not altered the Blocklist configuration in any way.

Thank you again!

15

How do you suggest we troubleshoot why this is happening so often please?

At least it is updating now!

# grep TOR /var/ipfire/ipblocklist/modified
TOR_ALL=1702364493
TOR_EXIT=1702362691

When I run those through date -d @<string> they are the current time.

16

As far as troubleshooting this, the only thing I can think of is that either the cgi or the pl code has become corrupted/modified in some way or the code as written allows what you experience to happen under certain circumstances.

The second situation would require someone to go through the ipblocklist.cgi and ipblocklist-functions.pl code to see how this could happen.

The first situation we can test by comparing the b2sum hashes for the code on your system with what comes from the same files in the git repo.

Run the following two commands and provide the hash responses they provide.

b2sum /srv/web/ipfire/cgi-bin/ipblocklist.cgi
b2sum /var/ipfire/ipblocklist-functions.pl
17

Thanks, so here are the hashes:

d5286fc16847c263995f2edda92ed996b07975c1de60211fc2092411d494e6d6014385279a098deb2980724bc0ffc3f00e4e08a965a61ccdb9902f5432f2827e  /srv/web/ipfire/cgi-bin/ipblocklist.cgi
72647dade0e65f18f9cc43578e43cd04d3658f7a5ff3febd68fc77d0b21e0bb8f09102ebf66212a50ac2033027a440c2a0d08fce27d69e9601dc6994ac2a0162  /var/ipfire/ipblocklist-functions.pl

I’ve gotta ask, why use blakes2? This isn’t a password hash, just a checksum for file content verification. People tend to use sha1 for this purpose most often, but even md5 would be adequate.

18

The b2sum hashes match for both files against my production system and against the same files in the git repository. So no corruption or modification of the involved files, so that is good.

I use blake2 hash because I am generally using them anyway for all the package update patches that I submit for IPFire-2.x. So it just gets typed automatically by myself. Yes sha1 would have been fine as well bu there is nothing wrong with it being blake2.

I think I have figured out what is happening and this looks like it would only be a problem with the TOR lists.

If IPFire is within 15 minutes of trying another download from the TOR lists and a manual download is carried out then a 15 minute timer is started for your IP.

Clicking the TOR_EXIT link in the IP Block Lists page opens up the TOR_EXIT list and is considered to be a download.

So TOR_EXIT now has a 15 min timer on your IP but with less than 15 minutes when the next hourly download would be attempted by the IP Blocklist code based on the Epoch time for the last download that is stored in /var/ipfire/ipblocklist/modified

When the IPFire Blocklist code tries to download the TOR_EXIT list it will fail due to the previous manual attempt by clicking the web link on the page. Additionally the TOR_EXIT website now resets the 15 minute timer for your IP and you are now in the mode that it will never successfully download again unless you disable the TOR_EXIT list for at least 15 minutes.

The same 15 minute timer applies to both TOR_EXIT and TOR_ALL so once you have clicked on the weblink for TOR_EXIT you also cannot access the TOR_ALL list.

So having both TOR_EXIT and TOR_ALL enabled at the same time would also be another way to trigger the problem, even without clicking the weblink in the IP Blocklist WUI page. This is because after the blocklist code has updated say TOR_EXIT, if it then tries to update TOR_ALL the 15 minute timer will be reset and the download will fail.

One solution would be to change the weblink to the next level up but this won’t fix it in all cases because if the user has clicked on the link and sees a whole list of different blocklists but is interested in the TOR Nodes list they would likely press that link on the dan(dot)me(dot)uk webpage and that will show the blocklist and start the 15 minute timer.

The first thing that I will do is to modify the code so that if TOR_ALL is selected then TOR_EXIT cannot be selected and vice versa as TOR_ALL includes TOR_EXIT in it. This at least means that the problem will not automatically occur if you select both of the TOR lists.

However there also needs to be some additional code so that if that download failure message is found from the TOR list, because the weblink was clicked and the 15 min timer started, that the IP Blocklist does not try to download again from the TOR list for at least say 30 minutes.

This issue only happens with the TOR list due to the fact that all the other IP Blocklist weblinks point to the blocklist website, not the blocklist itself.

dan (dot)me(dot)uk web page for the TOR lists also shows the list itself on the page and that is considered a download because it has ended up on your PC’s browser.

I will think about the best way to resolve this for the TOR lists.

4 Likes
19

Could you put in a delay in and add pop-up or note on WUI.
Stating could take 30 minutes to take affect.
Just a possible fix?

20

Thanks, I didn’t realise that TOR_ALL included TOR_EXIT!

I can verify that this now works:

The following block lists were updated:
    DSHIELD: 24 Time(s)
    TOR_ALL: 24 Time(s)
1 Like