Could not download blocklist - A download error occured

Security
1

Where can I find a detailed log ? /var/log/messages doesn’t show anything about this error.

I am trying to add a new source for IP Blocklist
and getting this error

Could not download blocklist - A download error occured.

I tried to add this to the source file in
/var/ipfire/ipblocklist/

'THREATVIEW_IO_BL'        => { 'name'     => 'Threat Intelligence.io Malicious IP Blocklist for known Bad IP addresses',
                                    'url'      => 'https://threatview.io/Downloads/IP-High-Confidence-Feed.txt',
                                    'info'     => 'https://threatview.io/#services',
                                    'parser'   => 'ip-or-net-list',
                                    'rate'     => '6h',
                                    'category' => 'reputation' },

and for the grammar geeks yes there is a misspelling in the error message above :innocent:

Here is a preview of the Blocklist TXT file

100.22.4.151
101.0.107.84
101.132.27.40
101.200.156.217
101.200.63.187
101.201.35.96
101.201.68.58
101.201.69.42
101.32.207.185
101.32.34.196
101.33.244.132
101.35.197.155
101.35.21.69
101.36.114.252
101.36.114.91

I checked with ipset list and not all IP show up so this might be a good list to add.

1 Like
2

The download failure message means that the download process failed 5 times.

There might be something different about their setup of the lists that is making the get request from the LWP::UserAgent function in the perl libwww module fail to download.

The same problem happened with another list from threatview. A third one I tried was successfully downloaded. So whatever the difference is it is not on all the lists they provide.

The version of libwww we currently have is from 2022 and there is a version from 2025.
An update of the whole perl set that we have, is on my list of things to do but currently I am still struggling with updating the python set we have.

1 Like
3

Thank you for using Threatview Feeds. We saw a message of feeds not being accessible and wanted to jump in to assist. May we know is this resolved ? It could also be because we made some recent changes in cloudflare configurations to avoid bots. Happy to support to ensure seamless integration of the feeds. Please feel free to reach out at feeds[@]threatview.io.

Thanks
Team Threatview.io

4 Likes
4

I tested out the latest version of perl-libwww, updating it from version 6.62 to 6.78 and installing the IPFire iso that I created.

The same download problem occurred. So the issue with the threatview list is not to do with the version of the perl module used for the download process.

I will try and run the perl-libwww download commands manually to see if there is some message provided that gives a clue what is going wrong.

However I will also say that currently I would not be looking at adding this ip blocklist into IPFire as I added in 4 lists around 6 months ago, which have now been removed without any notification or explanation.

The 3coresec people never responded back to my email, 2 weeks now, about why their lists were completely removed and the abuse.ch list was just left with a note in it that the list was deprecated since 3rd Jan 2025.

Before proposing a patch to add any new list in I would want some view that any new list was intended to be a long term support and not just to disappear without some warning and notification.

2 Likes
5

Running the code we use manually is getting back an error code 1010 which is coming from cloudflare and the code number according to my searching is described as “Access Denied: Bad Bot”.

I suspect that cloudflare does not like the default user agent used by the perl LWP::UserAgent module that is used for the downloading.

The default value for the agent with the code is
libwww-perl/#.###
where #.### is substituted with the version number of the libwww-perl library.

This is working fine for all other lists that IPFire uses.

I tried searching for which user agent string cloudflare would accept but so far I have not been successful in finding it.

EDIT:
It seems that Cloudflare provide the ability to block user agent strings but this is specified by each user.
So it looks like you (threatview.io) are blocking the default user agent for the perl LWP::UserAgent module. So you need to either accept the default LWP::UserAgent agent string or specify what user string should be used and if we ever add the Threatview list to IPFire then we will need to look at changing the coding used for downloading the ip block lists from threatview.

EDIT2:
I have just been able to successfully download the ip block list by setting the agent in LWP::UserAgent to a blank setting
agent => (""),

3 Likes
6

Thank you for taking out time to test. I have added the “libwww-perl” as a user agent string to skip WAF rules. Hope it works now. Let me know if you have time to test :slight_smile:

Thanks alot again

7

Thanks @threatview_io

I have just tested it but unfortunately it is still not working.

With a little investigation I have found that I can put anything in the user agent string, such as “freddy”, and it will work and also “libwww_perl” but just not “libwww-perl”.

So I thought it was the use of the - but that is not the case as “freddy-thomas” as a user string was accepted. Also “freddy-perl” and libwww-thomas". It seems to just be “libwww-perl” that is not accepted.

Changing one letter to give “libwww-perf” was accepted but “libwww-perlf” was not so it seems that the exact string “libwww-perl” is looked for and not accepted if it is anywhere in the user agent string.
“libwww3-perl” is accepted as that breaks up the libwww-perl string.

EDIT:
Clarified further. It is not acceptable to have “libwww-perl” at the start of the user agent string. If “dlibwww-perl” is used as a user agent string then it is accepted and anything further can be added at the end of the string and that will also be accepted.
So “libwww-perl” at the start of the user agent string is not allowed.

3 Likes
8

Thanks for testing, I can see those events in the Cloudflare logs. The events indicate while the user-agent test was skipped, the IP were blocked because of browser integrity checks, so I have now skipped that as well. Please try once again when you have a moment.

I have also did a curl with “curl -A “libwww-perl” -O “https://threatview.io/Downloads/High-Confidence-CobaltStrike-C2%20-Feeds.txt”” worked fine, CF skipped the checks.

Please let me know if it works for you now ?

Additionally wanted to mention that - we have been running the free community feeds since 2019 and dont have any plans per say to shut the service, it will be great if we can serve the IPfire Community :slight_smile:

Thank you :slight_smile:

1 Like
9

Just tried again manually with the default user agent string and everything worked fine.

I then also tried it with the IPFire ipblocklist using the standard code and it worked and downloaded the file.

Thanks very much for your help to resolve this.

That is very good to know.
We understand that things can change but with the other lists it was the fact that we had to find out because the lists in one case just stopped existing and in the other case the text file list was still available but it just contained the message that the list had been deprecated 1.5 months earlier.

10

Gladly !! Thanks for your patience throughout the process.

11

Totally understand and agree.

12

Hi Peppe Tech,

The matter has been resolved, please feel free to add Threatview.io feeds :slight_smile:

Best Regards
Team Threatview.io

1 Like