After upgrading from 175 to 177 I encountered problems when trying to visit for e.g. amazon.de, the page wouldn’t load at all or only partial.
First trial was to disable the webproxy. After that amazon.de was loaded again as expected. After fiddling some time with the settings of the webproxy it seems that activating “Deny access to destinations hosted on selectively announced networks:” leads to that problem. I therefore disabled that option leaving just “Deny access to destinations hosted on fast flux setups:” enabled.
I can verify the issue with amazon.it, though the problem is intermittent. Conversely, the web search engine kagi.com consistently fails to load some of its resources. I suspect that both organizations might be employing selective BGP announcements for parts of their network. In the context of the BGP protocol, when a network chooses to announce its routes to only a select group of peers and not to the entire global Internet, it’s engaging in selective announcements. This is employed by bad actors, hence the rule in the proxy. I think here we have some false positive.
What I find strange is that this behavior of the proxy never emerged before. Is there anything in 177 but not in 176 that could trigger this block? EDIT: maybe this?
@ms Should I open a bug report for libloc? Is this a bug or it is a feature?
I think this behavious is not due to Squid, but libloc. As stated by @samspade this is triggered by Deny access to destinations hosted on selectively announced networks.
That geofeeds item has been added to libloc-0.9.17 which as an independent package has been released. That released version has not yet been added into IPFire. IPFire is still running on libloc-0.9.16 from the beginning of this year.
The libloc database is of course updated on a more regular basis so maybe something changed with one of those updates.
You should find out what libloc says about the urls you are having problems with. Are they shown up as selective announcements.
If you believe that libloc is flagging them as making selective announcements incorrectly, ie a false positive, then you should raise a bug on it.
you just destroyed my hypothesis. Well, maybe I never had problems before because kagi and amazon started to do selective announcement recently. I need to think how to handle this false positive. Maybe I will exclude from the proxy kagi and amazon. A direct access does not help, which is logical as the problem is not the cache but the libloc detection of partial announcement.
If they have started doing selective announcements then they should be notified that they have a problem so they can fix it as reputable websites should not be doing that.
If they are not doing selective announcements but the libloc database has them as doing so then an IPFire bug report to get the libloc database updated to remove the false positive would be in order.
I a bit lost on how to investigate libloc response. Any suggestion?
EDIT: this is what I see in cache.log. I will write to the location mailing list.
kagi:
Aug 09 12:19:05 squid-asnbl-helper[18771] WARN: Destination 'assets.kagi.com' resolves to IP addresses '2600:9000:223f:be00:f:ce53:f180:93a1' without corresponding ASN, probably selectively announced
Aug 09 12:19:05 squid-asnbl-helper[18771] WARN: Destination 'assets.kagi.com' resolves to IP addresses '13.226.34.77' without corresponding ASN, probably selectively announced
Aug 09 12:19:05 squid-asnbl-helper[18771] INFO: Denying access to destination 'assets.kagi.com' due to suspected selective announcements
amazon:
Aug 09 12:19:00 squid-asnbl-helper[18771] WARN: Destination 'm.media-amazon.com' resolves to IP addresses '2600:9000:223e:b800:1d:d7f6:39d2:2dc1' without corresponding ASN, probably selectively announced
Aug 09 12:19:00 squid-asnbl-helper[18771] INFO: Denying access to destination 'm.media-amazon.com' due to suspected selective announcements
I am sorry but that is where I run out of ideas on how to track it down further. Maybe this would be worth writing an email to the IPFire dev mailing list.
That would ensure that the people who can answer the question of finding if the websites are really selectively announcing or if there is a false positive, get to see the question.
I have found some more info.
The fast flux and selective announcements screening from the libloc database is executed by the asnbl-helper package that acts as an interface script to squid.
This package had a bug fix implemented in CU177.
This bug was raised because some users were getting segfaults closing squid down. This was tracked down to the asnbl-helper script needing some changes to match with what python3 was expecting.
The python3 interpreter sometimes detects the local return variable if it was a misused global variable (false detection). To solve it, the local variable has be declared before it’s first use and the use of the global ASNDB has to be slightly changed.
Maybe the fix has resulted in an unexpected side effect, although the bug reporter who created the fix patch has had it running on his system since March this year without any reported problems.
Just flagging it up as it is a change in CU177 in the packages that are involved in selective announcement identification.
It is a change I found in CU176 to CU177. That does not mean it is causing the problems you have experienced.
I suspect the best approach is to write to the general dev mailing list and ask for help on how to debug where the issue is coming from to identify if it is an IPFire bug or not.
We don’t have enough knowledge on how to track down the root cause of this problem so their help is needed.
So I don’t know if it would belong here…
I updated the backup ipfire from 175 to 177
but the same problem was with the direct installation of 177
doesn’t go web proxy in status information is webproxy : stop
web pages loading via https OK
web pages loading via http don’t work
just deactivate and reactivate the web proxy and the http protocol works
on the main ipfire mam 176 and I don’t have this problem
So after a few attempts it seems that it helped me to delete in the settings
Network based access control and after saving and restart
web proxy seems to be functional, i.e. functional http