Blocked URLs not working

Hi, this is my first post and I’m new to IPFire. Please me gentle with me.

I have the Proxy Server working nicely, I have created a custom blacklist containing domains e.g. euro-millions-dot-com etc. Those domains are blocked as expected.

However, I’d also like to block certain URLs without blocking the domain. e.g. block bbc-dot-co-dot-uk/sport but still allow users to visit the bbc website. So, this will allow my users to land on the BBC website but get blocked to the sport section.

Under Network - URL Filter - Custom Blacklist
In the box for “Blocked domains”, I have the list of domains to block (e.g. somedomain-dot-com) on the left hand side of the web GUI.
On the right hand side, in the box for “Blocked URLs”, I have bbc-dot-co-dot-uk/sport as the blocked URL.

However, this does not seem to work as I can still access the sport section. I’ve tried with and without the “www.” prefix and also tried an extra slash at the end of sport/

The proxy is not running under transparent mode.

I am running Core Update 156

What am I missing?

Thanks in advance.

Sorry for all the -dot- entries but I can’t post links as I’m a newbie

Hi @teamdsrdaz, you are wellcome.

Have you configured proxy config in web browser client?.

If you not configure Clients, traffic no go across non-transparent proxy.

Try and say us something.

1 Like

Hi Roberto, many thanks for taking the time to reply. I’m not 100% sure what you’re trying to tell me but I’m guessing you feel that my clients are not using the Proxy server. If that’s the case then I can confirm that other blocks (using domain only - i.e. something-dot-com) are working fine. The blacklists are also working so I know the proxy is being used ( I also have logs of all web traffic). The problem I have is when attempting to block a URL rather than a domain.

Thanks again.

Hi @teamdsrdaz.

I have also tried to get it to work and have been unsuccessful. In my case, I have carried out the tests with: https://servicios.elpais.com/el-tiempo/

I have configured the Firefox Proxy, although it detects (activating the Squid Log) requests to 443, not for URLs.


IPFire proxy log
Fecha: Junio 15, 2021
Source IP: ALL
Username: ALL
Ignore filter: [.](gif|jpeg|jpg|png|css|js)$

13:10:52 10.254.0.89 - 10.254.0.1:444
13:10:49 10.254.0.89 - 10.254.0.1:444
13:09:19 10.254.0.89 - srv00.epimg.net:443
13:09:19 10.254.0.89 - maps.google.com:443
13:09:19 10.254.0.89 - maps.gstatic.com:443
13:09:19 10.254.0.89 - maps.googleapis.com:443
13:09:18 10.254.0.89 - servicios.elpais.com:443
13:09:18 10.254.0.89 - ep00.epimg.net:443
13:09:18 10.254.0.89 - sdk.privacy-center.org:443
13:09:18 10.254.0.89 - arcsubscriptions.elpais.com:443
13:09:18 10.254.0.89 - playertop.elpais.com:443
13:08:53 10.254.0.89 - http://ocsp.pki.goog/gts1o1core
13:08:27 10.254.0.89 - safebrowsing.googleapis.com:443
13:07:26 10.254.0.89 - 10.254.0.1:444
13:06:40 10.254.0.89 - 10.254.0.1:444
13:06:27 10.254.0.89 - community.ipfire.org:443
13:06:27 10.254.0.89 - srv00.epimg.net:443
13:06:27 10.254.0.89 - servicios.elpais.com:443
13:06:27 10.254.0.89 - www.google.com:443
13:06:27 10.254.0.89 - id.google.com:443
13:06:27 10.254.0.89 - elpais.com:443
13:06:27 10.254.0.89 - contributor.google.com:443
13:06:27 10.254.0.89 - imagenes.elpais.com:443
13:06:27 10.254.0.89 - www.google.com:443
13:06:27 10.254.0.89 - maps.googleapis.com:443
13:06:27 10.254.0.89 - 10.254.0.1:444
13:06:27 10.254.0.89 - incoming.telemetry.mozilla.org:443
13:06:27 10.254.0.89 - fonts.gstatic.com:443
13:06:26 10.254.0.89 - error:transaction-end-before-headers
13:06:26 10.254.0.89 - maps.google.com:443
13:06:26 10.254.0.89 - maps.google.com:443
13:06:25 10.254.0.89 - sdk.privacy-center.org:443
13:06:25 10.254.0.89 - sdk.privacy-center.org:443
13:06:25 10.254.0.89 - maps.google.com:443
13:06:25 10.254.0.89 - srv00.epimg.net:443
13:06:25 10.254.0.89 - srv00.epimg.net:443
13:06:25 10.254.0.89 - ep00.epimg.net:443
13:06:25 10.254.0.89 - 10.254.0.1:444
13:06:19 10.254.0.89 - static.elpais.com:443
13:06:19 10.254.0.89 - static.elpais.com:443
13:06:18 10.254.0.89 - static.elpais.com:443
13:06:18 10.254.0.89 - safebrowsing.googleapis.com:443
13:06:18 10.254.0.89 - error:transaction-end-before-headers
13:06:18 10.254.0.89 - static.elpais.com:443
13:06:17 10.254.0.89 - error:transaction-end-before-headers
13:06:17 10.254.0.89 - error:transaction-end-before-headers
13:06:07 10.254.0.89 - error:transaction-end-before-headers
13:05:54 10.254.0.89 - community.ipfire.org:443
13:05:54 10.254.0.89 - maps.gstatic.com:443
13:05:54 10.254.0.89 - maps.google.com:443
13:05:54 10.254.0.89 - contributor.google.com:443
13:05:54 10.254.0.89 - playertop.elpais.com:443
13:05:54 10.254.0.89 - srv00.epimg.net:443
13:05:54 10.254.0.89 - static.elpais.com:443
13:05:54 10.254.0.89 - encrypted-tbn0.gstatic.com:443
13:05:54 10.254.0.89 - arcsubscriptions.elpais.com:443
13:05:54 10.254.0.89 - maps.googleapis.com:443
13:05:54 10.254.0.89 - ep00.epimg.net:443
13:05:54 10.254.0.89 - sdk.privacy-center.org:443
13:05:54 10.254.0.89 - apis.google.com:443
13:05:54 10.254.0.89 - incoming.telemetry.mozilla.org:443
13:05:54 10.254.0.89 - 10.254.0.1:444
13:05:54 10.254.0.89 - incoming.telemetry.mozilla.org:443
13:05:54 10.254.0.89 - www.gstatic.com:443
13:05:54 10.254.0.89 - fonts.googleapis.com:443
13:05:54 10.254.0.89 - id.google.com:443
13:05:54 10.254.0.89 - www.google.com:443
13:05:54 10.254.0.89 - incoming.telemetry.mozilla.org:443
13:05:54 10.254.0.89 - incoming.telemetry.mozilla.org:443
13:05:54 10.254.0.89 - incoming.telemetry.mozilla.org:443
13:05:54 10.254.0.89 - incoming.telemetry.mozilla.org:443
13:05:54 10.254.0.89 - servicios.elpais.com:443
13:05:54 10.254.0.89 - 10.254.0.1:444
13:05:54 10.254.0.89 - www.google.com:443
13:05:54 10.254.0.89 - fonts.gstatic.com:443
13:05:38 10.254.0.89 - incoming.telemetry.mozilla.org:443
13:04:53 10.254.0.89 - 10.254.0.1:444
13:04:25 10.254.0.89 - api.accounts.firefox.com:443
13:04:09 10.254.0.89 - imagenes.elpais.com:443
13:04:09 10.254.0.89 - fonts.googleapis.com:443
13:04:07 10.254.0.89 - elpais.com:443
13:04:07 10.254.0.89 - assets.adobedtm.com:443
13:03:55 10.254.0.89 - 10.254.0.1:444
13:03:52 10.254.0.89 - srv00.epimg.net:443
13:03:52 10.254.0.89 - srv00.epimg.net:443
13:03:52 10.254.0.89 - srv00.epimg.net:443
13:03:52 10.254.0.89 - srv00.epimg.net:443
13:03:52 10.254.0.89 - srv00.epimg.net:443
13:03:50 10.254.0.89 - 10.254.0.1:444
13:03:46 10.254.0.89 - static.elpais.com:443
13:03:46 10.254.0.89 - static.elpais.com:443
13:03:46 10.254.0.89 - static.elpais.com:443
13:03:46 10.254.0.89 - imagenes.elpais.com:443
13:03:45 10.254.0.89 - imagenes.elpais.com:443
13:03:45 10.254.0.89 - ep00.epimg.net:443
13:03:44 10.254.0.89 - www.gstatic.com:443
13:03:44 10.254.0.89 - www.gstatic.com:443
13:03:44 10.254.0.89 - www.gstatic.com:443
13:03:43 10.254.0.89 - www.google.com:443
13:03:39 10.254.0.89 - 10.254.0.1:444
13:03:39 10.254.0.89 - 10.254.0.1:444
13:03:27 10.254.0.89 - http://detectportal.firefox.com/success.txt?
13:03:27 10.254.0.89 - http://detectportal.firefox.com/success.txt?
13:03:27 10.254.0.89 - http://detectportal.firefox.com/canonical.html
13:03:27 10.254.0.89 - 10.254.0.1:444
13:03:26 10.254.0.89 - http://detectportal.firefox.com/success.txt?
13:03:26 10.254.0.89 - http://detectportal.firefox.com/success.txt?
13:03:26 10.254.0.89 - http://detectportal.firefox.com/canonical.html
13:03:26 10.254.0.89 - 10.254.0.1:444
13:03:24 10.254.0.89 - http://detectportal.firefox.com/success.txt?
13:03:24 10.254.0.89 - http://detectportal.firefox.com/success.txt?
13:03:24 10.254.0.89 - http://detectportal.firefox.com/canonical.html
13:03:24 10.254.0.89 - 10.254.0.1:444
13:03:23 10.254.0.89 - maps.googleapis.com:443
13:03:23 10.254.0.89 - community.ipfire.org:443
13:03:23 10.254.0.89 - srv00.epimg.net:443
13:03:23 10.254.0.89 - 10.254.0.1:444
13:03:23 10.254.0.89 - servicios.elpais.com:443
13:03:23 10.254.0.89 - sdk.privacy-center.org:443
13:03:12 10.254.0.89 - 10.254.0.1:444
13:03:08 10.254.0.89 - error:transaction-end-before-headers
13:03:07 10.254.0.89 - maps.google.com:443
13:03:07 10.254.0.89 - 10.254.0.1:444
13:03:05 10.254.0.89 - error:transaction-end-before-headers
13:03:04 10.254.0.89 - error:transaction-end-before-headers
13:03:04 10.254.0.89 - error:transaction-end-before-headers
13:03:04 10.254.0.89 - error:transaction-end-before-headers
13:03:03 10.254.0.89 - ep00.epimg.net:443
13:03:03 10.254.0.89 - maps.google.com:443

Firefox config:

IPFire URL-Filter config:

And I can access without problems.

Something fails or something escapes us. You will have to open a Bugzilla. Explain it a bit by referring to this thread.

https://bugzilla.ipfire.org/

Regards.

1 Like

https anyone?

Logged in Bugzilla.

Thanks Roberto for taking the time to replicate this issue.

You could try with the ‘custom expression’
(.*bbc\.co\.uk/sport/.*)
This RE should block the same addresses.

I remember we discussed this issue before. I can’t remember the specific thread, but I found such an expression in my URL filter config, a remainder of tests about this,

1 Like

Thank you Bernhard.

I copied and pasted your expression into the Web GUI and, ticked the “Enable customer expression list” then pressed the “Save and Restart” button but I can still access the sport section of the BBC website.

image

Regular Expressions seem complicated for my tiny brain. I’ve never used them and a google search showed that I am perhaps too stupid to work it out. Does the expression work for you?

Thanks again for your valued time.

Hello together,

you simply cannot block bbc.co.uk/sport. The reason is that you are using HTTPS which creates a secure connection between your browser and the web server. The proxy does not know which URL is being accessed and therefore cannot make a blocking decision.

You can only block based on domain names using the proxy.

1 Like

That’s my experience, meanwhile too.
I just traced the traffic for an web access to bbc.co.uk/sport. No packet contains this URL ( unencrypted ).

BTW: the regex is correct. I checked this also. If URLFilter would get this URL it would block it.

Thank you Michael (and again Bernhard). That makes sense to me although it’s quite frustrating.

I’ll not waste any more time on this but would like to thank everyone for their patience, their time and their input.

Never mind. Such an investigation gives also insights for the investgator. :wink: