Proxy inconsistencies

Please help, I’m suffering a mental knot.
Webproxy on green is in conventional, blue transparent. DNS is send all through firewall by the two rules created as wiki says. Now… I did two more rules for passing all http/s traffic from blue/green through firewall - in the same fashion (redirecting). I found in community another solution (blocking http/s from green/ blue to red), but it didn’t work for me and to my liitle knowledge I don’t see how this could work… it’s data, not water.
Now what’s happening in proxy logs is a bit odd, traffic is registered unevenly… For the machines on green (mostly macs) proxy logs/ reports are ok and the url filter is pretty productive. A linux had some glitches but in the end logs normalised. But on blue no way to have proper logs from a bunch of androids and iPads. Sarg also miss web browsing traffic, it shows only some (some) traffic generated by several apps (whatsapp, antivirus)… very few. On the same devices url filtering seems to work though - but again, very few entries on log. I use a hagezi pro list and several hagezi native lists for various brands.
Now, passing the linux from green to blue for testing its logs remains consistent, the traffic in sarg is real, complete, doesn’t miss anything, it works equally consistent.
What do I miss? Why proxy on blue works so unevenly? I checked the config on those devices, but it’s not the case, things are OK… it is transparent so no need to config anything.
My main aim is a proper filtering on all machines for telemetrics/ ads and so… not to find for example a crapy tiktok add nested in shopping apps. Do I need to change firewall rules for http/s? I am lost…

firewall rules for http/s
standard networks: blue/ green
nat: port forwarding > firewall - automatic
destination: firewall - all
protocol: service groups - http/s (created for this purpose)

Thank you.

I just happened to read your post now. I hope I can still be of help. You said that on BLUE you enabled transparency, but be aware that transparency works ONLY for port 80 (HTTP).
Transparency DOES NOT WORK FOR HTTPS. For HTTPS, you must explicitly configure the proxy (IPFire’s BLUE IP) on all clients.
Maybe this could be your issue.

Phones love to use DoH. DNS over https.

You can try to block DoH

or try to set phone to use your DNS.

Definitely use your own name server with unboud plus filters.

For IPFire:

Use ISP-assigned DNS servers and disable them.

dns-config

DNS Firewall - enable what you want to block, e.g., advertising, malware, phishing.

Note, DNS Firewall consumes a lot of RAM.

Alternatively, if the IPFire machine has limited RAM, you could set up a dedicated Pi-hole server for domain blocking. That’s what I did. I connected the Pi-hole server to the IPFire Orange network, and I can confirm that it works very well. However, the downside here as well is that you need to set up another server.

The disadvantage of using DNS-based blocking instead of the IPFire proxy is that DoH can easily bypass these restrictions. By using Squid as a proxy and forcing all HTTP/HTTPS traffic through it, domain blocking becomes significantly harder to bypass, provided that firewall rules prevent direct Internet access and are correctly configured in IPFire.

You can also redirect all DNS request through IPFire.
A DOH (HTTP/S) request needs a DNS request ( name resultion of DOH server ) also. These request can be blocked by DOH blocking lists for unbound.
For DOH requests gong straight with HTTP(S) requests, a block in the proxy can supplement this.

The only solution is to block all communication to the internet and route all traffic through the IPFire proxy. The problem is that only http/https communication will work effectively.

This is true, because Squid is a web proxy. :wink:

Yes, I was getting confused too.

Classic DNS uses port 53.
DoH (DNS over HTTPS) uses HTTPS, therefore port 443.

So:

If I configure a rule on IPFire that blocks port 53 to the outside, allowing it only to the local DNS (for example Pi-hole), I can force clients to use Pi-hole as their DNS resolver.

However, this does not block DoH, because it goes through port 443 along with normal HTTPS traffic. If I blocked DoH in a “brute-force” way (i.e., by blocking port 443), I would also prevent normal web browsing.

To control DoH as well, more advanced techniques are required (e.g., proxy, HTTPS inspection, blocking specific DoH providers).

If I use Squid with forced proxy and filtering, then most of the control shifts there, and Pi-hole becomes less central (or redundant, depending on the configuration).

Is that correct?

Everything is correct, it’s a consequence

Using your own name server allows you to have a better picture of what’s happening on your network. For the server you’re using, you’ll have a wealth of information in its statistics.

Yes, DOH uses HTTPS. But there is a web server that does the name resolution job. And many of these are known and can be blocked in the name resolution.
A web access to doh.example.com is accomplished in several steps (simplified)

  1. resolve doh.example.com
  2. establish HTTP(S) connection to this IP
  3. process web task ( in this case name resolution )
  4. terminate connection

To block DOH, this can be done either in step 1 ( unbound ) or step 2 ( Squid with SquidGuard aka URLfilter ).

Well… this is an old topic I opened in search for help (or rather to make some light) a while ago. Meanwhile IPF went through an update that somehow addressed in part the issuse here. The DBL list curated by IPF team is brilliant, and more over is implemented on suricata level which is a bit overkill, I guess, but tunning a bit the blocklists brings some peace of mind. Right now the funny ocult taffic (more than regular scamware, the call home traffic generated by some cheap devices I have in the network) is not a bother anymore. I stick with the ermetic policy of IPF and don’t use (wouldn’t use) a DNS filtering solution as Pihole. Beside suricata the Location block list is a minimal but usefull saftey measure that works well. An ntop side machine is vigilating the traffic and all in all with nomore then the proxy, the IP Blocklists working on DNS level and IPS, I havent’t noticed any worrysome data/ traffic in a while.

Anyway, I really love the way IPF keeps things minimal. It’s a hellava jod the team is doing, in my opinion best opensource solution out there. It took me some to learn things but now I don’t feel I miss something. As long as DNS remains strong, the traffic is forced through proxy and IPS is live and dandy, the rest, at least for me, is not anymore such a reason for preocupation.

Anyway, thanks for answering and care.
O… I respond through mail, as IPF community platform is keep giving an error on loading the login page.
Thank you guys.

In further development DBL will be implemented in unbound, also. The mechanism is called RPZ.

I just logged out from the forum and from IPFire People and then logged back in and I got the login page in both cases. So that indicates that the authentication server is working.

I am not sure what that problem you are seeing is caused by. Maybe try clearing your browser cache to see if that helps.

This issue doesn’t appear to be related to the user’s computer.

After entering the username/password, I get:

“Oops - We Are Sorry, But Something Went Wrong
Error 500”

This doesn’t happen all the time.

In the last few weeks we have been having some bot systems making huge numbers of requests on our web site servers.

https://community.ipfire.org/t/www-ipfire-org-is-a-bit-slower-today/15657

Maybe the restrictions that were added are giving some restrictions for some browsers.

What browser are the two of you using. I was using Firefox and didn’t see the issue.

More problematically is if the problem is inconsistent as that will make it much harder to track down.

What browser are the two of you using. I was using Firefox and didn’t see the issue.

Firefox

I also had the same issues yesterday; clearing the Firefox browser cache didn’t help. It also didn’t work from other IPs or different ISPs. The problems were brief and temporary. I waited, and everything started working again.