Can not get non-Transparent web proxy to work

[jon]

I’ve been trying to get the non-Transparent web proxy setup and working. But no joy.

I am not sure if this is a Firewall Rules issue or a pac/wpad issue or a set-up issue. At the moment I am going down the Firewall Rules troubleshooting path.

I tried ip-mfg FR suggestion here:

Firewall rule for web-proxy

well, I use this rules in the outgoing traffic: Source: firewall (red) Destination: Standard networks (red) Protocol: Service Groups (Web-Proxy) and in the Service Groups: Web-Proxy (http 80 TCP; https 443 TCP) That’s all

And than I have no access to any web sites.

I tried Roberto’s FR #5:

Firewall rule for web-proxy

Hi @fstarter. After long tests, it works for me as follows: [imagen] In “Web Proxy Auto-Discovery Protocol (WPAD) / Proxy Auto-Config (PAC)” I have put this range to be able to access the devices that are in the RED range (Router, Printer, APs, etc …). [imagen] Although the “Option code 252 = text” does not work in Firefox, I put it for Chrome and Edge. [imagen] In “Host” I have defined the IPs of the devices that are in “RED”. [imagen] The device group in “RED” for the Firewall rul…

but that did not work for me. FYI - I do not have any devices in RED except for the Internet gateway.

If I turn off the Firewall Rule, then I have website access. (I only used one rule at a time)

So my question is: what Firewall Rule are you using to block Port 80 and Port 443?

Jon

[hvacguy (Shaun HVAC)]

@jon

I tried ip-mfg FR suggestion here:

Could you add a block rule ahead of this rule?
Like.
Green to red service group proxy 80,443 block.
Green to firewall port 80 to 3128
Green to firewall port 443 to 3128
?
Could this silently redirect https and http traffic?

[jon]

Hmmm - maybe… I’ll have to give it a try.

I think I found something that may work. I used the redirect (from the DNS/NTP redirect) and changed the ports from 53 &123 to 80 & 443.

Not sure how well this works…

[cfusco]

@jon This is the summary of this post; you can’t have everything working by just setting the firewall, that’s only when you use transparent proxy. You need to instruct any and all clients, automatically or manually to use the proxy, and whether you do it automatically or manually, it depends both on the OS and the client.

My understanding is that there are two separate issues:

A) correctly setup the squid proxy;
B) block at the firewall level any attempt to get on red from anyone in your network on the web ports.

IF you have both A and B, AND you manually set the browser of your laptop or phone to use http://ipfire.localdomain:800 and https://ipfire.localdomain:800 as proxies for port 80 and 443, THEN you must be able to browse internet, ELSE you have a miss-configured squid.

Another way to say the same thing is: IF you have A and B, UNLESS the clients are instructed to use the proxy, your clients wont access the web.

As far as I know, there are three methods to do so:

1. manual configuration of the client;
2. automatic configuration of the client by looking at the OS system settings; this works IF the OS is being manually configured, OR it is using proxy.pac, directly or by following the DHCP setting as suggested in the wiki; <— this is my system (OS manually configured), see below
3. automatic configuration by the client, using proxy.pac directly.

In my situation, for mobile devices, setting manually the proxy at the OS level in the WIFI configuration, pretty much works out of the box for any APP working on those devices except few specialized apps that do not handle a proxy server well.

Similarly but more tiresome, on MacOS I also had to set:

1. in the network configuration to use the proxy, and
2. in the client (Firefox), to use the proxy system settings.

This combination works well. Auto-discovery by using proxy.pac never worked for me. Not even once, on Android, iOS and MacOS.

[jon]

I should have added this in the first post. The client proxy is enabled and it seems to be working. And yes, this could be the problem. Since the client proxy seems to be working I figured I’d try the firewall rules side.

On MacOS I look at the scutil --proxy command:

This is all proxy disabled:

MBP6:~ jcmurphy$ scutil --proxy
<dictionary> {
  ExceptionsList : <array> {
    0 : *.local
    1 : 169.254/16
  }
  FTPPassive : 1
  HTTPEnable : 0
  ProxyAutoDiscoveryEnable : 0
}

and this is with Auto Proxy Discovery enabled:

MBP6:~ jcmurphy$ scutil --proxy
<dictionary> {
  ExceptionsList : <array> {
    0 : *.local
    1 : 169.254/16
  }
  FTPPassive : 1
  HTTPEnable : 0
  ProxyAutoConfigEnable : 1
  ProxyAutoConfigURLString : http://192.168.66.1:81/proxy.pac
  ProxyAutoDiscoveryEnable : 1
}

@cfusco

Auto-discovery by using proxy.pac never worked for me. Not even once, on Android, iOS and MacOS.

I saw your comment about Auto-discovery not working so I’ll check that first!

Out of curiosity - how do you block port 80 & 443 in the firewall rules?

[ip-mfg]

@jon

Out of curiosity - how do you block port 80 & 443 in the firewall rules?

Referring also to my firewall rule mentioned above I block all traffic for forward traffic as a general rule (see firewall options). Open is only mail and ICMP. So no traffic via port 80 o 443 is routed.

All traffic from clients go through squid automatically by the firewall itself.

On the outgoing firewall rules the rule mentioned above is working.
All traffic from client to port 80 or 443 goes now through the proxy.

On the client side proxy is manually activated in the OS (Opensuse) and in the browser (firefox)

Th Wpad rules worked on my Laptop (Opensuse) and are activated for the blue network. With other OSs I have no experience.

[cfusco]

@jon

Out of curiosity - how do you block port 80 & 443 in the firewall rules?

I first created a group including every network (blue, green, OpenVPN), then this group goes to the source, NAT is unchecked, destination is RED, service group is WEB (group that includes both TCP web ports 80 and 443), reject. This is the last rule of the firewall and any exception goes above, for example the machine where I use services that do not deal well with a proxy.

[cfusco]

@jon

The client proxy is enabled and it seems to be working. And yes, this could be the problem

@jon what happens if you disable transparent proxy? Can your client use it successfully in this case?

[jon]

The transparent proxy has been disabled since I started my experimenting. All of my experiments are on the BLUE network. ( I’ll do GREEN once I figure out what is not working. )

In the non-transparent mode

Depending on which firewall rule I am experimenting with - sometimes things work and sometimes not. I am having trouble figuring out if I am fighting against the Squid Proxy cache or fighting the browser cache or fighting a Firewall rule.

This is my current rule (current experiment!). I think it is similar to your description in Post #7. I wasn’t sure if your Firewall Rule (FR) RED is Standard RED of Firewall RED. So I just picked one.

[cfusco]

@jon

I wasn’t sure if your Firewall Rule (FR) RED is Standard RED of Firewall RED.

That’s correct. The group Web_proxy is composed of TCP 80 and 443, right? If yes, your rule is identical to mine.

[jon]

@cfusco

The group Web_proxy is composed of TCP 80 and 443, right?

Yes it is.

[cfusco]

@jon

Depending on which firewall rule I am experimenting with - sometimes things work and sometimes not. I am having trouble figuring out if I am fighting against the Squid Proxy cache or fighting the browser cache or fighting a Firewall rule.

Maybe you can figure out if the client is connecting to the proxy by looking at squid/access.log

 tail -f /var/log/squid/access.log

while you connect with the client’s browser to a website

[jon]

@cfusco

 tail -f /var/log/squid/access.log

Thanks. That’s been running since the beginning also.

(Browsers are WAY too busy when they should be sitting idle…)

[jon]

@cfusco

http://ipfire.localdomain:800 and https://ipfire.localdomain:800

what was done to setup the HTTPS side of the proxy server?

[cfusco]

@jon

what was done to setup the HTTPS side of the proxy server?

Allowed SSL ports, one per line

443 # https
563 # snews

I have Authentication method set to none, in case this is the problem.

[jon]

Mine is already set-up that way:

Was anything added to the pac/wpad file? I see this:

function FindProxyForURL(url, host)
{
if (
     (isPlainHostName(host)) ||
     (isInNet(host, "127.0.0.1", "255.0.0.0")) ||
     (isInNet(host, "192.168.6.0", "255.255.255.0")) ||
     (isInNet(host, "192.168.66.0", "255.255.255.0")) ||
     (isInNet(host, "10.10.10.0", "255.255.255.0")) ||
     (isInNet(host, "169.254.0.0", "255.255.0.0"))
   )
     return "DIRECT";

 else

if (
     (isInNet(myIpAddress(), "192.168.6.0", "255.255.255.0"))
   )
     return "PROXY 192.168.6.1:800";

 else

if (
     (isInNet(myIpAddress(), "192.168.66.0", "255.255.255.0"))
   )
     return "PROXY 192.168.66.1:800";
}

@cfusco

I have Authentication method set to none,

Mine is the same…

[cfusco]

This is mine:

function FindProxyForURL(url, host)
{
if (
     (isPlainHostName(host)) ||
     (isInNet(host, "127.0.0.1", "255.0.0.0")) ||
     (isInNet(host, "10.1.1.0", "255.255.255.0")) ||
     (isInNet(host, "10.1.3.0", "255.255.255.0")) ||
     (isInNet(host, "10.1.2.0", "255.255.255.0")) ||
     (isInNet(host, "169.254.0.0", "255.255.0.0"))
   )
     return "DIRECT";

 else

if (
     (isInNet(myIpAddress(), "10.1.1.0", "255.255.255.0")) ||
     (isInNet(myIpAddress(), "10.1.4.0", "255.255.255.0"))
   )
     return "PROXY 10.1.1.1:800";

 else

if (
     (isInNet(myIpAddress(), "10.1.3.0", "255.255.255.0"))
   )
     return "PROXY 10.1.3.1:800";
}

[jon]

they look very similar except for the IP address. Thanks for posting!

[jon]

Just wanted to add an update… I was getting frustrated with setting up the proxy so I figured I’d walk away for a few days and do something else.

I have the non-transparent (conventional) proxy running and things still don’t seem to be running correctly. I was hoping updating the CU 171 (test) might change something but still no joy.

I am guessing I have something configured incorrectly but I do not know what.

Right now browsing the Internet is painfully slow. To load a new web page takes about 30 seconds. If I enable transparent mode all of the slow disappears. I think it is my firewall rule but only because that is a big unknown for me.

My current firewall rule is in Post 9.

Can someone post a picture/screenshot of their entire Firewall Rule for blocking port 80 & Port 443?

[xperimental (Terry)]

I did the opposit. I do only allow a handfull of ports that are needed. Why should I keep every windows and door open? That’s just a security risk. So my default outgoing firewall settings is set to “blocked”.

I have 4 subnets for each green and blue.

1. is for stationary knows and trustfull infrastructure network members without internet access
2. is for stationary knows and trustfull enduser network members that must use the non-transparent proxy (otherwise no internet access)
3. is for mobile and trustfull enduser network members (may or may not use the non-transparent proxy, transparent proxy working for internet access)
4. is for any new/unknows network members (no internet access)

“Trustfull network members” are defined in the firewall hosts, therefore IP and MAC need to fit.

For a working internet access, it needs only rules for ICMP, NTP, DNS, HTTP and HTTPS (+ in our case ports for Email because of Thunderbird/Outlook):