This one does not work, using some defined pre-defined locations as source. Maybe I’ve not included all source countries for the LE challenge process, see questions below
Additionally:
LE uses a bunch of servers, located all over the world as the origin of the challenge requests. It’s not known where they are located, though.
This means my location blocks on IPFire does have some gaps, I cannot completely block almost all countries because of those LE servers locations.
Since I’ve added a chron top for automatically renew the certs, using a shell script, is there a shell command I could use to deactivate the location blocks (there is a WebIF checkbox) and reload the firewall prior executing the LE challenge process?
it sounds like the LE challenge will initiate request outside of ipfire to ipfire red0 interface from random LE servers across the world? does the request has any sort of unique traffic pattern? if there is, you could setup ipfire firewall rule to only allow incoming LE request based on the traffic payload pattern. tcpdump to capture LE request to red0 should reveal if there is any kind of unique traffic pattern.
Yes, that’s the case: The challenge is is initiated from outside, from RED and must reach IPFire’s web server under /var/www/dehydrated
To accomplish this I’ve created a vhost like:
Listen 5001
<VirtualHost *:5001>
DocumentRoot "/var/www/dehydrated"
Alias /.well-known/acme-challenge/ /var/www/dehydrated/
</VirtualHost>
This works already fine for a long time now as long as I won’t set any location blocks or at least if I exclude some countries I’ve guess where the LE challenge is started from. Unfortunately LE company adds some new locations over the time which means I’ve to adjust the excluded locations again and again. Most of the time I’m not aware that LE change the origins on their side and the challenge process fails in the background.
Anyway, HAproxy is configured to route the challenge request from external to this port 5001.
...
acl is_letsencrypt path_beg -i /.well-known/acme-challenge/
use_backend letsencrypt if is_letsencrypt is_ddns_domain
...
backend letsencrypt
server letsencrypt 127.0.0.1:5001 check
As you might notice, HAproxy looks at the incoming URI and routes the LE traffic to the local webserver, port 5001.
Since you mentioned it, is there a chance to add the internal traffic from RED to port 5001 using a specific firewall rule?
Let’s Encrypt, by policy, do not/will not publish or provide lists of the validation IP’s.
Additionally since 2020, they do validation via multiple regions via a single cloud provider and at that time they indicated that they would change to multiple region validation via multiple cloud providers. https://letsencrypt.org/2020/02/19/multi-perspective-validation
So I think it is just going to get more and more difficult to be bale to do any location blocking when using the HTTP-01 challenge.
Interesting use case, just want to be clear on your issue.
current traffic path:
LE challenge URI request → HAProxy *:80 (inspect URI) → ipfire apache 127.0.0.1:5001 ?
and you can’t use location blocks firewall to HAProxy listening on red0 port 80 since LE challenge could come from unknown locations to you ?
so if you allow all locations to HAProxy *:80, but only allow LE challenge URI pass through HAProxy, drops all other URI request, would this work? or you can’t because you need to have HAProxy *:80 serving other HTTP request as well.
Almost correct again see comment below about my subdomains.
Again, absolutely correct.
The LE challenge process is only one job, HAVProxy serves. I’ve got a WebDAV server, a CaslDAV server and some more running behind IPFire. Each service uses their own subdomain, though.
HAProxy inspects the URI, and forwards the traffic to the appropriate servers in LAN. As written above, this is done by subdomain request.
You are right, assuming that LE comes in at RED port 80 and HAProxy again inspects the URI searching for the path .well-known… and routes the request to IPfire, port 5001
If you are only using http (port 80) for the LE http challenge then the only option is to have a firewall rule that blocks port 80 and only opens it for the short time that the challenge is carried out, so once the certificates have less than 30 days validity left.
If you are using port 80 for other website activities via the red interface from some locations then you are stuck.
Maybe you could have the location blocking in place and only disable it in the time frame when you need the LE update to occur.
You would need to figure out how to turn on and off the location blocking via a script. It might be possible, but I don’t know myself.
That would be perfect, honestly I have done this before, using some commands in my renewing shell script, but unfortunately they now fail since IPFire uses a different location blocking mechanism.
And to be fair, LE used just US servers before and I had to unblock a single country. Now, LE uses many countries as their origin which we do not know (you just gave the link above, thanks for this, Adolf)
So, I could imagine to stop all location block while renewing starts and acivate the blocks immediately afterwards. This would include, restarting the firewall, too, as it is necessary in WebIF.
However, I don’t know the exact commands to stop location blocks, restart the firewall and vice versa.
EDIT: I assume, the firewall is running before HAProxy, correct. Otherwise, the location block would be obsolet, correct?
This is why I will need a firewall rule to allow incoming traffic on RED to be forwarded to IPFire itself?
I maybe miss something basic, aren’t all traffic initiated from outside/internet to red0 interface blocked by default in ipfire? how do you allow HAProxy handle other web services when ipfire has default firewall policy to block incoming traffic to red0. the general solution for ipfire to forward incoming traffic to servers in LAN network is to use port forward, but you are using HAProxy, not port forward.
This rule is what causes me some headaches because basically all external traffic is routed to IPFire on port 80 and 443.
This is fine of course if you have got servers (docker containers etc.) behind IPFire and don’t want to use the orange zone, but may open the door for some bad guys.
Hence, I had some additional location blocks in place, too, but those are now interfering with the LE challenge process.
That’s why I’ve come here an asked for some tips on how to add more security for this special case
It depends on what your servers are serving. If they are intended to be available for general users on the internet then you have no alternative but to let all users in. You just have to make sure that your server via apache or nginx or… is tightened down enough so any users can not access things they should not.
If your servers/domain names are only intended for internal use, which is my situation, then except for the LE challenge, which I have specified in each Virtual Host to be allowed, every other http access is automatically redirected to https and the https virtual host definition says that only specified IP’s can access the server, which means that any external accesses get redirected to https and then get a “403 Forbidden” response.
All of this is done via the apache virtual host definitions and not via IPFire, as my apache server for my needs is on an internal machine.
I see, since you are using HAProxy as LB, there is alternative LB solution like GitHub - loxilb-io/loxilb: eBPF based cloud-native load-balancer. Powering Kubernetes|Edge|5G|IoT|XaaS Apps. that works at Linux kernel TC network layer, which would bypass ipfire netfilter iptable firewall rules, so you do not need to open up ipfire firewall on red0 and keep the default deny iptables firewall policy. LoxiLB would handle the load balance function at TC layer to load balance incoming traffic to port 80/443 to your servers in LAN network. LoxiLB could also inspect the LE URI to redirect it to ipfire apache 127.0.0.1:5001, the URI inspecting feature may not be available now in current LoxiLB release, but LoxiLB has just added layer 7 fullproxy feature that could do the URI inspecting, I have been working with LoxiLB developer recently and they are open to ideas. See the LoxiLB load balance configuration here https://youtu.be/DxrhLGrifE4?si=GhJt7rbbyNlG_Zqc on BPFire which is clone of IPFire 2.x
By the way, HAproxy can’t bypass firewall rules because the traffic path to Linux is like ->red0->XDP-TC->netfilter(ipfire use)->tcp/ip stack->socket->HAProxy
see comment below about my subdomains.
