HAProxy, Dehydrated, Let's Encrypt certs, Apache vhost

Hi,

as the title possibly describes already, this will be a rather long posting about configuring HAProxy, Dehydrated and Apache vhost, all running on IPFire to create a LE certificate.

Some background information:
I successfully managed to set up package HAProyx so far to forward requests to IPFire’s port 80 (port 443 will follow as soon as there is a LE cert) to some of my internal servers.

I’m using this firewall rule as shown below:


Source is set up with two GeoIP countries: DE and US for security reasons (have tried standard network: RED, too)

HAProxy’s config file below. The bind IP address is the RED IP, IPFire is running behind a cable modem as exposed host.

global
log 127.0.0.1 local1
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user nobody
group nobody
daemon

tune.ssl.default-dh-param 2048

defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch except 172.0.0.0/8
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000

frontend http_https
mode http
bind 172.17.0.2:80

acl is_webdav_domain hdr_beg(host) -i webdav.mydomain.de

acl is_letsencrypt path_beg -i /.well-known/acme-challenge/
use_backend letsencrypt if is_letsencrypt is_webdav_domain

backend letsencrypt
server letsencrypt 192.168.0.1:5001

192.168.0.1 is IPfire green IP address, have tried 127.0.0.1 as well. Port 5001 is a vhost of IPFire’s own Apache webserver.

I’ve configured a vhost to answer Let’s Encrypt requests, of course Apache was restarted (many times):

Listen 5001
<VirtualHost *:5001>
DocumentRoot “/var/www/dehydrated”
Alias /.well-known/acme-challenge /var/www/dehydrated

Options Indexes MultiViews

Dehydrated is set up, IMO, correctly, option WELLKNOWN points to path /var/www/dehydrated and contains a single file index.html for debugging purposes.
Option CONTACT_EMAIL is set.

File domains.txt contains one single domain so far: webdav.mydomain.de. Subdomain “webdav” is configured as a CNAME at my providers DNS setup, targetting a DynDNS domain: ddns.mydomain.de. The DynDNS’ IP address gets regularly updated by IPFire.

Using a browser, I can reach webdav.mydomain.de and the web site is loading as expected. Calling URL webdav.mydomain.de/.well-known/acme-challenge/index.html shows the debug content, some formatted texts, btw.

With this above setup above, I thought I will get Deyhrated to create a certificate for the domain webdav.mydomain.de. However, I had no luck so far!

Using command dehydrated --register --accept-terms was successful:

INFO: Using main config file /etc/dehydrated/config

  • Generating account key…
  • Registering account key with ACME server…
  • Fetching account ID…
  • Done!

Right after this command I’ve used dehydrated -c

HAProxy’s log file showed that an external IP was accessing a generic file name, at the time of executing deydrated:

Jun 2 09:32:51 localhost haproxy[30978]: 18.196.96.172:55688 [02/Jun/2020:09:32:51.305] http_https letsencrypt/letsencrypt 1/0/0/1/2 200 269 - - ---- 1/1/0/0/0 0/0 {webdav.mydomain.de} “GET /.well-known/acme-challenge/0p9KTsImDf7ruVEtPMJZsONvAspVo_eB5nuLr22erLw HTTP/1.1”

I even could access this file, too if I was fast enough and dehydrated did not delete it so far.

While dehydrated managed to create some cert files inside path /etc/dehydrated/certs, the process failed at the end with output:

Challenge validation has failed :frowning:
ERROR: Challenge is invalid! (returned: invalid) (result: {
“type”: “http-01”,
“status”: “invalid”,
“error”: {
“type”: “urn:ietf:params:acme:error:connection”,
“detail”: “Fetching http://webdav.mydomain.de/.well-known/acme-challenge/0p9KTsImDf7ruVEtPMJZsONvAspVo_eB5nuLr22erLw: Timeout during connect (likely firewall problem)”,
“status”: 400
},
“url”: “https://acme-v02.api.letsencrypt.org/acme/chall-v3/4954445819/BBqt1w”,
“token”: “0p9KTsImDf7ruVEtPMJZsONvAspVo_eB5nuLr22erLw”,
“validationRecord”: [
{
“url”: “http://webdav.mydomain.de/.well-known/acme-challenge/0p9KTsImDf7ruVEtPMJZsONvAspVo_eB5nuLr22erLw”,
“hostname”: “webdav.mydomain.de”,
“port”: “80”,
“addressesResolved”: [
“178.11.222.33”
],
“addressUsed”: “178.11.222.33”
}
]
})

I’ve now tried everything that came into my mind, to no avail so far! I did two days of researching in internet but no useful information was found, that would help in my situation.

This includes setting the firewall forward behaviour from blocked to allowed and changing the above firewall rule from Source: GeoIP to Standard Networks: RED.

So, if somebody has successfully set up HAProxy with IPFire and managed to create some certs, I would be glad he could post a running config here!

Thank you,
Michael

Further information:
Just discovered the website https://letsdebug.net and used it to verify my domain reachability. The test revealed:


I had to create a file letsdebug-test beforehand, though.

Now I’m clueless because when using the URL mentioned in the hardcopy above and requesting access to it from a browser or from IPFire command line, all seems to be OK:

curl http://webdav.mydomain.de/.well-known/acme-challenge/letsdebug-test -v
Return code is 200 in this case.

Wait a minute!!
I just pressed the Apply Change button in WebIF - Firewall rules and rerun the test in parallel and the test immediately finished OK!
This obviously means during the time of resetting the firewall and reloading the rules, the test succeeded.

So where is the problem, because after the WebIF finished reloading the rules so did the firewall, a subsequent test, failed again!

Obviously it’s not HAProxy or the vhost, but one of my firewall rules.

A quick idea: I do have global GeoIP blocking in place, for all countries but DE. Maybe this is the issue? Does GeoIP blocking take place prior to a firewall rule? My rule for allowing access to port 80 and 443 has as source DE and US. So I assumed this is enough for granting access to those ports from both DE and US.

A quick solution showed: Excluding US from GeoIP blcking did the job!! Hurray! I leave the above lines for reference though.

Last question: My current firewall rule for HAProxy and port 80 and 443 is located in section " Incoming Firewall Access" in WebIF.
Any chance to create another rule that overrides the GeoIP blocking. How should such a rule be defined?

Of course I could exclude country US from GeoIP forever, however, I rather would see this exception in a separate rule and exclusively for ports 80 and 443.

cu and thanks for reading my long post :wink:
Michael