Reverse Path Filtering prevents certbot renewal using HTTP-1 acme-challenge

I have a next cloud server in orange, behind IPFire and even though I disabled any possible rule, when I initiate a web based renewal process it fails because let’s encrypt does not receive any answer and interpret this as a failure. I think the firewall drops those packets. These are the kernel logs I see when I start the certbot renew process

Sep  9 14:52:26 ipfire kernel: DNAT IN= OUT=lo SRC=80.253.88.254 DST=80.253.88.254 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=34887 DF PROTO=TCP SPT=57720 DPT=443 WINDOW=65495 RES=0x00 SYN URGP=0 
Sep  9 14:52:27 ipfire kernel: DNAT IN= OUT=lo SRC=80.253.88.254 DST=80.253.88.254 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=49412 DF PROTO=TCP SPT=57732 DPT=443 WINDOW=65495 RES=0x00 SYN URGP=0 
Sep  9 14:52:27 ipfire kernel: DNAT IN= OUT=lo SRC=80.253.88.254 DST=80.253.88.254 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40792 DF PROTO=TCP SPT=57742 DPT=443 WINDOW=65495 RES=0x00 SYN URGP=0 
Sep  9 14:52:27 ipfire kernel: DNAT IN= OUT=lo SRC=80.253.88.254 DST=80.253.88.254 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=57921 DF PROTO=TCP SPT=57752 DPT=443 WINDOW=65495 RES=0x00 SYN URGP=0 
Sep  9 14:52:29 ipfire kernel: DROP_INPUT IN=red0 OUT= MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=89.248.163.237 DST=80.253.88.254 LEN=40 TOS=0x00 PREC=0x00 TTL=240 ID=51379 PROTO=TCP SPT=40441 DPT=1067 WINDOW=1024 RES=0x00 SYN URGP=0 
Sep  9 14:52:30 ipfire kernel: DROP_INPUT IN=red0 OUT= MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=87.246.7.198 DST=80.253.88.254 LEN=40 TOS=0x00 PREC=0x00 TTL=241 ID=54594 PROTO=TCP SPT=48828 DPT=851 WINDOW=1024 RES=0x00 SYN URGP=0 
Sep  9 14:52:49 ipfire kernel: DROP_INPUT IN=red0 OUT= MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=89.248.165.20 DST=80.253.88.254 LEN=40 TOS=0x00 PREC=0x00 TTL=240 ID=21813 PROTO=TCP SPT=44274 DPT=8153 WINDOW=1024 RES=0x00 SYN URGP=0 
Sep  9 14:52:53 ipfire kernel: DNAT IN=red0 OUT= MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=18.119.10.60 DST=80.253.88.254 LEN=60 TOS=0x00 PREC=0x00 TTL=39 ID=57653 DF PROTO=TCP SPT=55690 DPT=80 WINDOW=62727 RES=0x00 SYN URGP=0 
Sep  9 14:52:53 ipfire kernel: FORWARDFW IN=red0 OUT=orange0 MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=18.119.10.60 DST=10.1.2.100 LEN=60 TOS=0x00 PREC=0x00 TTL=38 ID=57653 DF PROTO=TCP SPT=55690 DPT=80 WINDOW=62727 RES=0x00 SYN URGP=0 
Sep  9 14:52:53 ipfire kernel: DNAT IN=red0 OUT= MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=35.90.117.55 DST=80.253.88.254 LEN=60 TOS=0x00 PREC=0x00 TTL=39 ID=560 DF PROTO=TCP SPT=62832 DPT=80 WINDOW=62727 RES=0x00 SYN URGP=0 
Sep  9 14:52:53 ipfire kernel: FORWARDFW IN=red0 OUT=orange0 MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=35.90.117.55 DST=10.1.2.100 LEN=60 TOS=0x00 PREC=0x00 TTL=38 ID=560 DF PROTO=TCP SPT=62832 DPT=80 WINDOW=62727 RES=0x00 SYN URGP=0 
Sep  9 14:52:53 ipfire kernel: DNAT IN=red0 OUT= MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=23.178.112.107 DST=80.253.88.254 LEN=60 TOS=0x00 PREC=0x00 TTL=44 ID=439 DF PROTO=TCP SPT=17448 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0 
Sep  9 14:52:53 ipfire kernel: FORWARDFW IN=red0 OUT=orange0 MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=23.178.112.107 DST=10.1.2.100 LEN=60 TOS=0x00 PREC=0x00 TTL=43 ID=439 DF PROTO=TCP SPT=17448 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0 
Sep  9 14:52:56 ipfire kernel: DNAT IN=red0 OUT= MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=3.73.48.232 DST=80.253.88.254 LEN=60 TOS=0x00 PREC=0x00 TTL=44 ID=39482 DF PROTO=TCP SPT=23434 DPT=80 WINDOW=62727 RES=0x00 SYN URGP=0 
Sep  9 14:52:56 ipfire kernel: FORWARDFW IN=red0 OUT=orange0 MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=3.73.48.232 DST=10.1.2.100 LEN=60 TOS=0x00 PREC=0x00 TTL=43 ID=39482 DF PROTO=TCP SPT=23434 DPT=80 WINDOW=62727 RES=0x00 SYN URGP=0 
Sep  9 14:52:56 ipfire kernel: DNAT IN= OUT=lo SRC=80.253.88.254 DST=80.253.88.254 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=28626 DF PROTO=TCP SPT=42700 DPT=443 WINDOW=65495 RES=0x00 SYN URGP=0 
Sep  9 14:53:21 ipfire kernel: DROP_INPUT IN=red0 OUT= MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=122.202.54.44 DST=80.253.88.254 LEN=40 TOS=0x00 PREC=0x00 TTL=46 ID=39894 PROTO=TCP SPT=12289 DPT=23 WINDOW=14601 RES=0x00 SYN URGP=0 
Sep  9 14:53:22 ipfire kernel: DROP_INPUT IN=red0 OUT= MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=167.248.133.141 DST=80.253.88.254 LEN=44 TOS=0x00 PREC=0x00 TTL=34 ID=48989 PROTO=TCP SPT=40600 DPT=5000 WINDOW=1024 RES=0x00 SYN URGP=0 
Sep  9 14:53:26 ipfire kernel: DNAT IN= OUT=lo SRC=80.253.88.254 DST=80.253.88.254 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=22818 DF PROTO=TCP SPT=60202 DPT=443 WINDOW=65495 RES=0x00 SYN URGP=0 
Sep  9 14:53:27 ipfire kernel: DNAT IN= OUT=lo SRC=80.253.88.254 DST=80.253.88.254 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=57316 DF PROTO=TCP SPT=60212 DPT=443 WINDOW=65495 RES=0x00 SYN URGP=0 
Sep  9 14:53:27 ipfire kernel: DNAT IN= OUT=lo SRC=80.253.88.254 DST=80.253.88.254 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=43527 DF PROTO=TCP SPT=60216 DPT=443 WINDOW=65495 RES=0x00 SYN URGP=0 
Sep  9 14:53:35 ipfire kernel: DROP_CTINVALID IN=red0 OUT= MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=103.141.158.237 DST=80.253.88.254 LEN=40 TOS=0x00 PREC=0x00 TTL=45 ID=0 DF PROTO=TCP SPT=1812 DPT=44714 WINDOW=0 RES=0x00 ACK RST URGP=0 
Sep  9 14:53:36 ipfire kernel: DNAT IN=red0 OUT= MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=51.81.167.146 DST=80.253.88.254 LEN=60 TOS=0x00 PREC=0x00 TTL=45 ID=7154 DF PROTO=TCP SPT=50720 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0 
Sep  9 14:53:36 ipfire kernel: FORWARDFW IN=red0 OUT=orange0 MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=51.81.167.146 DST=10.1.2.100 LEN=60 TOS=0x00 PREC=0x00 TTL=44 ID=7154 DF PROTO=TCP SPT=50720 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0 
Sep  9 14:53:36 ipfire kernel: DNAT IN=red0 OUT= MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=51.81.167.146 DST=80.253.88.254 LEN=60 TOS=0x00 PREC=0x00 TTL=45 ID=52310 DF PROTO=TCP SPT=46098 DPT=443 WINDOW=64240 RES=0x00 SYN URGP=0 
Sep  9 14:53:36 ipfire kernel: FORWARDFW IN=red0 OUT=orange0 MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=51.81.167.146 DST=10.1.2.100 LEN=60 TOS=0x00 PREC=0x00 TTL=44 ID=52310 DF PROTO=TCP SPT=46098 DPT=443 WINDOW=64240 RES=0x00 SYN URGP=0 
Sep  9 14:53:42 ipfire kernel: DROP_INPUT IN=red0 OUT= MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=121.231.79.51 DST=80.253.88.254 LEN=40 TOS=0x00 PREC=0x00 TTL=49 ID=64974 PROTO=TCP SPT=3917 DPT=23 WINDOW=12405 RES=0x00 SYN URGP=0 
Sep  9 14:53:56 ipfire kernel: DNAT IN= OUT=lo SRC=80.253.88.254 DST=80.253.88.254 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=22934 DF PROTO=TCP SPT=41250 DPT=443 WINDOW=65495 RES=0x00 SYN URGP=0 

in core update 168 there was a security improvement, I quote:

IPFire now drops any packet that is received on a different interface than it would have been routed back to.

There were several thread on how to disable this feature. I can’t remember anything to help me find those thread. Can someone point out one of those thread or tell me how to disable this feature?

Thank you, regardless.

Found it:

sysctl net.ipv4.conf.default.rp_filter=2
sysctl net.ipv4.conf.all.rp_filter=2

As I suspected, with these two commands, let’s encrypt now works. I will report the problem to their forum and see If I can do something about it.

1 Like

That was about Asymmetric routing.
If you search for that you will find it.

Are you using location block?

Found it. No other block. Now everything works. It’s the way certbot servers do the challange. It activates the RPF filtering. I am considering opening a bug report with them.

How do I reactivate the filter without rebooting?

I opened a bug report on certbot issue tracker: Reverse Path FIltering prevents a succesful HTTP-1 acme-challenge · Issue #9403 · certbot/certbot · GitHub

2 Likes

I found the answer, rp_filter | sysctl-explorer.net

sysctl net.ipv4.conf.default.rp_filter=1
sysctl net.ipv4.conf.all.rp_filter=1
1 Like

To fix the problem in an automatic way, this is what I did:

  1. my web server sends a certbot renew every Sunday at 00:01;
  2. one minute before, IPFire through the user fcronuser will run a shell script changing the RPF status;
  3. 15 minutes later, it reverts to the default status.

To accomplish this, I followed the fantastic @bonnietwin tutorial to create fcronuser with all the right configuration to have fcrontab starting the script with superuser privileges. I works flawlessly.

00 0 * * 7 "sudo /home/cfusco/bin/rpf.sh 2"
15 0 * * 7 "sudo /home/cfusco/bin/rpf.sh 1"

of course take care of changing the path of the script.

This is the heart of the script:

#!/bin/bash

########################################################### 
# set strictness of Reverse Path Filtering; see RFC 3704, #
# section 2.2: 1, RPF strict; 2, RPF relaxed.             #
###########################################################

/sbin/sysctl net.ipv4.conf.default.rp_filter=$1
/sbin/sysctl net.ipv4.conf.all.rp_filter=$1

HI,

I have found the trouble this night.
just uncheck A3 Worldwide Anycast Instance in the location block menu.

it is working fine now for new and renew certificate :smiley:

In my case, disabling all location block did not help. It was the first thing I tried, being obvious as a possibility, since the certbot server location is unknown. Only disabling the reverse path filter solved the problem.

I managed to successfully complete a certboot renewal without relaxing the RP filtering. I suspect I did something wrong the first time and the location block was the real culprit. Apologies for any confusion I might have caused.

2 Likes