Core 203: Knot Resolver fails to answer DNS queries coming through WireGuard and OpenVPN tunnels (reply from unexpected source)

[d0cx1989 (d0cX)]

Hello everyone,

After updating to Core 203 I ran into a DNS problem that also seems to affect other users with OpenVPN, so I want to document it properly here with the analysis and a working fix.

Setup

My Green network is 192.168.99.0/28 and the firewall itself is 192.168.99.1. Clients connecting over WireGuard use 192.168.99.1 as their DNS server. This worked perfectly before Core 203.

Symptom

After the update, name resolution over the tunnel stopped working. Resolving raw IP addresses was fine, but DNS queries failed. On the client I saw this:

nslookup google.de 192.168.99.1
;; reply from unexpected source: 192.168.4.250#53, expected 192.168.99.1#53
;; connection timed out; no servers could be reached

So the query goes to 192.168.99.1, but the answer comes back from 192.168.4.250, which is my Red (WAN) interface. The client rejects the reply because the source does not match, and the lookup times out.

Analysis

The new Knot Resolver configuration in /etc/knot-resolver/config.yaml only binds to the wildcard address:

network:
  listen:
    - interface: 0.0.0.0@53

When a socket is bound to 0.0.0.0, the kernel decides which source address to use for outgoing replies based on the routing table. For traffic leaving through the tunnel interface, it picks the Red interface IP instead of the Green IP the query was originally sent to. That is why the client sees the answer coming from an unexpected source and discards it.

Fix

Adding an explicit listen socket for the Green interface address solves the problem completely:

network:
  listen:
    - interface: 0.0.0.0@53
    - interface: 192.168.99.1@53

After restarting the service, both workers open dedicated sockets on the Green IP in addition to the wildcard, and there is no address conflict:

ss -tulpn | grep :53
udp   UNCONN   192.168.99.1:53    users:(("kresd",...))
udp   UNCONN        0.0.0.0:53    users:(("kresd",...))
tcp   LISTEN   192.168.99.1:53    users:(("kresd",...))
tcp   LISTEN        0.0.0.0:53    users:(("kresd",...))

Local resolution on the firewall still works, and queries through the tunnel now return correct answers with the correct source address.

The problem with this fix

The file starts with a clear warning:

# DO NOT EDIT as any changes will be overwritten

So this manual change does not survive a DNS settings change or a core update, because the configuration file gets regenerated. This means it is only a temporary workaround, not a real solution.

Suggestion

Would it be possible to have the configuration generator include the configured interface addresses as explicit listen sockets in addition to 0.0.0.0? Alternatively the resolver could be told to reply from the same address the query was received on. Either approach would restore tunnel DNS behaviour for WireGuard and OpenVPN users.

Can anyone confirm this on their own system?

Thanks a lot.

[kuni (kuni bertus)]

Thx for your investigation.

@d0cx1989

Adding an explicit listen socket for the Green interface address solves the problem completely:

I got the problem with my workaround, that knot-resolver ist not starting correctly, when i reboot ipfire. Does this fix work over a reboot?

@d0cx1989

So this manual change does not survive a DNS settings change or a core update, because the configuration file gets regenerated. This means it is only a temporary workaround, not a real solution.

Yes it’s only a workaround. I think there will be a bugtrack soon.

Edit:

Forgot to mention: If you have another DNS resolver/server (like a windows dns-server) in your network, you can put that ip-adress in the vpn dns-configuration instead.

[d0cx1989 (d0cX)]

Thanks for testing this.

You are right to be cautious. Because of the # DO NOT EDIT as any changes will be overwritten header, I would not expect this workaround to survive a reboot reliably. The configuration file is regenerated by the IPFire scripts, so any manual change to it is at risk of being overwritten whenever the DNS settings are rebuilt, and possibly on boot as well.

I want to be honest here, i run this on a production firewall at a remote site, so I cannot freely reboot it to reproduce the boot behaviour myself right now. That is exactly why I think the real fix has to come from the configuration generator rather than from a manual edit. A hand edited file is never going to be a safe solution in production.

The startup problem you describe sounds like a separate issue though. If knot-resolver does not start correctly after a reboot, that points to something in the generation or startup order rather than to the manual listen socket itself.

This is really something the developers should look at, so that the configured interface addresses are included in the generated config automatically.

[tphz (iptom)]

@d0cx1989

Clients connecting over WireGuard

Which ‘AllowedIPs’ addresses do these clients use?

[d0cx1989 (d0cX)]

Good question. These clients use a split tunnel, not a full redirect. The AllowedIPs are limited to my Green subnet only:

AllowedIPs = 192.168.99.0/28

So there is no 0.0.0.0/0 in the configuration. Only traffic destined for the Green network goes through the tunnel, including the DNS queries to 192.168.99.1. Everything else stays on the client’s normal route.

This is actually part of why the unexpected source problem stands out so clearly. The query is sent specifically to 192.168.99.1, which is inside the tunnel, but the reply comes back from the Red interface address, which is outside the tunnel scope entirely. The client rightly rejects it.

[kuni (kuni bertus)]

@tphz I use split tunnel too. In one network, I use full-tunnel (AllowedIPs = 0.0.0.0/0) for some wireguard roadwarriors, but I didn’t checked, whether dns-resolving is working here. Do you think, the problem only occurs with split-tunnel?

[balooh (Kurt Kalloch)]

Hello, see: Core 203 and wireguard

[tphz (iptom)]

@kuni

Do you think, the problem only occurs with split-tunnel?

I don’t know. I don’t need DNS via VPN for my connections.
CU203 introduces some major changes.
At the moment, I don’t have enough time to analyse these issues in detail.

The links below may be helpful

Core Update 203: Recommendations Before and After Upgrading

If you are upgrading from an earlier IPFire version, or performing a fresh installation followed by restoring a backup created before CU203, the following steps are recommended after the upgrade. 1. Regenerate the DHCP configuration The migration from Unbound to Knot Resolver requires the DHCP configuration to be regenerated. Go to Network → DHCP Server and click the Save button (no configuration changes are required). 2. Review DHCP fixed leases and hostnames In CU203, hostnames for DHCP fix…

Cu 203 knot-resolver openvpn not working

Personally, I use a DNS redirection rule to the firewall, as described here, for each of my networks (GREEN, BLUE, OpenVPN): https://www.ipfire.org/docs/configuration/firewall/rules/redirect-dns (Adding TCP 853 to DNS services group for DoT) After upgrading to CU203, I don’t have any DNS issues with OpenVPN. Hope this helps.

[trompette722 (Nico Gilbert)]

Two things worth adding while #14030 is being fixed: the documented workaround can break your resolver, and there is a safer one

(Reply to Core 203: Knot Resolver fails to answer DNS queries coming through WireGuard and OpenVPN tunnels)

I traced this one end to end on my own box before finding
bug #14030, so I will not
repeat the root cause — it is already correctly identified there, and a
proof-of-concept patch has been submitted upstream. Nothing to add on that front.

Two practical points are missing from both the bug and this thread, though, and
one of them explains why several people here ended up with a resolver that would
not start.

The documented workaround can leave you without DNS at all

Bug #14030 and this thread both suggest adding the zone addresses to
/etc/knot-resolver/config.yaml instead of relying on the wildcard:

network:
  listen:
    - interface: 0.0.0.0@53
    - interface: 192.168.1.1@53

Applied as-is, this makes kresd fail to start on the next reboot. The reason is
the boot order:

S11knot-resolver    <-- resolver starts here
S20network          <-- interfaces get their addresses here
S50wireguard
S85firewall

Knot Resolver starts at S11, before network at S20. At bind time
192.168.1.1 does not exist yet, the bind fails, and the daemon aborts. This is
almost certainly why IPFire configured 0.0.0.0 in the first place — it is the
only address guaranteed to exist that early.

The Knot Resolver 6 schema already has the option that fixes this
(knot_resolver/datamodel/network_schema.py, ListenSchema):

network:
  listen:
    - interface: 0.0.0.0@53
    - interface: 192.168.1.1@53
      freebind: true

freebind allows binding to an address that is not local yet. If you are running
the workaround today, this is worth adding before your next reboot.

It still gets wiped on every Core Update, since the file carries
# DO NOT EDIT as any changes will be overwritten.

A workaround that survives Core Updates and cannot break DNS

The wrong source address is picked because the WireGuard pool route has no src
attribute — /etc/rc.d/init.d/wireguard, line 168:

ip route add "${CLIENT_POOL}" dev "${intf}"

wg0 has no address of its own, so the kernel falls back to the RED address:

# ip route get 10.11.12.1
10.11.12.1 dev wg0 src <RED_IP> uid 0

Giving the route a source address fixes it without touching the resolver. In
/etc/sysconfig/firewall.local, start) section:

if ip link show wg0 >/dev/null 2>&1; then
        ip route replace 10.11.12.0/24 dev wg0 scope link src 192.168.1.1
fi

Adjust the pool and the GREEN address to your setup. Three reasons I ended up
preferring this over editing config.yaml:

• firewall.local is a user file — IPFire never overwrites it, so it survives
  Core Updates.
• It cannot stop the resolver from starting. A failed bind takes DNS down for the
  entire network; a failed route only affects the tunnel.
• The boot order works in its favour: S50wireguard runs before S85firewall,
  so wg0 and its route already exist. rules.pl also re-runs
  firewall.local reload on every firewall rebuild.

The limitation is that it is WireGuard-specific — OpenVPN needs the equivalent on
tun+.

Before / after

Same client, same 3-minute tcpdump -i wg0 window:

	before	after
Source of DNS replies	<RED_IP>.53	192.168.1.1.53 (19/19)
ICMP port unreachable	on every query	0
Retransmissions	every query, 5 s apart	none
Total packets in tunnel	241	5778

The packet count is the most telling number: with DNS broken the tunnel was
carrying almost nothing, because nothing could resolve.

For anyone diagnosing this, the signature is unmistakable in
tcpdump -i wg0 -nn "port 53" — query to the GREEN address, answer from the RED
address, then an ICMP port unreachable from the client, then the same query ID
again five seconds later.

Happy to test patches once something lands.