After Core Update 203: DNSSEC validation failure for tm.svc.cloud.microsoft: valid NSEC rejected by Knot Resolver

Network DNS
1

Hi. After upgrading to IPFire 203 we are experiencing DNS issues. I’m not a DNS expert so have been doing my best to diagnose with the help of AI, which I’ve then used to write this summary (it probably explains the issue better than I can myself). APoligies if this offends…

Summary

DNS resolution of www.office.com fails with SERVFAIL, while the apex domain office.com and other domains resolve normally.

Our IPFire (Core 203) forwards queries to Google DNS (8.8.8.8 and 8.8.4.4).

Knot rejects a valid NSEC response beneath tm.svc.cloud.microsoft with:

EDE: 12 (NSEC Missing)

Google DNS validates the same response successfully. Disabling checking for an individual query with +cd also succeeds. Clearing the complete Knot cache did not resolve the problem.

Basic reproduction

Run locally on the affected IPFire machine:

nslookup office.com
nslookup www.office.com

Results:

office.com       → resolves successfully
www.office.com   → SERVFAIL

www.office.com follows this CNAME chain:

www.office.com
→ officehomemcm.anc.tm.svc.cloud.microsoft
→ officehomemcm.afdcafe.tm.svc.cloud.microsoft
→ home-office365-com.b-0004.b-msedge.net
→ b-0004.b-msedge.net

Diagnostic evidence

A normal local query fails:

dig www.office.com A

Result:

status: SERVFAIL
EDE: 12 (NSEC Missing)

Disabling DNSSEC checking for the query succeeds:

dig +cd www.office.com A

Querying the configured upstream resolvers directly also succeeds:

dig @8.8.8.8 www.office.com A
dig @8.8.4.4 www.office.com A

The failure can be reproduced more specifically:

dig tm.svc.cloud.microsoft NS +dnssec

Result:

status: SERVFAIL
EDE: 12 (NSEC Missing)

The same local query succeeds with validation disabled:

dig +cd tm.svc.cloud.microsoft NS +dnssec

It returns a signed NSEC proof containing:

tm.svc.cloud.microsoft.
  NSEC \000.tm.svc.cloud.microsoft. RRSIG NSEC

Google DNS validates that proof successfully and sets the AD flag:

dig @8.8.8.8 tm.svc.cloud.microsoft NS +dnssec

The cloud.microsoft chain of trust validates successfully locally:

dig cloud.microsoft DS +dnssec
dig cloud.microsoft DNSKEY +dnssec

Both return NOERROR with the AD flag.

The complete resolver cache was cleared:

kresctl cache clear .

The failure reproduced immediately afterward, so it is not a cache issue.

2

I dont know is or not a bug, but go to: https://10.254.0.1:444/cgi-bin/dnsbl.cgi and put “www.office.com” in Allowed domains (one per line) and it work.

In cmd:

Microsoft Windows [Versión 10.0.19044.7548]
(c) Microsoft Corporation. Todos los derechos reservados.

C:\Windows\system32>nslookup
Servidor predeterminado:  bs.northsecure.es
Address:  10.254.0.1

> www.office.com
Servidor:  bs.northsecure.es
Address:  10.254.0.1

*** bs.northsecure.es no encuentra www.office.com: Server failed
> www.office.com
Servidor:  bs.northsecure.es
Address:  10.254.0.1

Respuesta no autoritativa:
Nombre:  atm.common.mira.tm.svc.cloud.microsoft
Addresses:  2603:1061:1600:3405::365:c0
          2603:1061:1600:3407::365:c0
          2603:1061:1600:3401::365:c0
          2603:1061:1600:3404::365:c0
          23.103.244.5
          23.103.244.0
          23.103.244.1
          23.103.244.4
Aliases:  www.office.com
          officehomemcm.anc.tm.svc.cloud.microsoft
          officehomemcm.afdcafe.tm.svc.cloud.microsoft

The first is the mistake, and the second, after doing what I tell you.

3

That is because by doing this (adding www.office.com to the custom allow list), you bypass the DNS redirection to Google DNS server.
In this case, the request for www.office.com is resolved in recursor mode (root servers).

4

I still don’t understand why we need to configure additional DNS servers, like Google or OpenDNS, when we already have Knot, which I think is sufficient. If the root servers stop working, Google’s DNS servers will stop working as well.

My point is that you have to make a decision. Either use the IPFire knot or use external DNS. Both options cause confusion with proper resolution.

5

knot and unbound ( used until CU203 ) are no DNS servers but resolvers. A resolver needs assitance from external DNS servers or samples information in recursor mode.

@pscar13 , the addition to the allow list doesn’t bypass the resolver, but the DNS firewall if www.office.com is listed in a blocklist. If the listing is a false positive, the name should be reported to IPFire DNSBL.

6

@bbitsch

That’s exactly what I meant, maybe I accidentally wrote dns instead of resolver.

7

Why do you jump on me every time I post?
I was just replying to @roberto.
I only ran a test, and I never wrote that the resolver was being bypassed.
I said that, in this case, the resolver wasn’t using the redirect to Google’s DNS.

This thread has nothing to do with a false positive.

8

The processing is just another way:

  • if www.office.com is listed in a DNSBL, no server is asked for resolution, the resolver just answers with ‘no IP information’.
  • therefore resolution is not done by asking a defined DNS server or in recursor mode.

Your statement means that an addition to the allow list would circumvent the configured DNS sources. That is not true.

BTW: I ‘jump’ on false postings only.

9

1 - @quin never mentioned DNSDBL.
2 - www.office.com is not on any DBL list.
3 - I ran the test @roberto suggested.

The problem is that if you configure a redirect to DNS 8.8.8.8, the domain www.office.com is no longer resolved by knot-resolver in IPFire.
And if you add www.office.com to the DNS Firewall allowed list, it works.

It is not my habit to speak up without verifying what I am writing.

10

With 8.8.8.8 as single DNS server, I can’t resolve any name. This includes www.office.com :wink:

11

To answer @quin:

To solve this problem, use servers other than Google or none at all (recursor mode) and everything will work.

12

Thank you everybody for the helpful discussions.

I have followed the advice of @pscar13 and found both solutions to work. Thank you!

A summary of the situation would appear to be that knot resolver returns SERVFAIL / EDE 12 (NSEC Missing) for www.office.com when forwarding through Google Public DNS. Resolution succeeds in recursor mode and when forwarding through Cloudflare (1.1.1.1). Direct queries to Google also succeed. So could there perhaps be an interop issue in knot’s local DNSSEC validation of the response variant received through Google, rather than generally invalid Google DNS data?