You’re moving the goalposts. You said “monetary benefit.” No, it’s at substantial monetary cost. But because it’s public-benefit, and I am a member of the public, I am also a beneficiary, as is everyone else who uses it, and everyone else who uses another recursive resolver that has now been forced to deploy DNSSEC and encryption, in order to cross the minimum-standard bar that we set.
Again, my apologies, I’m the infrastructure guy, not one of the DNS guys. Here’s an answer relayed from them:
We don’t answer the NXDOMAIN with the authority bit set. So we don’t claim it is a DNSSEC-signed answer. The correct response will be to eventually answer with either NXDOMAIN with extended error of “blocked by policy” OR SERVFAIL with the same extended error. The debate on which is more appropriate is still unclear.
But, again, the point is that there’s no 100% correct alternative to be implemented until draft-ietf-dnsop-extended-error becomes real. In the mean time, if you really can’t live with the current compromise, you can be a purist, live with malware, and use the non-blocking service rather than the blocking service.
How is that not sophistry? If truth and falsehood make no difference to you, why bother to say anything at all?
What has the authority bit to do with this? This bit has only to be set by the authoritative DNS server of a zone. On a recurser it should never set.
Bill, of course they make a difference to me and I was telling you my interpretation of how I read Peter’s post.
You might disagree and you might think that I am wrong, but that does not mean that I am actually wrong. It still remains my interpretation. And I would be happy to be proven wrong, but you do not have anything else but a claim - and so does Peter.
I would like to thank you to help everyone to understand how you see things behind it. I am sure that there is enough information out there now that everyone can decide whether they want to use this service or not - for whatever reason.
Technically that is what I was outlining before. There is no better alternative to it. But still, the big problem that needs to be solved is how the client can trust that this response was not spoofed by an attacker. That won’t be possible without a signature.
And as Arne mentioned, we believe that unbound struggles with these responses, which might simply be a bug in unbound. Until that is resolved, Quad9 makes your DNS slow.
2 Likes
I’d like to thank Bill for reaching out and being able to not only participate in the forum, but also have a technical deep dive discussion here. Rare that we have someone that can do both of these things well. I’m also disappointed at the responses toward Quad9/Bill.
I’m not sure this is the case? I think Quad9 is slow with IPFire. I run pfSense, OPNsense, and IPFire. IPFire out of the box is the only distro that I have to aggressively tune to get decent DNS performance. I say this not out of animosity, but to try to suggest that you look in to the optimizations and defaults shipping with IPFire.
The blog post on DNS also has an interesting conundrum with DNSSEC. That is, if we are forwarding to a DNSSEC provider, including Quad9, having Unbound do additional DNSSEC validation is wasted and slows down the process. pfSense and OPNsense have the ability to disable DNSSEC, and this comes in handy when using Unbound in forwarding mode with a DNSSEC and DoT provider, such as Quad9. Something else to think about before making these blanket statements that Quad9 makes DNS slow. In my experience, Quad9 is quite fast…when used with a properly tuned unbound that isn’t rigidly forced to use presets that aren’t optimal. For reference, my post on some optimization I have to use with IPFire to get decent performance: Unbound - custom configuration - #13 by anon84413319
One more item I have noticed, and perhaps Bill can elaborate on this. Quad9 does seem more aggressive at closing DoT connections. For instance, when I run openssl s_client -connect '9.9.9.9:853' the connection will close after about 10 seconds. If I do the same thing with CloudFlare, it stays open for over a minute.
Finally, when I’ve installed IPFire fresh out of the box, it seems to run Unbound on a single thread vs spawning a thread for each processor core. I suspect that this single threaded nature and the aggressive connection closing may be at odds with each other. Perhaps something the IPFire devs can also validate more to see if this theory is correct?
1 Like
Hi,
The blog post on DNS also has an interesting conundrum with DNSSEC. That is, if we are forwarding to a DNSSEC provider, including Quad9, having Unbound do additional DNSSEC validation is wasted and slows down the process. pfSense and OPNsense have the ability to disable DNSSEC, and this comes in handy when using Unbound in forwarding mode with a DNSSEC and DoT provider, such as Quad9.
I absolutely disagree. Since you cannot rely on your DNS provider (let’s put it that way) for validating DNSSEC, turning the validation off renders the security you gain from DNSSEC void.
In my experience, Quad9 is quite fast…when used with a properly tuned unbound that isn’t rigidly forced to use presets that aren’t optimal. For reference, my post on some optimization I have to use with IPFire to get decent performance: Unbound - custom configuration
Could we please keep this split so we can talk about possible Unbound configuration optimisations there, while we focus on Quad9 here?
Finally, when I’ve installed IPFire fresh out of the box, it seems to run Unbound on a single thread vs spawning a thread for each processor core. I suspect that this single threaded nature and the aggressive connection closing may be at odds with each other. Perhaps something the IPFire devs can also validate more to see if this theory is correct?
This is correct and intentional, and was introduced with commit 0f0f3ae7dc5da502c1aaf4bb295778d7657a0af5.
Thanks, and best regards,
Peter Müller
Hi Bill,
first, thanks for reaching out and sorry for my late reply on this.
How is that not sophistry?
From my point of view, it is not. Given the sensitivity of DNS traffic, a construct of non-profit organisations ran by or linked to profit-oriented companies looks suspicious. As far as I understand, it is quite easy to become a non-profit organisation in the US (by self declaratory), where elsewhere in the world there are more hurdles to skip in order to become a non-profit.
As far as I am concerned, nobody in this world is doing things for free and altruistic purposes. Especially not if “doing a thing” involves lots of expenses (I assume running Quad9 is not inexpensive), people are not going to reward it and one happen to have access to pretty sensitive data. Sorry, but to me, that does not check out.
If truth and falsehood make no difference to you, why bother to say anything at all?
Of course they make a difference. There is no other way to answer this question (why bother asking it, anyway?). It’s just a thought-terminating cliché.
One more point regarding DNSSEC: Since Quad9 breaks it (for purists, if you like) on 9.9.9.9 et al., and you will have to scroll down a lot on the FAQ page to read about a non-filtering version at 9.9.9.10, I guess it is safe to say Quad9 is breaking DNSSEC.
Thanks, and best regards,
Peter Müller
If you would have read more closely what I was saying, then you would have understood that I was just saying that. Arne has said that he thinks there is a bug. I assume that he is right, but I did not verify this, yet. And once again: I assume that 9.9.9.9 is stalling unbound for some reason until the bug is solved.
You should raise these things on the development mailing list by proposing patches. This is a support forum and not many of the developers follow this very closely.
You will also have to argue why your patch should be accepted. I gave you feedback on here before about things that are very unsafe to configure. They might work well for you in your use-case, but IPFire users come from all sorts of places running IPFire in all sorts of environments. Things that might work for you don’t necessarily work well for others.
This is not the place for this, but in short: This has nothing to do with any performance issues. unbound opens more than one connection per thread. If on your firewall the DNS resolver saturates more than one CPU core you probably have a different problem.
I was specifically talking about a situation where we use a DNSSEC enabled and DoT enabled provider, like Quad9.
I agree with you that it is 100% good policy to leave DNSSEC enabled if we use Unbound in recursor mode, where it uses UDP/53 and queries DNS root servers for results. In this situation, there is no doubt DNSSEC is beneficial and guarantees some level of confidence in the responses that we receive back from the root providers.
However, if we use Unbound in Forwarding mode with DoT, we’re now relying on another resolver such as Quad9 to do the DNSSEC validation for us, and we’re sending/receiving responses to that provider in a secure tunnel, thus removing much of the tampering risk that DNSSEC is supposed to help with when we have plaintext open traffic. What is the benefit of running DNSSEC for Unbound when we are just Forwarding queries securely and not resolving them directly through Unbound in recursor mode?
This was not clear in your previous post. And I would say that you cannot assume that. I do not have any usage figures, but I doubt very much that TLS traffic is the majority - if even anything significant at all.
And I would argue that it still matters to do local validation. You assume that Quad9 (as well as any other DNS provider) is trustworthy and cannot be tempered with. We have already covered what people might or might not find trustworthy, but there is always a chance of infiltration or simply software malfunction. You are creating a single point that needs to be taken over and then many users will have compromised DNS.
Doing validation locally is way more expensive to break for an attacker if they want to compromise many users. Therefore this is no reason for me to disable local validation.
To be clear, Quad9’s position is that everyone should ideally be doing their own DNSSEC validation, and that the current situation of people having to trust recursive resolvers is an unfortunate consequence of OS vendors failing to sufficiently support it in system resolvers.
That said, doing it twice is wasteful, and if it creates performance problems, that’s an excellent reason not to do it.
That suggests the potential for a need for a supported combination of features which includes blocking but does not include DNSSEC validation; a combination we haven’t had any real demand for before.
Yes, that’s correct. A reasonable security posture shouldn’t depend on external DNSSEC validation. In the real world, for most people, a reasonable security posture isn’t achievable because it’s undermined by compromises made by OS and browser vendors.
Quad9’s position has always been that, when it’s not unreasonably difficult to do so, we give people the tools and control that they want, so that they can put together the solutions that work for them. Thus our support for DNScrypt (which isn’t an Internet standard) and DoH (which is an abomination that exists solely to track users as they move between NATs) even though DoT exists and is better.
So, you’re having a performance problem when you force people to double DNSSEC validate on malware-blocked domains… the easy answer is to let people choose not to do that, if they don’t want to. In a perfect world, all DNSSEC validation would be done on the end-node (which isn’t their firewall, either); in the world we actually live in, though, different people want to make different choices when they’re forced to compromise.
Find me one example of a significant non-profit that refuses donations from for-profit companies. And if you can find such an example, tell me how replicating it would benefit anyone.
1 Like
No that is true. But if someone is in your local network and can intercept your traffic losing DNSSEC is probably not your biggest problem. On the Internet side it is a lot easier to temper with things and therefore I think that our users are gaining a lot from doing it on the edge of their own network.
What we are doing here for example is to be not dependent on one donor, but multiple. So not single party is contributing that much money that they want to exercise (maybe even indirectly) control with it.
I tried to frame it as clearly as I could in the previous post.
When we’re using Unbound in forwarding mode, I think it’s a bit of a circular argument to try to discuss DNSSEC. If we choose to forward queries to a specific provider, whoever it may be, we’re assuming some level of trust in that provider already even before we take DNSSEC in to account. If we can securely forward queries to that provider using DoT, and we know that provider also performs DNSSEC on our forwarded queries as they resolve them at the provider, this is about as “safe” an assumption as we’re going to get. The DNSSEC argument seems to get misconstrued with the root resolver/recursor mode argument, which I 100% agree with and DNSSEC is definitely needed there. This is where having more flexibility within IPFire (without having to dig in to a custom .conf file) would be appreciated as we start to bridge the gap from plaintext DNS to encrypted DNS.
I do think its a mistake to classify Quad9 in the same rank as non-DNSSEC providers that also modify bad URLs to paid sites when a user accidentally types in a wrong address. Level3/4.2.2.1 is a good example of this negative behavior and they strip DNSSEC as well. It makes perfect sense to leave this provider on the “not recommended” list.
Quad9 is a non-profit and arguably, provides a huge value by not returning known malware infected domains. And yet the lack of nuance in this group means that we are classifying Quad9 in the same rank as a known bad actor DNS like 4.2.2.1. As a result, this is likely to push people towards Google and CloudFlare if they want to use DNSSEC and DoT. These two providers clearly don’t have a non-profit model and don’t hold themselves publicly accountable in any meaningful way regarding user privacy. I also doubt we’ll have any technical input from anyone working at Google or CloudFlare regarding their service.
In my opinion, this is not a benefit to IPFire users and this community as a whole. The stubborn adherence to the view that “Quad9 breaks DNSSEC” makes no sense. In reality there is no actual standard in place for the malware filtering service that Quad9 provides, and they are simply using the most elegant solution currently available. Ironically, by classifying Quad9 in “not recommended”, we are funneling users toward solutions that will be even less likely to serve in their best interest and this is a disservice, in my opinion.
I’d also like to reiterate that I have a huge appreciation for the folks posting here. I think it is great to see this level of involvement from developers and directors alike! Discussions like these will only help us get to a better solution so once again, thank you everyone for taking the time to post and lets keep this going!
DNSSEC is actually about that I do not need to trust my resolver. It does not stop any third party (like my ISP) from eavesdropping on my DNS requests, but I can be certain to never get a forge response from my resolver if I validate the response. TLS does not guarantee that and it does not protect me against any cache poisoning attacks or from untrustworthy resolvers. I can authenticate that I am using the one that I wanted to use, but I have to trust that the resolver will always response with the “correct” reply.
So TLS and DNSSEC compliment each other and I would not want to use only one.
I cannot cite any research on this, but from my experience DNSSEC creates on considerable overhead on the firewall or the client. So there is no benefit from disabling it. I only see downsides.
The IPFire developers have had this discussion many many times and we have talked about this for hours and hours and users have requested a button to disable DNSSEC many many times. Nobody has been able to make a point that convinced me. The arguments brought forward mainly were “it doesn’t work” which was often MTU-related or bad upstream resolvers. Those are network problems that will cause other issues further down the line and so I do not consider DNS to run fine on a broken network.
I do not think that the list should be understood as such. This is simply a good and “not good” list. It evaluates if a resolver works technically. The reason does not really matter and we didn’t add them to the list.
Again, this is not what this list is doing. We do not “recommend” any of the other resolvers here to be used without further consideration. On top of the page is a short note that makes that clear and links a lengthy blog post that makes clear what potential risks are.
Other people will come to a different conclusion and of course can use those if they like. That all depends on who you are, where you are and who you trust and don’t trust. That is impossible to put into a single list.
You might call that stubborn, but Bill has confirmed this. There is no RFC for this and therefore we are in unchartered territory here. There might be a small consensus on what to do to filter the “really bad guys”, but that isn’t understood by our or any other validating resolver.
Even less likely? So Quad9 is not serving everybody’s best interests? How so?
I only speculated that Google is using the collected data to create user profiles. I have no proof, and I made that very clear, but I would be surprised if they would not use the data. I put that warning out there that this is possible and why wouldn’t this be possible for others, too?
I understand that you guys (and I am right to count @firewell as being affiliated with Quad9?) are working hard for your cause and I am sure that you have many IPFire users happily using 9.9.9.9 - or even better 9.9.9.10 - but I personally am still not going to use your service.
That is totally independent from how you are running it and if you are RFC-conform or not. We made the technical side of this crystal clear and it is in the right section as the page is designed now (if anyone wants to change the list then that is another story).
If anyone uses Google or CloudFlare over Quad9, then they have not read my post (i.e. done their homework). It mentions that there is a risk using mono-cultures - and having only three big resolvers with an n.n.n.n IP address format is not enough diversity for my taste.
I am all for fighting this and diversifying our risk although the world is going the opposite way: We have very few cloud providers, the vast majority of SSL certificates is coming from Let’s Encrypt, we have about one and a half smartphone operating systems.
Too many projects are being hosted on GitHub and every time they go down many software developers drink coffee for the rest of the day. Don’t think about downtimes only now. I am aware how an any cast network works. Think outside of the box: Compromised, faulty, setting a bad example in terms of RFC-conformity.
Open standards are there for everyone to implement and use. Isn’t that a better world?
2 Likes
I agree with you. While I personally find it a bit semantics, ultimately DNSSEC is the only way to be sure. I’ll just say one more time though, when we are in Forwarding mode, DNSSEC or not, we are already placing more trust in the forwarded resolver than if we otherwise used DNSSEC with UDP/53 to the root servers. It’s a balance of security and privacy. I personally trust my ISP (Google) less than I trust a non-google forwarder, so in my case it’s DoT all day long to anyone but Google. If I can get additional malware filtering in there and send queries to a company with a stated goal of not profiting on my data, all the better.
Not affiliated, I just use their service. In the same way I am not affiliated with IPFire but, I do use it and enjoy learning. I was initially a heavy user of CloudFlare DoT when all of this was first made available around April 2018. You’ll also find me in the old forum threads, in fact we had some discussions about DoT a few years ago: IPFire Community
When I see something that looks like a good technical solution, I like to talk about it!
I think we should update the Wiki to include better language. There’s clearly a section at the bottom that says “not recommended”. This will imply to quite a few people that the rest of the providers at the top of the page are, in some way, more viable and/or “safer choices” when in reality, many of those providers are for profit and will not be forthcoming about how they use their DNS metadata.
The general assertion, and one that I agree with by the way, is that a large company offering a “free” service should always be looked at with suspicion. Nothing is truly free and their intentions cannot be fully known. The difference is that we have Google and CloudFlare not being fully open about where the data goes. We have Quad9 who is non-profit, states what data they use and why, and how they use it for malware prevention. To me these are not comparable because if I’m going to use a forwarder, I will choose the one that is likely to be the most reliable (scales well) and is going to be transparent about their logging practices.
I’ve tried using other small DoT services and frankly, it’s a mess. They don’t scale well, the services sporadically fail and reappear, when they work they are slow. The whole thing runs like a highschool science project. I prefer Quad9 because it’s one of the first DNSSEC/DoT providers that has enough scalability to be reliable. I’m here to learn and find the best solution in this evolving space. If you’re aware of an RFC-Compliant service (since that is what you prefer) that has a stable infrastructure, provides worldwide peering with DNSSEC/DoT, and has clearly defined privacy goals, I will jump onboard and use it.
This confuses me. You are advocating for DNSSEC as mandatory (which I can agree with!) however, the 9.9.9.10 service does not provide it? I’m not sure how you are thinking 9.9.9.10 would be better?
Screenshot from Quad9’s FAQ:
I Love this. Egos Aside.
bullets. I love bullets
1 Quad9 DNSSEC to it’s self not to me!?
2 Is DNSSEC to much to ask from Quad9?
3 Is there any elegant way to filter the results? and not cause Ipfire to lag/fault.
4 Is this a bug in Ipfire?
5 Quad9 can you give us a fully qualified DNSSEC response?
6 Quad9 are you legally bound not to?Lord Knows my ISP router blocks port 853.?
Quad9 your service sounds as trustworthy as any.
At our last tests 9.9.9.10 is also DNSSec validating now. So the faq seems incorrect.
Not if the client after the filter validates DNSSec because every answer, also NXDOMAIN has to be signed by the next higher level to make altering the messages detectable. It is the main goal of DNSSec to make the DNS not changeable on the way by a man in the middle.