The .work TLD ironically, doesn't work

Greetings,
Got a new domain name that the TLD is .work. I can’t get it to work through ipfire. Seriously.

[Edit: For the TL;DR crowd, any lookup from a client just disappears and never shows up in logging. However, if I Network -> Domain Name System -> DNS Configuration -> Protocol for DNS queries and make a change, then it works for a VERY short time. TLS, just a few seconds before breaking. TCP, a few minutes before breaking. UDP, a little longer before breaking. What is breaking this lookup? :man_shrugging:]

For example, when I try to dns query about.work I get back:
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

If I try to ping, it takes forever then returns back about.work.mylocal.domain meaning that it isn’t coming back as a TLD but rather being treated as a short name that is searched on mylocal.domain.

If I try to connect to http://about.work/ in Firefox…it just spins until it times out. Open up Tor and http://about.work/ connects immediately with a “under construction” page.

I /did/ have rules preventing any/all DNS queries except through IPFire (using the second method as per the wiki https://wiki.ipfire.org/configuration/firewall/dns). However, I ripped out ALL of them and rebooted IPFire. So now, I can query other servers like 9.9.9.9. Should work now, right?

nope.

Any device on my green or orange interface that tries to query any DNS server for ANY .work domain returns “connection timed out; no servers could be reached”. However, the EXACT same device on either my tether to the cell network or the coffee shop down the street all work just fine with those same .work domain queries.

Ah. Then the DNS servers must not be working… Wrong…Immediate response with 9.9.9.9 and google.com (or any other domain I care to look up).

$ dig @9.9.9.9 google.com
; <<>> DiG 9.11.3-1ubuntu1.13-Ubuntu <<>> @9.9.9.9 google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 11214
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;google.com.			IN	A
;; ANSWER SECTION:
google.com.		146	IN	A	172.217.15.78
;; Query time: 35 msec
;; SERVER: 9.9.9.9#53(9.9.9.9)
;; WHEN: Wed Sep 16 23:55:08 CDT 2020
;; MSG SIZE  rcvd: 55

So on my Green/Orange network I can query a slew of DNS servers now (I’ve tested MANY) with a slew of .com/.org/.net domains and they all work. Query ANY of the .work domains and none of them work.

Fine. Let’s set some rules to treat .work as a pass through…nope. None of those worked.

What about the IPFire box itself? If I log in as root via SSH and do a dig, it works immediately. Even when I query the server itself! Gah!!

I’m going nuts looking at the log files. Anyone know what is going on? Is this just me? Or is this an issue with unbound?

Thanks!

Using core 148, I have no problems in pinging about.work nor displaying http://about.work in Firefox from my PC. Pinging from IPFire console works also.
I’m using the DNS servers of my provider (German Telekom).

Thanks for checking. I’ve been poking at it again this morning and I’m at a loss… It doesn’t matter which DNS servers I use in IPFire, the query never makes it through.

I dig/ping/http about.work w/o issues.

[root@ipfire ~]# dig about.work

; <<>> DiG 9.11.20 <<>> about.work
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36851
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;about.work.			IN	A

;; ANSWER SECTION:
about.work.		145	IN	A	103.67.235.120

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Thu Sep 17 07:34:00 MST 2020
;; MSG SIZE  rcvd: 55

[root@ipfire ~]# ping -c3 about.work
PING about.work (103.67.235.120) 56(84) bytes of data.
64 bytes from sp-hosting01.per01.ds.network (103.67.235.120): icmp_seq=1 ttl=50 time=215 ms
64 bytes from sp-hosting01.per01.ds.network (103.67.235.120): icmp_seq=2 ttl=50 time=215 ms
64 bytes from sp-hosting01.per01.ds.network (103.67.235.120): icmp_seq=3 ttl=50 time=211 ms

--- about.work ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 211.688/214.193/215.787/1.871 ms
[root@ipfire ~]#

So I continue to poke. Updated my unbound.conf to set verbosity to the highest level of 5. When I do a dig @my.ipfire.ip.address ipfire.org I get a whopping 553 lines of unbound messages! Whooo! That’s great debugging.

So what happens when I dig @my.ipfire.ip.address about.work? Zilch. Not a single log line appears. My only response is from dig

; <<>> DiG 9.11.3-1ubuntu1.13-Ubuntu <<>> @my.ipfire.ip.address -q about.work
; (1 server found)
;; global options: +cmd
;; connection timed out; no servers could be reached

If I query another DNS server like 9.9.9.9 it’s fine. If I take the same client to the cell-tether it’s fine. So I don’t think it is the client.

Then, here’s another kicker… If I do a dig against about.work2, there is an IMMEDIATE log in unbound as it tries to find and ultimately returns nothing.

It’s almost as if there’s a blacklist against the .work domain somewhere…but I can’t find it anywhere…

Thoughts?

Greetings,
Thanks for checking. It works perfect if I log into my IPfire system. It is only the clients behind the firewall that have issues. Could you test from a client please?
Thanks

just tested from a client behind my ipfire, dig/ping/http work.

Thanks for testing. I have no idea what is going on. Anything I try in the .work TLD is just like disappearing into a black hole and I have no idea what is doing it. I’m not even finding any drops in the firewall.

can you provide a screenshot of your ipfire DNS page?

This is how it looks now…I’ve been swapping them out like crazy just to try different things. But as I said above, unbound never sees the request.

try restarting unbound, /etc/init.d/unbound restart

I used dig @dns about.work for each of the 3 dns I see, all resolved correctly.

Thanks. Restarting unbound didn’t help.
So I setup tcp dump to watch the green interface on the ipfire system with this: tcpdump -s0 -n -vvv -S -i green0 port 53

A request for ipfire.org, immediate response.
11:35:51.144001 IP (tos 0x0, ttl 64, id 13336, offset 0, flags [none], proto UDP (17), length 79)
192.168.1.20.48896 > 192.168.1.1.53: [udp sum ok] 32064+ [1au] A? ipfire.org. ar: . OPT UDPsize=4096 (51)
11:35:51.144555 IP (tos 0x0, ttl 64, id 24114, offset 0, flags [none], proto UDP (17), length 83)
192.168.1.1.53 > 192.168.1.20.48896: [udp sum ok] 32064$ q: A? ipfire.org. 1/0/1 ipfire.org. [7m59s] A 81.3.27.38 ar: . OPT UDPsize=1232 (55)

A request for about.work has three attempts, then fail out.
11:35:51.152076 IP (tos 0x0, ttl 64, id 13338, offset 0, flags [none], proto UDP (17), length 79)
192.168.1.20.60669 > 192.168.1.1.53: [udp sum ok] 56469+ [1au] A? about.work. ar: . OPT UDPsize=4096 (51)
11:35:56.152021 IP (tos 0x0, ttl 64, id 14282, offset 0, flags [none], proto UDP (17), length 79)
192.168.1.20.60669 > 192.168.1.1.53: [udp sum ok] 56469+ [1au] A? about.work. ar: . OPT UDPsize=4096 (51)
11:36:01.152177 IP (tos 0x0, ttl 64, id 14953, offset 0, flags [none], proto UDP (17), length 79)
192.168.1.20.60669 > 192.168.1.1.53: [udp sum ok] 56469+ [1au] A? about.work. ar: . OPT UDPsize=4096 (51)

:man_shrugging:

my tcpdump, you can see the A record

09:47:14.062996 IP (tos 0x0, ttl 64, id 5773, offset 0, flags [none], proto UDP (17), length 83)
10.0.0.1.53 > 10.0.0.26.49349: [udp sum ok] 28219 q: A? about.work. 1/0/1 about.work. [4m13s] A 103.67.235.120 ar: . OPT UDPsize=1232 (55)

Oh…Geez… :man_facepalming:

So…uh…I know the problem - Well, more specifically I have some useful information…Not sure how to fix it yet though…I saw that blog post a while back on how to make IPFire DNS queries more secure…and I did it…Which means Network -> Domain Name System -> DNS Configuration -> Protocol for DNS queries -> set to TLS.

If I move that over to TCP, guess what? It works! Every single one of the .work domains I’ve been testing works…but then after a minute or so…they ALL stop working again… :man_shrugging:

Setting them to UDP and they seem to also be working…but they haven’t stopped functioning either…

This makes no sense…

In the DNS GUI when someone changes " Protocol for DNS queries" and hits save, what processes kick off?

Because TLS works if I /immediately/ query…but after a few minutes it breaks again. TCP works after a save but roughly 5 minutes later it stops working…UDP works after a save but roughly 10 minutes later it stops working…

Once it stops working, it is just a /dev/null void…I can’t find what is happening to the query. So I wonder what services are being reset that maybe I can dig into more…I mean, I know unbound, but I’ve already got the logging up as high as I can and once it goes weird I don’t see ANYTHING.

Well. I tried a few things and reverted those changes…but now when I restart unbound it seems to be consistent in that it will respond with a DNS lookup in the log files in the first three minutes…after that…nothing…The packet comes in and just disappears… :man_shrugging:

OH!!! A HUGE step forward in progress!!!

So just as a fluke because I’m watching ALL the logs go by ( I made a few iptable rules to just CRANK up the logging trying to figure out what was sending these packets to /dev/null) I noticed something…Right after I restart unbound and everything is working I don’t see anything unusual…but then about 3 to 5 minutes after I notice this:

suricata: rule reload complete
suricata: Signature(s) loaded, Detect thread(s) activated.

And IMMEDIATELY I can no longer query anything on the .work TLD. So… Firewall -> Intrusion Prevention and I disabled it for the Green and I watched suricata reload and IMMEDIATELY I was able to query again…

So…something in the Intrusion Detection rules is breaking things for me…Now to figure out what and how to watch suricata logs…

Found it! ZOMG! So that suricata lead was EXACTLY what I was after. Geez…I really wish the IDS was clearer on what it is doing…because that was so much time and effort to figure out…then again…I figured it out so at least I learned something…

Anyway. Went digging. I’ve got most of the IDS rules turned on. So checking one by one would be time consuming. Instead I downloaded off of the rules from emergingthreats.net. Then ran a regex looking for the .work domain. Guess what I found…

emerging-info.rules:alert dns $HOME_NET any -> any any (msg:"ET INFO Observed DNS Query to .work TLD"; dns_query; content:".work"; nocase; endswith; reference:url,www.spamhaus.org/statistics/tlds/; classtype:bad-unknown; sid:2027868; rev:3; metadata:affected_product Any, attack_target Client_Endpoint, created_at 2019_08_13, deployment Perimeter, former_category INFO, signature_severity Major, updated_at 2019_09_28;)

and

emerging-info.rules:alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"ET INFO HTTP Request to Suspicious *.work Domain"; flow:established,to_server; content:".work"; fast_pattern; http_host; endswith; reference:url,www.spamhaus.org/statistics/tlds/; classtype:bad-unknown; sid:2027877; rev:3; metadata:created_at 2019_08_13, deployment Perimeter, former_category HUNTING, performance_impact Low, signature_severity Minor, updated_at 2019_09_28;)

So I disabled just that one emerging-info rule…and it all works now! :partying_face:

Guess I need to figure out what I need to do to get that lifted.

3 Likes

Jeeez.
@stack, that makes sense.
I’m just using a limited set of suricata rules, so your .work problem didn’t show up here.
Had a similar problem lately concerning tor, not working as expected because the tor suricata rule was activated. After disabling it, tor worked fine again.

Do you have any tips for debugging when a suricata rule is blocking something that I want to allow?
Thanks!