DNS queries for arduino.cc are dropped by IPS

I am now on the MiniPC which has Core154 installed.
Everything works! It’s some gremlin in Core 160.

root@sdrbox1:~# aptitude update
Hit Bytemark - Mirror impish InRelease
Hit Bytemark - Mirror impish-updates InRelease
Hit Bytemark - Mirror impish-security InRelease
Hit Bytemark - Mirror impish-backports InRelease
Get: 1 Index of /ubuntu impish-security InRelease [103 kB]
Get: 2 Index of /ubuntu impish-security/main i386 Packages [20.7 kB]
Get: 3 Index of /ubuntu impish-security/main amd64 Packages [81.0 kB]
Get: 4 Index of /ubuntu impish-security/main Translation-en [20.5 kB]
Get: 5 Index of /ubuntu impish-security/main amd64 DEP-11 Metadata [4,680 B]
Get: 6 Index of /ubuntu impish-security/universe i386 Packages [11.5 kB]
Get: 7 Index of /ubuntu impish-security/universe amd64 Packages [15.3 kB]
Get: 8 Index of /ubuntu impish-security/universe Translation-en [8,204 B]
Get: 9 Index of /ubuntu impish-security/multiverse i386 Packages [1,080 B]
Get: 10 Index of /ubuntu impish-security/multiverse amd64 Packages [1,088 B]
Get: 11 Index of /ubuntu impish-security/multiverse Translation-en [324 B]
Get: 12 Index of /ubuntu impish-security/restricted i386 Packages [12.9 kB]
Get: 13 Index of /ubuntu impish-security/restricted amd64 Packages [73.8 kB]
Get: 14 Index of /ubuntu impish-security/restricted Translation-en [10.4 kB]
Fetched 365 kB in 2s (185 kB/s)

Current status: 1 (+1) upgradable, 66324 (+217) new.
root@sdrbox1:~# aptitude full-upgrade
The following packages will be upgraded:
linux-libc-dev
1 packages upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 1,287 kB of archives. After unpacking 8,192 B will be used.
Do you want to continue? [Y/n/?]
Get: 1 Index of /ubuntu impish-security/main amd64 linux-libc-dev amd64 5.13.0-21.21 [1,287 kB]
Fetched 1,287 kB in 1s (1,402 kB/s)
(Reading database … 322418 files and directories currently installed.)
Preparing to unpack …/linux-libc-dev_5.13.0-21.21_amd64.deb …
Unpacking linux-libc-dev:amd64 (5.13.0-21.21) over (5.13.0-20.20) …
Setting up linux-libc-dev:amd64 (5.13.0-21.21) …

Current status: 0 (-1) upgradable.
root@sdrbox1:~#

Last night I installed Ubuntu 20.04.3 as a vm on my virtualbox test bed together with a CU 160 IPFire that has been running for some time.

This morning I just ran apt -u update followed by apt -u upgrade and the update and upgrade were successfully carried out with no hiccups.

Still absolutely fine on Core 154 on the MiniPC. I’ll probably swap back to the PC and try Core 161 Beta later in the week.
root@sdrbox1:~# aptitude update
Hit Bytemark - Mirror impish InRelease
Hit Bytemark - Mirror impish-updates InRelease
Hit Bytemark - Mirror impish-security InRelease
Hit Bytemark - Mirror impish-backports InRelease
Hit Index of /ubuntu impish-security InRelease

root@sdrbox1:~# aptitude full-upgrade
No packages will be installed, upgraded, or removed.
0 packages upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B of archives. After unpacking 0 B will be used.

root@sdrbox1:~#

FIXED:- The mirror directories were all wrong for aarch64, needed to point to ubuntu-ports.

http://mirror.bytemark.co.uk/ubuntu
Changed to:
http://gb.ports.ubuntu.com/ubuntu-ports/ubuntu-ports/

I am now on core-163 and still experiencing problems. The problem originally seen on Core-160.
If I disable the LAN connection on any PC/SBC and connect to my cable modem via Wifi there is no problem.
PC → IPfire → Cable Modem → (PROBLEM)
PC → Cable Modem via WiFi (AOK).
root@sdrbox1:~# nslookup www.arduino.cc
Server: 127.0.0.53
Address: 127.0.0.53#53

Non-authoritative answer:
www.arduino.cc canonical name = www.arduino.cc.cdn.cloudflare.net.
Name: www.arduino.cc.cdn.cloudflare.net
Address: 104.18.28.45
Name: www.arduino.cc.cdn.cloudflare.net
Address: 104.18.29.45
Name: www.arduino.cc.cdn.cloudflare.net
Address: 2606:4700::6812:1c2d
Name: www.arduino.cc.cdn.cloudflare.net
Address: 2606:4700::6812:1d2d

root@sdrbox1:~# dig www.arduino.cc

; <<>> DiG 9.16.15-Ubuntu <<>> www.arduino.cc
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1400
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;www.arduino.cc. IN A

;; ANSWER SECTION:
www.arduino.cc. 187 IN CNAME www.arduino.cc.cdn.cloudflare.net.
www.arduino.cc.cdn.cloudflare.net. 187 IN A 104.18.28.45
www.arduino.cc.cdn.cloudflare.net. 187 IN A 104.18.29.45

;; Query time: 3 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Thu Feb 17 00:24:35 GMT 2022
;; MSG SIZE rcvd: 122

On the IpFire box (No Wifi) – nslookup www.arduino.cc or dig www.arduino.cc return – Connection timed out: no server could be reached.
This is with DNS set to UDP, TLS or TCP.

I assume sdrbox1 is your ipfire installation.

;; Query time: 3 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)

from my ipfire, digs shows:
;; Query time: 21 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)

cat /etc/hosts
127.0.0.1 localhost

Why do you have 127.0.0.53 instead of 127.0.0.1 ?
(systemd-resolved does this trick but ipfire has no systemd)

1 Like

Does this issue only happen with www.arduino.cc or with any website. Are all websites timing out when you try to access via your IPFire?

As your DNS servers are showing no problems try dig dns.google on your CU163 IPFire. I get the following.

-bash-5.1$ dig dns.google

; <<>> DiG 9.16.22 <<>> dns.google
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 26269
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;dns.google.			IN	A

;; ANSWER SECTION:
dns.google.		44	IN	A	8.8.8.8
dns.google.		44	IN	A	8.8.4.4

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Thu Feb 17 10:50:06 CET 2022
;; MSG SIZE  rcvd: 71

Hi @anon42188109,

Looking back over this thread I have found that sdrbox1 is a pc running Ubuntu

I believe that the testing of the dig commands is being done from a lan pc with IPFire either in place or out of loop.

Looking back over the thread it seems to be more related to problems in having apt-get to work properly.
@sboyce should confirm if that is still the problem being experienced.

If yes then this could be related to a bug that Ubuntu and Debian have (or have had, I don’t know if it has been fixed in later releases) where if an ipv6 communication occurs from the dns system then Ubuntu will only try to update via ipv6 even if an ipv4 address is provided.
See thread https://community.ipfire.org/t/dns-returning-ipv6-addresses-can-cause-breakage/5595

I ran a test with Ubuntu 20.04.3 which is an LTS release and had no problems with running apt-get update or upgrade. From the thread it looks @sboyce is on Ubuntu Impish which is 21.10, an interim release.
Later on I will test out installing Ubuntu Impish on a vm and see if I have a similar problem or not. I will also test it with aptitude update which is what was used by @sboyce but as far as I am aware aptitude just is a higher level operation of apt-get so the same final program is run.

3 Likes

So I have installed Ubuntu Impish (21.10) on a vm on my vm network which has a vm IPFire running Core Update 163.

I stopped the install doing any package or security updates.

I the successfully installed aptitude and ran it. It highlighted 53 upgradable packages and these were then successfully updated with no issues making connections to the Ubuntu repository.

I did an Ubuntu Server install with no additional packages specified when the list of additional packages was shown.

@sboyce have you modified the config of aptitude or apt-get. So far I have been unable to duplicate the issue you are seeing.

2 Likes

Nothing has been modified on any of the numerous boxes I have. It all happened since I upgraded to Core160 as well as the current MiniPC with Core160 upgraded to Core163.

They are all on the same Gigabit switch getting DHCP addresses from the IPFire box.

The raspios and Ubuntu PC and SBC’s all show the same problem, the openSUSE boxes all cannot resolve ardiono.cc.
openSUSE Tumbleweed x86_64

slipstream:~ # dig www.arduino.cc

; <<>> DiG 9.16.25 <<>> www.arduino.cc
;; global options: +cmd
;; connection timed out; no servers could be reached

slipstream:~ #
However, no problem with the Fedora 35 x86_64 box.
[root@Fedora ~]# dig www.arduino.cc

; <<>> DiG 9.16.24-RH <<>> www.arduino.cc
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 26255
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;www.arduino.cc. IN A

;; ANSWER SECTION:
www.arduino.cc. 170 IN CNAME www.arduino.cc.cdn.cloudflare.net.
www.arduino.cc.cdn.cloudflare.net. 170 IN A 104.18.28.45
www.arduino.cc.cdn.cloudflare.net. 170 IN A 104.18.29.45

;; Query time: 2 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Mon Feb 21 17:28:53 GMT 2022
;; MSG SIZE rcvd: 122

[root@Fedora ~]#

slipstream:~ # ip r
default via 192.168.10.103 dev enp3s0 proto dhcp metric 100
default via 192.168.0.1 dev wlp9s0f3u4u1 proto dhcp metric 600
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
192.168.0.0/24 dev wlp9s0f3u4u1 proto kernel scope link src 192.168.0.11 metric 600
192.168.10.0/24 dev enp3s0 proto kernel scope link src 192.168.10.224 metric 100
192.168.100.0/24 dev virbr1 proto kernel scope link src 192.168.100.1 linkdown
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown
slipstream:~ #

[root@Fedora ~]# ip r
default via 192.168.10.103 dev eno1 proto dhcp metric 100
default via 192.168.0.1 dev wlp2s0 proto dhcp metric 600
192.168.0.0/24 dev wlp2s0 proto kernel scope link src 192.168.0.13 metric 600
192.168.10.0/24 dev eno1 proto kernel scope link src 192.168.10.220 metric 100
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown
[root@Fedora ~]#

The fact that you have the problem also with OpenSUSE rules out any issue with the apt-get bug because OpenSUSE uses YaST.

Can you confirm if you only have a problem with certain websites like the Ubuntu Package Update servers and www.arduino.cc but other websites can be accessed with no problems on those affected OS’s or are they unable to access any websites?

Hi,

running an OpenSuSE machine behind IPFire as well, I just tried to reproduce the issue with www.arduino.cc (while I did not experience notable DNS issues on that operating system combination before).

Did not work tough…

$ dig soa www.arduino.cc

; <<>> DiG 9.16.6 <<>> soa www.arduino.cc
;; global options: +cmd
;; connection timed out; no servers could be reached

… but for a different reason:

[root@maverick ~]# grep ".cc TLD" /var/log/suricata/fast.log
02/21/2022-23:15:43.594855  [Drop] [**] [1:2027758:5] ET DNS Query for .cc TLD [**] [Classification: Potentially Bad Traffic] [Priority: 2] {UDP} x:37883 -> x:53
02/21/2022-23:15:48.588608  [Drop] [**] [1:2027758:5] ET DNS Query for .cc TLD [**] [Classification: Potentially Bad Traffic] [Priority: 2] {UDP} x:37883 -> x:53
02/21/2022-23:15:53.590232  [Drop] [**] [1:2027758:5] ET DNS Query for .cc TLD [**] [Classification: Potentially Bad Traffic] [Priority: 2] {UDP} x:37883 -> x:53

While this might be a bit off-topic, I think it makes sense to double-check the IPS configuration, to avoid it to drop DNS queries (in addition to the actual bug we are trying to track down here). Just thought I’d mention that… :slight_smile:

Thanks, and best regards,
Peter Müller

3 Likes

No problem with any other website.
Fedora 35 does something smart, if both LAN and Wifi are up, it gets to www.arduino.cc via Wifi, fails the same as every other box if Wifi is down.

From IPfire

[root@ipfire ~]# ping www.arduino.cc
ping: www.arduino.cc: Name or service not known
[root@ipfire ~]#

Disabling Intrusion Prevention System solves the problem. Problem returns if IPS is enabled.
All rulesets enabled.

So all boxes are having problems reaching www.arduino.cc

As Peter hinted at, it would be good to eliminate some other aspects. Do you have IPS running and blocking failed rules that it finds? If IPS is enabled then try disabling it and then see if you can access www.arduino.cc

See previous post, Peter was right.
Everything works, updates of Ubuntu x86_64 and SBC’s.
ping www.arduino.cc on IPFire box works.
It could be one of the enabled rulesets.
I may have a play later by disabling all and enabling one at a time to see if one or more causes it.

2 Likes

I disabled all the rulesets, enabled and applied 6 at a time. Now with all enabled there is no problem.
If it returns I know where to look.

Hi,

glad to have helped. :slight_smile:

Yes, this is a good idea.

Such a procedure is called “baselining”, and especially important prior to running an IPS in a corporate network. First, you try to find out which IPS rules normally trigger (for example, by enabling the “monitor only mode”), to get an idea which ones will cause false positives, and which ones won’t. Simultaneously, you gradually enable try to get a feeling on what is “normal” in your network.

Second, you gradually enable the rulesets you want, going from those likely not to cause any issues to the more delicate ones. Depending on the network’s size and how well everything is documented, even middle-sized companies can need months until they got their IPS fully operational in production.

Hm, interesting. This might be due to DNS caching issues…

Thanks, and best regards,
Peter Müller

2 Likes

Most likely. When all rulesets are eventually enabled and applied everything is fine for a matter of minutes before reverting to be a problem.

Hi,

well, considering the TTL of 300 seconds (= 5 minutes), this does not come at a surprise:

$ dig a www.arduino.cc

; <<>> DiG 9.16.22-Debian <<>> a www.arduino.cc
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 3795
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;www.arduino.cc.			IN	A

;; ANSWER SECTION:
www.arduino.cc.		300	IN	CNAME	www.arduino.cc.cdn.cloudflare.net.
www.arduino.cc.cdn.cloudflare.net. 300 IN A	104.18.28.45
www.arduino.cc.cdn.cloudflare.net. 300 IN A	104.18.29.45

;; Query time: 124 msec
;; SERVER: 172.28.1.1#53(172.28.1.1)
;; WHEN: Wed Feb 23 16:13:17 UTC 2022
;; MSG SIZE  rcvd: 122

To reflect the actual issue better, I will rename this thread. Also, I consider it solved unless there are some aspects not being clarified yet.

Thanks, and best regards,
Peter Müller

1 Like