DNS is hijacked by ISP and IPfire cannot fix it

I am finally at network where ISP hijacks DNS and I can test IPfire on this network. I hoped I will see that when I use IPfire, I can reach DNS without censorship but it seems it doesn’t work, DNS is still hijacked.

My IPfire is version 183 and running at RPI. It has enabled one DNS server, “dns.quad9.net”, DNS server is 9.9.9.9. When I run validation test, I see green OK and note that DNS server is DNSSEC validating.

I know that ISP is transparently hijacking DNS traffic, I hope they do that to protect customers from some viruses those modify DNS configuration.

There is a simple test to check if DNS is hijacked:

$ host on.quad9.net 9.9.9.9
Using domain server:
Name: 9.9.9.9
Address: 9.9.9.9#53
Aliases: 

on.quad9.net is an alias for no.quad9.net.
no.quad9.net has address 216.21.3.104
no.quad9.net has IPv6 address 2620:0:871:9000::104

When the answer has line on.quad9.net is an alias for no.quad9.net., it is a sign that DNS traffic is hijacked. When DNS traffic is not hijacked, I see on.quad9.net has address a.b.c.d

I hoped that when I will login to IPfire console and run host on.quad9.net, I will receive answer that shows that DNS traffic was not hijacked (because DNS of IPfire was configured to use “QUAD9 DNS server”. Unfortunately, I see answer with on.quad9.net is an alias for no.quad9.net, DNS traffic is still hijacked… :frowning:

IPfire doesn’t solve the problem of hijacked DNS traffic. The computers in the LAN connected to IPfire cannot pass test at page “https://on.quad9.net/
Do I miss something?

Other test for DNS hijacking is this one:

dig +short ch txt id.server. @9.9.9.9

When the command returns empty answer, DNS traffic is hijacked…
Details are described in the FAQ


Other interesting observation. It seems like IPfire cannot resolve “.net” hosts when I disable all DNS servers in IPfire; I force IPfire to DNS recursive mode. I am not sure what is source of this problem, it could be related to the issue that DNS traffic is hijacked by the ISP. DEMO:

I cannot resolve “.net”:

[root@rpifire ~]# host quad9.net
Host quad9.net not found: 2(SERVFAIL)

I can resolve it when I use other well known DNS; well known is important, some small DNS servers are blocked by ISP…

[root@rpifire ~]# host quad9.net 1.1.1.1
Using domain server:
Name: 1.1.1.1
Address: 1.1.1.1#53
Aliases: 

quad9.net has address 216.21.3.77
quad9.net has IPv6 address 2620:0:871:9000::77
quad9.net mail is handled by 5 mx1.quad9.net.
quad9.net mail is handled by 20 mx2.quad9.net.
quad9.net mail is handled by 10 mx4.quad9.net.

When I try other domain, like ‘.one’, no issue:

[root@rpifire ~]# host one.one.one.one
one.one.one.one has address 1.1.1.1
one.one.one.one has address 1.0.0.1
one.one.one.one has IPv6 address 2606:4700:4700::1111
one.one.one.one has IPv6 address 2606:4700:4700::1001

The same for .org:

[root@rpifire ~]# host linux.org 
Host linux.org not found: 2(SERVFAIL)
[root@rpifire ~]# host linux.org 1.1.1.1
Using domain server:
Name: 1.1.1.1
Address: 1.1.1.1#53
Aliases: 

linux.org has address 172.67.73.26
linux.org has address 104.26.14.72
linux.org has address 104.26.15.72
linux.org has IPv6 address 2606:4700:20::681a:e48
linux.org has IPv6 address 2606:4700:20::681a:f48
linux.org has IPv6 address 2606:4700:20::ac43:491a
linux.org mail is handled by 10 mx1.improvmx.com.
linux.org mail is handled by 20 mx2.improvmx.com.

UPDATE. I fixed it! I missed something important. My IPfire was using UDP protocol to query DNS servers. I switched DNS to TLS mode and I see that my queries are not hijacked anymore. Good… I run my IPfire on RPI without battery, I assume I will have a serious problem after next reboot, when correct time will be lost and IPfire will not be able to fetch correct time from the internet because TLS protocol requires correct time at host to resolve DNS name of NTP server… :frowning:

DNS was switched to TLS mode and I can pass “quad9” hijacking test:

[root@rpifire ~]# host on.quad9.net
on.quad9.net has address 216.21.3.77
on.quad9.net has IPv6 address 2620:0:871:9000::77
Host on.quad9.net not found: 2(SERVFAIL)

For some time now, IPFire has recognised that SBC often don’t have battery backed settings - consequently IPFire syncronises time on boot.

I know that. But when TLS is used for DNS queries, time has to be correct otherwise TLS certificate is not valid… And when NTP server is configured with a hostname, it cannot be translated to IP address because time is not correct… I have not checked details of time synchronisation process in IPfire, it is possible that an fallback IP address of some NTP server is hardcoded in the synchronization script… I have to test it :wink:

UPDATE: I tried the reboot and I cannot login to WiFi AP at RPI, it is gone, not in the air. I assume IPfire is waiting in a loop for correct time from NTP server… I have to find some ethernet cable, to login from LAN and check details and fix configuration… NTP servers were configured with 0.ipfire.pool.ntp.org and 1.ipfire.pool.ntp.org

In case of startup problems I recommend to use the a very ‘simple’ system access.
This means either a serial connection or a KVM ( keyboard, video, mouse ) access.
IPFire usually installs a console shell on this native interface. Most ( all, usable for IPFire ) systems have such an interface.

Using an access method based on ethernet/IP demands drivers/programs for this are running.

Keep It Simple: don’t use hostnames for NTP unless strictly necessary, bypass the need for a DNS query.

ISP owns the connection, you’re simply on loan. I hope that you will succeed in your task but if the ISP cut down again your “other DNS access” (it’s quite simple to route elsewere a simple public list of services) your only option will be a DNS on the other side of a VPN connection or… via TOR.

1 Like

I am not in my lab, I am on a trip. I am limited in troubleshooting options, I do not have KVM or serial console in my “backpack”. Connection with Ethernet cable was not working, I am not sure if the issue was with Ethernet cable or IPfire just not reached full running state (DHCP server was not started). I found microSDHC card reader and I was able to modify configuration on RPI microSDHC card, so I am back in the business but I am not in the mood to run more experiments; “IPfire is easy to break”… :frowning:

I am disappointing with the quality of connection that I get when I connect to RPI. When I connect my notebook directly to the WiFi router (it is about 10 m away, one wall between me and the router), I have reliable connection. When I connect to the same router RPI with IPfire and I connect to RPI, I see that connection is not stable, I just observe connection issues, DNS sometimes doesn’t work, pages with more data are loading slowly, just not good, it is not good user experience it works for a while then it doesn’t work for a minute. This could be related to RPI hardware. I am in the environment where are many WiFi networks around me, about 20. And I assume that WiFi hardware at RPI with small antenna on PCB just fails to create reliable link. Connection at 2.4GHz is much better than connection at 5.0 GHz, but still not good enough. There was a router from TP-Link in the past in this location but it was not able to cover this area reliably with the WiFi signal (802.11n, two antennas 5dBi). Then I replaced router with dual band router “Linksys”, it has 3 antennas and even antennas are smaller than antennas at TP-Link router, signal coverage was better, I do not see any issue when I work on my notebook. RPI (WiFi client) just cannot create stable link here… I cannot explain why LinkSys router created better WiFi coverage, what is the difference. Are antennas better quality? Or is advanced modulation used in WiFi transmitter/receiver? This is a magic and I do have access to special analyzer to answer these questions…

BTW, Wifi client at RPI reports signal strength “Uplink Bit Rate: 50% @ -75 dBm”; this is not good enough in this environment…

I’m sorry for your issue, unfortunately…

I don’t agree with this statement, I’m sorry (and I’m far from being biggest IPFire fan currently).

IPFire is born on a x86 architecture and now it’s ported to several others, working maybe not as bulletproof as on PC.
Moreover: install a firewall distro on a device without RTC and clock battery is far from ideal. And it’s not an opinion: firewall devices like Zyxel appliances have inside a CR2032 battery, even it’s not reported or suggested to check/replace.

Now: would I use Raspberry Pi as hardware for a router distro? No.
Raspberry are wonderful devices with toy and tinker (indeed are not toys at all) however for doing some tasks, they can but its not their bread and butter.
For IPFire a compact x86 box is far more power hungry, however is a far better device:

  • more CPU power (means also more services could be enabled with far less performance hit)
  • more ram
  • less corruptable storage by default (logging on SD card can shorten the life to few months, with a dead device after storage corrupts)
  • RTC, clock battery
  • more bandwith between CPU and peripherals
  • chance to install more adapters if necessary (not all boxes can do that, and device-added Raspi are quite a janky contraption)

IpFire works and works fine on Rasperry, but it’s not the most proper hardware for.

3 Likes

I was using RPI with IPfire in my lab, and in other location during my trip. It worked fine (once DHCP client was broken and it is still broken after OS upgrade) but I have not observed issues with WiFi; I was close to the router at those locations, I was in the same room, just 2-3 m away from WiFi router. I was preparing RPI with IPfire to test location, where I know that ISP hijacks DNS traffic; mission accomplished…

I like that RPI is small but without access to KVM or serial console, it is not easy to “fix it”… The issue is “setup” command that restarts network (when network configuration has to be changed, like to switch from WiFi client to USB2ETH adapter or back), such change cannot be executed when I am connected to WiFi AP (USB dongle). Other bad surprise was that I was not able to fix the issue that NTP server was “not reachable” (switch DNS from UTP to TLS, then shutdown and remove power for a minute).

Battery is good improvement but those have to be replaced from time to time. When battery is weak, user can observe the same issue when he uses TLS to sync time or to connect to DNS. PC will lost correct time and TLS communication will fail because time the the gateway is not correct; there is no fallback supported in IPfire… RPI without backup battery just amplifies this issue, it is visible…

Good test for internet connectivity is to try to play some online game…

Battery intended as CR2032 battery lasts… years. Easily.
Battery power is scarcely used while the device is on, unless a unwise mainboard/circuit diagram.
I found only GX280 from dell, in my experience, that had a “dead battery” before was nice (4 years while still being plugged on and working). On a 8 years installed server CR2032 battery was never requested to be replaced (aka Mainboard signals that the battery is to be replaced.

Normal IPFire should detect a non working dns at boot time and try to sync the time with 81.3.27.46 which is the ipfire timeserver.

unbound initskript:

 283 fix_time_if_dns_fails() {
 284         # If DNS is working, everything is fine
 285         if resolve "0.ipfire.pool.ntp.org" &>/dev/null || \
 286            resolve "1.ipfire.pool.ntp.org" &>/dev/null ; then
 287                 return 0
 288         fi
 289 
 290         # Try to sync time with a known time server
 291         boot_mesg "DNS not functioning... Trying to sync time with ntp.ipfire.org (81.3.27.46)..."
 292         loadproc /usr/local/bin/settime 81.3.27.46
 293 }
 294 
4 Likes

this might help:

2 Likes

RTC module could help, I agree. I should have some RTC module for RPI somewhere, I will try to play with it one once I will be back in my lab. The key question is if RTC module is supported by IPfire for RPI. The point is that those modules are not installed by default and some of them needs some configuration changes, to tell Linux kernel to use them; this was required in the past. I was not playing with them for years, so I am not aware of the current situation. I will try it and report how well is IPfire ready for RTC module, if it works out of the box or some configuration work is needed…


This RTC module is interesting, it has RTC and UART pins and is cheap…

The perfect hat will have a RTC, UART, beeper and RGB LED to show status but I cannot find such hat… Several buttons are always useful to trigger shutdown or reboot, to activate/deactivate red interface, etc. HW watchdog would be nice too.

This was a simple and cool project for IPcop, 4 buttons and status LED on serial port:
https://www.ban-solms.de/t/IPCop-comled.html

If you can not get your time updated when you boot then you have some other issue.

As indicated by @arne_f IPFire, if it can not resolve the timeserver URL’s will get the time from a specified IP, so no domain name resolution is required.
If IPFire is not getting any response back from the specific IP (81.3.27.46) then likely your ISP is not only hijacking your DNS but probably also blocking a range of ports.

If you try your DNS with TLS so that it is encrypted then I would expect your ISP to not be able to hijack it but will probably block the access.

If you have an ISP who is doing all this bad stuff, I would be looking to move to a new ISP, or are all ISP’s like this where you are located.

I agree. I am still on the trip and I do not have access to KVM or serial console, so it is difficult to troubleshoot. The last time I was not able to boot and I cannot explain why (like DNS was not able to resolve DNS of NTP server). I see that IPfire code has a fallback IP address for NTP (and that NTP server works, tested). NTP was working when I was using DNS over UDP, so NTP was not blocked… The next time I should add UART to USB adapter to my backpack…


I think I see what is wrong with that time sync code. Just a guess. I was in the environment where RPI had problems to reliably connect to the AP, to reach internet; too many WiFi networks in the air, small WiFi antenna on RPI PCB, maybe something more). When RPI tried to fetch correct time from NTP server (IP address) and that failed, then there was no other try to fix time, correct?? That could be the explanation why RPI failed to sync time…

I have RTC module for RPI. And it doesn’t work with IPfire core 184.
RPI is RPI-3B. All advises to add RTC module to RPI to keep time when device is powered off are useless…

I have small RTC module that is described in this article and it is not detected by RPI3.

The issue is that I2C is not there.

I2C module is loaded:

[root@rpifire ~]# lsmod | grep i2c_
i2c_bcm2835            16384  0

No I2C device is detected:

[root@rpifire ~]# i2cdetect -y 0
Error: Could not open file `/dev/i2c-0' or `/dev/i2c/0': No such file or directory

[root@rpifire ~]# ls /dev/i2c*
ls: cannot access '/dev/i2c*': No such file or directory

hwclock doesn’t use RTC module, that is expected…

[root@rpifire ~]# hwclock 
hwclock: Cannot access the Hardware Clock via any known method.
hwclock: Use the --verbose option to see the details of our search for an access method.

[root@rpifire ~]# pakfire status
Core-Version: 2.29-aarch64
Core-Update-Level: 184
Last update: 1d 4h 9m 51s ago
Last core-list update: 14m 53s ago
Last server-list update: 15m 1s ago
Last packages-list update: 14m 57s ago
Core-Update available: no
Package-Updates available: 0
Reboot required: no

Adafruit article about RTC


I can activate I2C bus with modprobe i2c-dev, then I can see my RTC chip:

[root@rpifire ~]# lsmod  | grep i2c
i2c_dev                24576  0
i2c_bcm2835            16384  0

[root@rpifire ~]# i2cdetect -y 1
     0  1  2  3  4  5  6  7  8  9  a  b  c  d  e  f
00:                         -- -- -- -- -- -- -- -- 
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
50: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- 
60: -- -- -- -- -- -- -- -- 68 -- -- -- -- -- -- -- 
70: -- -- -- -- -- -- -- --     

Modules for RTC chips are in /lib/modules/6.6.15-ipfire/kernel/drivers/rtc/

This is required to activate RTC chip:

modprobe rtc-ds1307
echo ds1307 0x68 > /sys/class/i2c-adapter/i2c-1/new_device

Then hwclock can work with the chip.


There should be an easy way to add RTC to OS, to add following line to the end of file /boot/config.txt

dtoverlay=i2c-rtc,ds1307

But it doesn’t work, overlay i2c-rtc is missing. And compiler for device tree configurations, dtc is missing in IPfire too… :frowning:

The previous RTC module was based on chip DS3231, it is in 16-pin package and it uses driver rtc-ds1307. Once I have found how to configure it, it worked without issue.

Then I tried RTC module with chip DS1307, it is a small chip in 8-pin package, it uses driver rtc-ds1307. It should use the same configuration but it was not working, hwclock reported error, again and again. I tried different module and I observed the same error, like this:

[root@rpifire ~]# hwclock --verbose
hwclock from util-linux 2.39.2
System Time: 1711239913.721366
Trying to open: /dev/rtc0
Using the rtc interface to the clock.
Last drift adjustment done at 1711232909 seconds after 1969
Last calibration done at 1711232909 seconds after 1969
Hardware clock is on UTC time
Assuming hardware clock is kept in UTC time.
Waiting for clock tick...
ioctl(3, RTC_UIE_ON, 0): Invalid argument
Waiting in loop for time from /dev/rtc0 to change
hwclock: ioctl(RTC_RD_NAME) to /dev/rtc0 to read the time failed: Invalid argument
...synchronization failed

It seems there is a big in rtc-ds1307 driver, it cannot handle correctly some unexpected initial values in chip registers. The fix is easy, hwclock -w writes correct time to the chip and after that hwclock works as expected… It took me long time to find what is wrong, I was looking for a hardware failure…