Frustrating NTP issues

trish · 12 June 2025 06:52

Anyone has any idea of how to troubleshoot NTP sync issues ?
/var/log/messages shows no issues

I am using the standard Firewall rule www.ipfire.org - Force clients to use IPFire DNS Server forwarding all traffic to port 123 and 53 respectively and it’s been working fine for years.

Then suddenly Netgear access points started to drift like 2-3 years off. It’s possible that Netgear expect SNTP or something more sophisticated than NTP.

Now I started to use smart plugs / switches and they just drift 55 years back all day long and never sync the time. There are ESP8285 devices with 1024 KB Flash ROM, so no way they expect TLS or anything sophisticated

Again Windows and Linux Machines have no issues using IPFire as their NTP.

I even hardcode the IPFire IP as ntpserver, or use NIST external IP’s etc so DNS is not an issue here

Smart switches and Netgear AP wouldn’t sync ever, although Firewall log shows that the they are trying every 2-3 minutes.

22:42:09	DNAT	green0	UDP	192.168.1.180	62892	192.168.1.1	123
22:42:09	INPUTFW	green0	UDP	192.168.1.180	62892	192.168.1.1	123
22:43:30	DNAT	green0	UDP	192.168.1.181	54714	192.168.1.1	123
22:43:30	INPUTFW	green0	UDP	192.168.1.181	54714	192.168.1.1	123
22:44:24	DNAT	green0	UDP	192.168.1.180	62892	192.168.1.1	123
22:44:24	INPUTFW	green0	UDP	192.168.1.180	62892	192.168.1.1	123
22:45:45	DNAT	green0	UDP	192.168.1.181	54714	192.168.1.1	123
22:45:45	INPUTFW	green0	UDP	192.168.1.181	54714	192.168.1.1	123
22:46:39	DNAT	green0	UDP	192.168.1.180	62892	192.168.1.1	123
22:46:39	INPUTFW	green0	UDP	192.168.1.180	62892	192.168.1.1	123
22:48:00	DNAT	green0	UDP	192.168.1.181	54714	192.168.1.1	123
22:48:00	INPUTFW	green0	UDP	192.168.1.181	54714	192.168.1.1	123
22:48:54	DNAT	green0	UDP	192.168.1.180	62892	192.168.1.1	123
22:48:54	INPUTFW	green0	UDP	192.168.1.180	62892	192.168.1.1	123
22:50:15	DNAT	green0	UDP	192.168.1.181	54714	192.168.1.1	123
22:50:15	INPUTFW	green0	UDP	192.168.1.181	54714	192.168.1.1	123
22:51:09	DNAT	green0	UDP	192.168.1.180	62892	192.168.1.1	123
22:51:09	INPUTFW	green0	UDP	192.168.1.180	62892	192.168.1.1	123
22:52:30	DNAT	green0	UDP	192.168.1.181	54714	192.168.1.1	123
22:52:30	INPUTFW	green0	UDP	192.168.1.181	54714	192.168.1.1	123
22:53:24	DNAT	green0	UDP	192.168.1.180	62892	192.168.1.1	123
22:53:24	INPUTFW	green0	UDP	192.168.1.180	62892	192.168.1.1	123
22:54:45	DNAT	green0	UDP	192.168.1.181	54714	192.168.1.1	123
22:54:45	INPUTFW	green0	UDP	192.168.1.181	54714	192.168.1.1	123
22:55:39	DNAT	green0	UDP	192.168.1.180	62892	192.168.1.1	123
22:55:39	INPUTFW	green0	UDP	192.168.1.180	62892	192.168.1.1	123
22:57:00	DNAT	green0	UDP	192.168.1.181	54714	192.168.1.1	123
22:57:00	INPUTFW	green0	UDP	192.168.1.181	54714	192.168.1.1	123
22:57:54	DNAT	green0	UDP	192.168.1.180	62892	192.168.1.1	123
22:57:54	INPUTFW	green0	UDP	192.168.1.180	62892	192.168.1.1	123
22:59:15	DNAT	green0	UDP	192.168.1.181	54714	192.168.1.1	123
22:59:15	INPUTFW	green0	UDP	192.168.1.181	54714	192.168.1.1	123
23:00:09	DNAT	green0	UDP	192.168.1.180	62892	192.168.1.1	123
23:00:09	INPUTFW	green0	UDP	192.168.1.180	62892	192.168.1.1	123
23:01:30	DNAT	green0	UDP	192.168.1.181	54714	192.168.1.1	123
23:01:30	INPUTFW	green0	UDP	192.168.1.181	54714	192.168.1.1	123
23:02:24	DNAT	green0	UDP	192.168.1.180	62892	192.168.1.1	123
23:02:24	INPUTFW	green0	UDP	192.168.1.180	62892	192.168.1.1	123

Smart switch (ESP8285) just keeps drifting to 1970 and Wifi is not the issue here

22:33:43.171 CMD: NTPSERVER
22:33:43.174 RSL: RESULT = {"NtpServer1":"192.168.1.1","NtpServer2":"132.163.97.1","NtpServer3":"129.6.15.28"}
22:34:43.422 CMD: time
22:34:43.426 RSL: RESULT = {"Time":"1970-01-01T00:30:43"}

I would appreciate any feedback or suggestions

xperimental · 12 June 2025 19:44

Why do you think your random source ports have something to do with NTP?

trish · 13 June 2025 00:24

Hey Terry,

That’s what the Firewall log shows for all traffic going to port 123

hvacguy · 13 June 2025 02:13

Reload firewall rules?
File corruption?
Using other port?Internet Time Service Firewall information | NIST

xperimental · 13 June 2025 06:52

Provide time to local network is checked?

firecherry · 13 June 2025 18:37

Sir,

nothing worked when I tried to configure ntp

This older thread helped me:

bbitsch · 13 June 2025 19:23

This very old thread closes with the announcement, that the problem is solved.
The NTP servers are listed in /etc/ntp/ntpInclude.conf

trish · 13 June 2025 22:58

Yes, Provide Time … is checked.. Again it is working for other Windows and Linux machines.

Firewall rules are reloaded
What file would be corrupt?
I can’t really change the ntp port for these devices. They don’t even have SSH..

I think I enabled debug mode now, hope something will show:

ntpd -d

stinga · 16 June 2025 00:57

Yeah, fixed but not correct really…

Need to be able specify pool vs server.
Currently we are using server and can only add 2 servers.
This is the worst case you can have.
ntp will never know which is correct.

The easy option is just to edit /etc/ntp.conf and add your pool servers…
Before

[root@wr-fw ~]# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 LOCAL(0)        .LOCL.          10 l   6d   64    0    0.000   +0.000   0.000
*159.196.3.239 ( 192.168.0.92     2 u  524 1024  377   77.326   -0.506   1.922
+pauseq4vntp1.da 14.202.65.230    2 u  412 1024  377   57.391   +1.958   0.760

after

root@wr-fw ~]# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 0.au.pool.ntp.o .POOL.          16 p    -   64    0    0.000   +0.000   0.000
 1.au.pool.ntp.o .POOL.          16 p    -   64    0    0.000   +0.000   0.000
 2.au.pool.ntp.o .POOL.          16 p    -   64    0    0.000   +0.000   0.000
 3.au.pool.ntp.o .POOL.          16 p    -   64    0    0.000   +0.000   0.000
 LOCAL(0)        .LOCL.          10 l   58   64   17    0.000   +0.000   0.000
*syd.clearnet.pw 194.195.249.28   3 u   51   64   17   57.460   -4.486   0.179
+220.158.215.20  126.11.196.147   2 u   53   64   17   57.971   -1.264   0.076
+time.cloudflare 10.84.8.25       3 u   51   64   17    7.795   -0.072   0.184
 my.blockbluemed 52.64.168.208    5 u   51   64   17   63.495   -1.422   0.737
-y.ns.gin.ntt.ne 129.250.35.222   2 u   52   64   17  282.156  -65.951   0.132
+toc.ntp.telstra 202.6.131.118    2 u   53   64   17    9.093   -0.641   0.054
 mansfield.id.au 203.36.227.3     2 u   50   64   17   55.437   -0.789   0.132
-159.196.3.239 ( 192.168.0.92     2 u   53   64   17   75.825   -4.758   0.696
+ec2-13-55-50-68 203.206.205.83   3 u   51   64   17   52.847   -3.026   0.110
 ap-southeast-2. 203.35.83.242    2 u   47   64   17   59.257   -0.033   1.406
 pauseq4vntp2.da 203.36.227.3     2 u   47   64   17   57.276   +1.385   0.949
 time.tfmcloud.a 203.35.83.242    2 u   44   64   17   54.868   -0.713   1.567
 bitburger.simon .GPS.            1 u   48   64   17   66.527   -3.391   1.424
 220.158.215.21  8.145.32.135     2 u   45   64   17   59.038   -0.088   1.430
 14-202-65-230.t .GPS.            1 u   41   64   17   62.259   +1.846   1.679
 time.cloudflare 10.84.8.25       3 u   40   64   17    9.241   +1.929   1.952

Much better

And you can turn off set time at boot.

downside, need to remember to fix after upgrade…

stinga · 16 June 2025 01:03

Replying to myself… hate that!

You can just add

pool 0.pool.ntp.org
pool 1.pool.ntp.org
pool 2.pool.ntp.org
pool 3.pool.ntp.org

to /etc/ntp.conf
IE:

[root@wr-fw ~]# cat /etc/ntp.conf
disable monitor
restrict default nomodify noquery
restrict 127.0.0.1
pool 0.pool.ntp.org
pool 1.pool.ntp.org
pool 2.pool.ntp.org
pool 3.pool.ntp.org
server  127.127.1.0
fudge   127.127.1.0 stratum 10
driftfile /etc/ntp/drift
includefile /etc/ntp/ntpInclude.conf

and let ntp sort itsself out.
With that I get

[root@wr-fw ~]# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 0.pool.ntp.org  .POOL.          16 p    -   64    0    0.000   +0.000   0.000
 1.pool.ntp.org  .POOL.          16 p    -   64    0    0.000   +0.000   0.000
 2.pool.ntp.org  .POOL.          16 p    -   64    0    0.000   +0.000   0.000
 3.pool.ntp.org  .POOL.          16 p    -   64    0    0.000   +0.000   0.000
 LOCAL(0)        .LOCL.          10 l   17   64    7    0.000   +0.000   0.000
 time.cloudflare 10.84.8.25       3 u   15   64    7    9.735   +4.921   0.251
 ntp1.ds.network 162.159.200.123  4 u    9   64    7   58.894   +4.863   0.134
 159.196.3.239 ( 192.168.0.92     2 u   18   64    7   83.777   -3.282   2.992
 ec2-13-55-50-68 203.206.205.83   3 u   14   64    7   54.326   +1.728   0.235
 vps-b7eaeed7.vp 119.18.6.37      2 u   13   64    7   58.533   +4.457   0.074
 ap-southeast-2. 203.35.83.242    2 u   14   64    7   59.381   +3.245   0.167
 smtp.juneks.com .PPS.            1 u   10   64    7   59.417   +3.649   0.143
 14-202-65-230.t .GPS.            1 u    8   64    7   62.118   +4.702   0.140
 mel.clearnet.pw 110.142.180.39   2 u    8   64    7   50.028   +4.988   0.097
 203.206.205.83  110.142.180.39   2 u    5   64    7   60.349   +5.700   0.910
 lancelot.empty. 103.160.116.13   3 u    5   64    7   59.104   +2.455   0.268
 time.cloudflare 10.84.8.25       3 u    6   64    7    9.296   +4.766   0.189
 pauseq4vntp2.da 203.36.227.3     2 u    5   64    7   57.202   +5.211   0.135
 bitburger.simon .GPS.            1 u    6   64    7   69.841   -1.144   1.674
 toc.ntp.telstra 202.6.131.118    2 u    4   64    7   10.169   +4.289   0.282
 pauseq4vntp1.da 14.202.65.230    2 u    8   64    3   57.146   +3.604   0.105

Later…

After a bit we end up with

[root@wr-fw ~]# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 0.pool.ntp.org  .POOL.          16 p    -   64    0    0.000   +0.000   0.000
 1.pool.ntp.org  .POOL.          16 p    -   64    0    0.000   +0.000   0.000
 2.pool.ntp.org  .POOL.          16 p    -   64    0    0.000   +0.000   0.000
 3.pool.ntp.org  .POOL.          16 p    -   64    0    0.000   +0.000   0.000
 LOCAL(0)        .LOCL.          10 l 1457   64    0    0.000   +0.000   0.000
*time.cloudflare 10.84.8.25       3 u    6   64  377    8.755   +0.103   0.119
+ntp1.ds.network 162.159.200.123  4 u   13   64  377   58.422   +0.012   0.100
-14-202-65-230.t .GPS.            1 u    9   64  377   62.087   -0.131   0.958
-mel.clearnet.pw 110.142.180.39   2 u   56   64  377   49.725   +0.346   0.107
+time.cloudflare 10.84.8.25       3 u   10   64  377    8.971   +0.084   0.098

The * + - at the start indicate fings about what ntpq is doing!

* This symbol indicates the currently selected system peer, the one the local machine is actively using for time synchronization.
+ These peers are considered good candidates for synchronization and are used in the clock selection and clustering algorithms to determine the system peer.
- These peers are deemed unreliable or inaccurate and are excluded from the synchronization process.

stinga · 16 June 2025 02:53

@trish what is the output of ntpq -p?
That will tell us what ntpq is doing.

Also contents of /etc/ntp.conf and /etc/ntp/ntpInclude.conf

bbitsch · 16 June 2025 08:32

You can add these pools ( only two ) in the WUI.

Further I suppose the TO has problems to publish the time to some devices.
To investigate this, it would help to log the data traffic for NTP query and response.
Possibly some devices don’t like to send queries to a NTP server outside the local network and to receive the response from IPFire.

stinga · 16 June 2025 14:34

You can’t add those pools in the WUI
You can servers in the WUI, but not pools

bonnietwin · 16 June 2025 15:10

Those server addresses are also pools.

I just did a ping on 0.ipfire.pool.ntp.org with a small gap in between and each time I got a different server.

PING 0.ipfire.pool.ntp.org (86.59.113.124) 56(84) bytes of data.
64 bytes from s1.holub.co.at (86.59.113.124): icmp_seq=1 ttl=53 time=26.2 ms
64 bytes from s1.holub.co.at (86.59.113.124): icmp_seq=2 ttl=53 time=26.6 ms
64 bytes from s1.holub.co.at (86.59.113.124): icmp_seq=3 ttl=53 time=27.0 ms
64 bytes from s1.holub.co.at (86.59.113.124): icmp_seq=4 ttl=53 time=26.7 ms

--- 0.ipfire.pool.ntp.org ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3169ms
rtt min/avg/max/mdev = 26.181/26.609/26.952/0.279 ms
[ahb@hyperion ~]$ ping -c4 0.ipfire.pool.ntp.org
PING 0.ipfire.pool.ntp.org (86.59.80.170) 56(84) bytes of data.
64 bytes from palmers.nobody.at (86.59.80.170): icmp_seq=1 ttl=53 time=27.2 ms
64 bytes from palmers.nobody.at (86.59.80.170): icmp_seq=2 ttl=53 time=27.1 ms
64 bytes from palmers.nobody.at (86.59.80.170): icmp_seq=3 ttl=53 time=27.3 ms
64 bytes from palmers.nobody.at (86.59.80.170): icmp_seq=4 ttl=53 time=27.3 ms

--- 0.ipfire.pool.ntp.org ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 26.411/27.159/27.509/0.439 ms
[ahb@hyperion ~]$ ping -c4 0.ipfire.pool.ntp.org
PING 0.ipfire.pool.ntp.org (185.144.161.170) 56(84) bytes of data.
64 bytes from mail.somenet.org (185.144.161.170): icmp_seq=1 ttl=53 time=27.3 ms
64 bytes from mail.somenet.org (185.144.161.170): icmp_seq=2 ttl=53 time=28.2 ms
64 bytes from mail.somenet.org (185.144.161.170): icmp_seq=3 ttl=53 time=27.9 ms
64 bytes from mail.somenet.org (185.144.161.170): icmp_seq=4 ttl=53 time=27.9 ms

--- 0.ipfire.pool.ntp.org ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3003ms
rtt min/avg/max/mdev = 27.316/27.836/28.208/0.324 ms

the pool syntax is used for type s addresses, server for type s and r addresses and peer for type s addresses.

So server can cover the same as pool but can also cover more.

Type s addresses are a remote server or peer (IPv4 class A, B and C)
Type r addresses are a reference clock address (127.127.x.x)

This is covered in the ntp.conf man page.

So server or pool do the same thing unless you want to use a reference clock address, when you have to use server.

server and pool are both one way. They obtain the time from that source.
peer is for two way time synchronisation so not used in the IPFire situation.

stinga · 18 June 2025 00:19

I believe pool is better, whilst pool and server are similar they are not the same.
Also only having 2 servers is sub-optimal.
I believe (very basic understanding) server is fixed to the server you specify (round robin) whilst pool allows dynamic changes to the servers being queried
For most maybe server is OK, I have a use case (FM radio transmission) where the time requires accuracy.

trish · 19 June 2025 06:35

I think I found a workaround for my NTP issues.
Setup another NTP server on a Raspberry Pi inside the Green network.

Then changed the IPFire DHCP configuration to use the Pi as a primary NTP server and also pointing all devices that have issues to the Pi. IPFire Firewall log still shows that the query was DNATed and forwarded to the IPFire’s IP address. This works with all modern devices like smart switches and AP’s that have the option to customize NTP.

I still have 2 older access points that wouldn’t update time, but I can totally live without the correct time on these AP’s.

bbitsch · 19 June 2025 12:15

Congrats for your solution.

But for the sake of our project, it would be nice what the reason was and how the NTP service in Pi differs from IPFire. This would allow to assure that the IPFire systems is gateway for all local devices: for internet access, name resultion, time service, …

Any side applications in the local network complicate the local configuration, IMO.

trish · 25 June 2025 05:50

@stinga
Here is the Output

[root@ipfire ~]# ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 LOCAL(0)        .LOCL.          10 l  19h 1024    0    0.000   +0.000   0.000
*129.6.15.28     .NIST.           1 u  261 1024   21   56.452   +1.638   2.666
+129.6.15.29     .NIST.           1 u  821 1024    1   84.363   +0.036   1.365

ntp.conf

disable monitor
restrict default nomodify noquery
restrict 127.0.0.1
server  127.127.1.0
fudge   127.127.1.0 stratum 10
driftfile /etc/ntp/drift
includefile /etc/ntp/ntpInclude.conf

ntpinclude.conf


server	129.6.15.28 prefer
server	129.6.15.29