I’m getting a lot of DNS server failures showing up in /var/log/messages over the last two days. Here’s a sample from just the last few hours:
Dec 29 10:58:31 ipfire unbound: [32411:0] error: SERVFAIL <gateway.icloud.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 10:58:31 ipfire unbound: [32411:0] error: SERVFAIL <gateway.fe.apple-dns.net. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 10:59:52 ipfire unbound: [32411:0] error: SERVFAIL <iphone-ld.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 10:59:52 ipfire unbound: [32411:0] error: SERVFAIL <e9338.d.akamaiedge.net. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:00:05 ipfire unbound: [32411:0] error: SERVFAIL <weather-data.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:00:05 ipfire unbound: [32411:0] error: SERVFAIL <a2047.dscb.akamai.net. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:00:29 ipfire unbound: [32411:0] error: SERVFAIL <gsp85-ssl.ls.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:00:38 ipfire unbound: [32411:0] error: SERVFAIL <bag.itunes.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:00:38 ipfire unbound: [32411:0] error: SERVFAIL <e673.dsce9.akamaiedge.net. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:00:39 ipfire unbound: [32411:0] error: SERVFAIL <mesu.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:00:39 ipfire unbound: [32411:0] error: SERVFAIL <mesu.g.aaplimg.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:04:43 ipfire unbound: [32411:0] error: SERVFAIL <gspe35-ssl.ls-apple.com.akadns.net. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:04:44 ipfire unbound: [32411:0] error: SERVFAIL <e6987.e9.akamaiedge.net. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:04:46 ipfire unbound: [32411:0] error: SERVFAIL <iphone-ld.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:04:46 ipfire unbound: [32411:0] error: SERVFAIL <e9338.d.akamaiedge.net. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:04:47 ipfire unbound: [32411:0] error: SERVFAIL <reports.crashlytics.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:16 ipfire unbound: [32411:0] error: SERVFAIL <captive.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:16 ipfire unbound: [32411:0] error: SERVFAIL <iphone-ld.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:16 ipfire unbound: [32411:0] error: SERVFAIL <init.itunes.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:16 ipfire unbound: [32411:0] error: SERVFAIL <api-glb-chi.smoot.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:16 ipfire unbound: [32411:0] error: SERVFAIL <smoot-api-glb-chi.v.aaplimg.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:16 ipfire unbound: [32411:0] error: SERVFAIL <captive.g.aaplimg.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:16 ipfire unbound: [32411:0] error: SERVFAIL <e9338.d.akamaiedge.net. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:16 ipfire unbound: [32411:0] error: SERVFAIL <e673.dsce9.akamaiedge.net. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:17 ipfire unbound: [32411:0] error: SERVFAIL <gsp10-ssl.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:17 ipfire unbound: [32411:0] error: SERVFAIL <gsp10-ssl.ls.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:18 ipfire unbound: [32411:0] error: SERVFAIL <gsp10-ssl.ls-apple.com.akadns.net. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:22 ipfire unbound: [32411:0] error: SERVFAIL <configuration.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:40 ipfire unbound: [32411:0] error: SERVFAIL <gsp64-ssl.ls.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:40 ipfire unbound: [32411:0] error: SERVFAIL <gsp64-ssl.ls-apple.com.akadns.net. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
I have tried restarting unbound, I have tried enabling a different set of DNS servers, I primarily use openDNS servers, but after having issues I turned on my AT&T DNS servers and 1.1.1.1. Any suggestions?
I added the entry for 8.8.8.8. IPFire 2.25 update 153. Another bizare issue that is going on, /var/log/messages has grown to be extremely large, as of Jan 4 14:39 the file is 822 MB. There was a regular process that was taking old mail and messages log files and compressing them as .gz files, but the last mail and messages files are date stamped from August 23. Something is not running right to compress old messages and mail files and zip them up. What process is running that procedure or how can I clean this up?
[root@ipfire log]# logrotate -f /etc/logrotate.conf
[root@ipfire log]# logrotate -v /etc/logrotate.conf
Ignoring /etc/logrotate.conf because it’s empty.
Reading state from file: /var/lib/logrotate.status
Allocating hash table for state file, size 64 entries
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
I moved the empty file to .broken, then copied yours in and when running logrotate -f /etc/logrotate.conf I get [root@ipfire log]# logrotate -f /etc/logrotate.conf
error: /etc/logrotate.conf:3 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:6 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:9 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:12 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:15 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:17 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:21, unexpected text after }
error: /etc/logrotate.conf:22 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:23 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:29, unexpected text after }
error: /etc/logrotate.conf:30 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:31 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:42, unexpected text after }
error: /etc/logrotate.conf:43 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:44 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:49, unexpected text after }
error: /etc/logrotate.conf:50 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:51 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:57, unexpected text after }
error: /etc/logrotate.conf:58 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:59 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:69, unexpected text after }
error: /etc/logrotate.conf:70 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:71 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:78, unexpected text after }
error: /etc/logrotate.conf:79 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:80 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:87, unexpected text after }
error: /etc/logrotate.conf:88 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:89 lines must begin with a keyword or a filename (possibly in double quotes)
logrotate_script: $’\r’: command not found
logrotate_script: $’\r’: command not found
/bin/find: missing argument to -exec' /bin/find: missing argument to -exec’
logrotate_script: $’\r’: command not found
/bin/find: missing argument to -exec' /bin/find: missing argument to -exec’
error: stat of /var/log/dhcpcd.log failed: No such file or directory
[root@ipfire log]#
The first time around I copied the text to notepad on windows 10, then used winscp to copy it over, so that’s probably where the line separator issue came in. I used wget to pull in the file directly to ipfire, then put it in /etc, then from /etc/ I ran
logrotate -v logrotate.conf:
[root@ipfire etc]# logrotate -v logrotate.conf
reading config file logrotate.conf
including /etc/logrotate.d
Ignoring .empty because it's empty.
Reading state from file: /var/lib/logrotate.status
Allocating hash table for state file, size 64 entries
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Handling 9 logs
rotating pattern: /var/log/wtmp weekly (1 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/wtmp
Now: 2021-01-05 08:36
Last rotated at 2021-01-05 08:34
log does not need rotating (log has already been rotated)
rotating pattern: /var/log/httpd/access_log /var/log/httpd/error_log /var/log/ht tpd/ssl_request_log /var/log/httpd/ssl_engine_log weekly (52 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/httpd/access_log
Now: 2021-01-05 08:36
Last rotated at 2021-01-05 08:34
log does not need rotating (log has already been rotated)
considering log /var/log/httpd/error_log
Now: 2021-01-05 08:36
Last rotated at 2021-01-05 08:34
log does not need rotating (log has already been rotated)
considering log /var/log/httpd/ssl_request_log
Now: 2021-01-05 08:36
Last rotated at 2021-01-05 08:34
log does not need rotating (log has already been rotated)
considering log /var/log/httpd/ssl_engine_log
log /var/log/httpd/ssl_engine_log does not exist -- skipping
not running postrotate script, since no logs were rotated
rotating pattern: /var/log/suricata/*.log weekly (52 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/suricata/fast.log
Now: 2021-01-05 08:36
Last rotated at 2021-01-05 08:34
log does not need rotating (log has already been rotated)
considering log /var/log/suricata/stats.log
Now: 2021-01-05 08:36
Last rotated at 2021-01-05 08:34
log does not need rotating (log has already been rotated)
rotating pattern: /var/log/squid/access.log /var/log/squid/user_agent.log /var/l og/squid/referer.log weekly (52 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/squid/access.log
Now: 2021-01-05 08:36
Last rotated at 2021-01-05 08:34
log does not need rotating (log has already been rotated)
considering log /var/log/squid/user_agent.log
Now: 2021-01-05 08:36
Last rotated at 2021-01-05 08:34
log does not need rotating (log has already been rotated)
considering log /var/log/squid/referer.log
log /var/log/squid/referer.log does not exist -- skipping
rotating pattern: /var/log/squid/cache.log weekly (3 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/squid/cache.log
Now: 2021-01-05 08:36
Last rotated at 2021-01-05 08:34
log does not need rotating (log has already been rotated)
rotating pattern: /var/log/squid/store.log weekly (3 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/squid/store.log
log /var/log/squid/store.log does not exist -- skipping
rotating pattern: /var/log/messages /var/log/bootlog /var/log/dhcpcd.log /var/lo g/mail weekly (52 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/messages
Now: 2021-01-05 08:36
Last rotated at 2021-01-05 08:34
log does not need rotating (log has already been rotated)
considering log /var/log/bootlog
Now: 2021-01-05 08:36
Last rotated at 2021-01-05 08:34
log does not need rotating (log has already been rotated)
considering log /var/log/dhcpcd.log
error: stat of /var/log/dhcpcd.log failed: No such file or directory
considering log /var/log/mail
Now: 2021-01-05 08:36
Last rotated at 2021-01-05 08:34
log does not need rotating (log has already been rotated)
rotating pattern: /var/log/squidGuard/*.log weekly (4 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/squidGuard/squidGuard.log
Now: 2021-01-05 08:36
Last rotated at 2021-01-05 08:34
log does not need rotating (log has already been rotated)
rotating pattern: /var/log/updatexlrator/*.log weekly (4 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/updatexlrator/cache.log
Now: 2021-01-05 08:36
Last rotated at 2021-01-05 08:34
log does not need rotating (log has already been rotated)
considering log /var/log/updatexlrator/checkup.log
Now: 2021-01-05 08:36
Last rotated at 2021-01-04 16:30
log does not need rotating (log has been rotated at 2021-01-04 16:30, which is less than a week ago)
considering log /var/log/updatexlrator/download.log
Now: 2021-01-05 08:36
Last rotated at 2021-01-04 16:30
log does not need rotating (log has been rotated at 2021-01-04 16:30, which is less than a week ago)
then:
[root@ipfire etc]# logrotate -f logrotate.conf
error: stat of /var/log/dhcpcd.log failed: No such file or directory
[root@ipfire etc]#
Yet, messages is still a large file:
-rw-rw-r-- 1 root syslogd 20 Aug 2 00:01 mail.26.gz
-rw-rw-r-- 1 root syslogd 2.6M Aug 9 00:00 messages.26.gz
-rw-rw-r-- 1 root syslogd 20 Aug 9 00:01 mail.25.gz
-rw-rw-r-- 1 root syslogd 2.5M Aug 16 00:01 messages.25.gz
-rw-rw-r-- 1 root syslogd 20 Aug 16 00:01 mail.24.gz
-rw-rw-r-- 1 root syslogd 2.4M Aug 23 00:00 messages.24.gz
-rw-rw-r-- 1 root syslogd 20 Aug 23 00:01 mail.23.gz
drwxr-xr-x 4 root root 4.0K Aug 26 16:07 rrd
-rw-r--r-- 1 root root 44 Aug 31 15:14 setup.log
drwxr-xr-x 2 root root 4.0K Sep 1 10:01 bootlog-archive
drwxr-xr-x 16 root root 4.0K Oct 23 09:41 ..
-rw-r--r-- 1 root root 57K Nov 15 23:03 bootlog.old
-rw-r--r-- 1 root root 56K Dec 14 11:43 bootlog
drwxr-xr-x 2 root root 4.0K Dec 18 04:41 openvpn
-rw-rw-r-- 1 root syslogd 1.3K Dec 21 08:58 mail
drwxr-xr-x 2 root root 4.0K Dec 22 09:56 dhcpcd
drwxr-xr-x 2 root root 4.0K Dec 22 14:28 pakfire
drwxr-xr-x 18 squid squid 4.0K Jan 4 06:51 cache
-rw-rw-r-- 1 root utmp 295K Jan 4 15:54 lastlog
-rw------- 1 root root 81K Jan 4 16:28 btmp
drwxr-xr-x 2 root root 4.0K Jan 5 00:05 logwatch
-rw-rw-r-- 1 root utmp 20 Jan 5 08:34 wtmp.1.gz
drwxr-xr-x 2 squid squid 4.0K Jan 5 08:34 squidGuard
-rw-rw-r-- 1 root utmp 0 Jan 5 08:37 wtmp
drwxr-xr-x 3 root root 12K Jan 5 08:37 httpd
drwxr-xr-x 5 suricata suricata 4.0K Jan 5 08:37 suricata
drwxr-xr-x 2 squid squid 4.0K Jan 5 08:37 squid
drwxr-xr-x 20 root root 12K Jan 5 08:37 .
drwxr-xr-x 2 squid squid 4.0K Jan 5 08:37 updatexlrator
drwxr-xr-x 2 root root 4.0K Jan 5 08:38 vnstat
-rw-rw-r-- 1 root syslogd 825M Jan 5 08:38 messages
I tested on a VM (core 153). The BEFORE (8 .gz files) shows messages as 111844 bytes, then logrotate, the AFTER (9 .gz files) shows messages as 280 bytes. It created messages.1.gz (7520 byes) and pushed all the other compressed files one down (the .2 became .3, etc).
Can you reboot your ipfire, wait an hour for messages to be populated and then logrotate -f?
I’m still getting a lot of DNS server failures showing up in /var/log/messages. This part of the question was not answered yet. Looking for answers. I liked the information on the log rotate and how it worked, how to debug that problem. Good explanations.
On my core 190 system I observe similar messages from unbound, reason unknown:
|11:41:54|unbound: [25442:0]|info: validation failure <_dns.resolver.arpa. SVCB IN>: no NSEC3 records from 1.1.1.1 for DS resolver.arpa. while building chain of trust|
|---|---|---|
|11:14:48|unbound: [25442:0]|error: SERVFAIL : all the configured stub or forward servers failed, at zone . from 1.1.1.1 got SERVFAIL|
|11:14:48|unbound: [25442:0]|error: SERVFAIL : all the configured stub or forward servers failed, at zone . from 1.1.1.1 got SERVFAIL|
|11:11:13|unbound: [25442:0]|info: validation failure <_dns.resolver.arpa. SVCB IN>: no NSEC3 records from 88.99.98.111 for DS resolver.arpa. while building chain of trust|
|11:10:54|unbound: [25442:0]|error: SERVFAIL : all the configured stub or forward servers failed, at zone . from 1.1.1.1 got SERVFAIL|
|11:10:04|unbound: [25442:0]|error: SERVFAIL : all the configured stub or forward servers failed, at zone . from 8.8.8.8 got SERVFAIL|
|11:09:18|unbound: [25442:0]|error: SERVFAIL : all the configured stub or forward servers failed, at zone . from 8.8.8.8 got SERVFAIL|
|11:09:11|unbound: [25442:0]|error: SERVFAIL : all the configured stub or forward servers failed, at zone . from 8.8.8.8 got SERVFAIL|
|11:05:48|unbound: [25442:0]|error: SERVFAIL : all the configured stub or forward servers failed, at zone . from 8.8.8.8 got SERVFAIL|
|11:05:42|unbound: [25442:0]|error: SERVFAIL : all the configured stub or forward servers failed, at zone . from 8.8.8.8 got SERVFAIL|
|11:05:14|unbound: [25442:0]|error: SERVFAIL : all the configured stub or forward servers failed, at zone . from 8.8.8.8 upstream server timeout|
|11:05:08|unbound: [25442:0]|error: SERVFAIL : all the configured stub or forward servers failed, at zone . from 1.1.1.1 got SERVFAIL|
|10:42:27|unbound: [25442:0]|info: validation failure <_dns.resolver.arpa. SVCB IN>: no NSEC3 records from 1.1.1.1 for DS resolver.arpa. while building chain of trust|
|10:29:36|unbound: [25442:0]|info: validation failure <_dns.resolver.arpa. SVCB IN>: no NSEC3 records from 1.1.1.1 for DS resolver.arpa. while building chain of trust|
|10:15:08|unbound: [25442:0]|info: validation failure <_dns.resolver.arpa. SVCB IN>: no NSEC3 records from 88.99.98.111 for DS resolver.arpa. while building chain of trust|
|09:41:30|unbound: [25442:0]|info: validation failure <_dns.resolver.arpa. SVCB IN>: no NSEC3 records from 88.99.98.111 for DS resolver.arpa. while building chain of trust|
|09:11:19|unbound: [25442:0]|info: validation failure <_dns.resolver.arpa. SVCB IN>: no NSEC3 records from 1.1.1.1 for DS resolver.arpa. while building chain of trust|
|08:42:19|unbound: [25442:0]|info: validation failure <_dns.resolver.arpa. SVCB IN>: no NSEC3 records from 88.99.98.111 for DS resolver.arpa. while building chain of trust|
|08:26:55|unbound: [25442:0]|error: SERVFAIL : all the configured stub or forward servers failed, at zone . from 88.99.98.111 upstream server timeout|
|08:11:59|unbound: [25442:0]|info: validation failure <_dns.resolver.arpa. SVCB IN>: no NSEC3 records from 1.1.1.1 for DS resolver.arpa. while building chain of trust|
|07:41:16|unbound: [25442:0]|info: validation failure <_dns.resolver.arpa. SVCB IN>: no NSEC3 records from 88.99.98.111 for DS resolver.arpa. while building chain of trust|
|07:11:12|unbound: [25442:0]|info: validation failure <_dns.resolver.arpa. SVCB IN>: no NSEC3 records from 88.99.98.111 for DS resolver.arpa. while building chain of trust|
|03:58:05|unbound: [25442:0]|info: generate keytag query _ta-4a5c-4f66. NULL IN|
These are just info messages. These mean that some local machine has not got a local domain name and has been assigned resolver.arpa but was then sent out for a DNS request.
If you want to stop those resolver.arpa and service.arpa FQDN’s being sent externally for DNS resolving now and not wait for the unbound update then there is a workaround in the above thread link.
These messages mean that the DNS server that was being used got no response from the root dns servers. This can happen periodically and after a short while it should work again.
Sometimes you have the message showing that the DNS server you use has received a SERVFAIL from the root dns server and sometimes you have the message “upstream server timeout” which means it got no response at all from the root zone.
If you get 1000’s of the SERVFAIL messages in a day in your log then the DNS server you are using has some sort of problem and you should look at the status of the various ones you have selected and maybe disable the one giving a problem.
If you get 10 or 50 or so of these messages per day then that is normal for networks that occasionally can have a problem. As long as you can still do your DNS resolving I would not worry about it.
I have a script that collects the data on how many of these I am getting and typically I get between 10 and 200 per week.
Occasionally I have 2000 in a week but I know that was when I was working on my red connection, to try and use a red vlan connection so I could connect to my fibre line without needing to use the modem from the ISP. That took some time to fix and so there were long periods without any internet connection so all DNS requests would have failed and show up as SERVFAIL messages.
Your log seems to be covering an 8 hour period so I would not consider that bad at all.