I’m getting a lot of DNS server failures showing up in /var/log/messages over the last two days. Here’s a sample from just the last few hours:
Dec 29 10:58:31 ipfire unbound: [32411:0] error: SERVFAIL <gateway.icloud.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 10:58:31 ipfire unbound: [32411:0] error: SERVFAIL <gateway.fe.apple-dns.net. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 10:59:52 ipfire unbound: [32411:0] error: SERVFAIL <iphone-ld.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 10:59:52 ipfire unbound: [32411:0] error: SERVFAIL <e9338.d.akamaiedge.net. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:00:05 ipfire unbound: [32411:0] error: SERVFAIL <weather-data.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:00:05 ipfire unbound: [32411:0] error: SERVFAIL <a2047.dscb.akamai.net. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:00:29 ipfire unbound: [32411:0] error: SERVFAIL <gsp85-ssl.ls.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:00:38 ipfire unbound: [32411:0] error: SERVFAIL <bag.itunes.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:00:38 ipfire unbound: [32411:0] error: SERVFAIL <e673.dsce9.akamaiedge.net. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:00:39 ipfire unbound: [32411:0] error: SERVFAIL <mesu.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:00:39 ipfire unbound: [32411:0] error: SERVFAIL <mesu.g.aaplimg.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:04:43 ipfire unbound: [32411:0] error: SERVFAIL <gspe35-ssl.ls-apple.com.akadns.net. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:04:44 ipfire unbound: [32411:0] error: SERVFAIL <e6987.e9.akamaiedge.net. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:04:46 ipfire unbound: [32411:0] error: SERVFAIL <iphone-ld.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:04:46 ipfire unbound: [32411:0] error: SERVFAIL <e9338.d.akamaiedge.net. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:04:47 ipfire unbound: [32411:0] error: SERVFAIL <reports.crashlytics.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:16 ipfire unbound: [32411:0] error: SERVFAIL <captive.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:16 ipfire unbound: [32411:0] error: SERVFAIL <iphone-ld.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:16 ipfire unbound: [32411:0] error: SERVFAIL <init.itunes.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:16 ipfire unbound: [32411:0] error: SERVFAIL <api-glb-chi.smoot.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:16 ipfire unbound: [32411:0] error: SERVFAIL <smoot-api-glb-chi.v.aaplimg.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:16 ipfire unbound: [32411:0] error: SERVFAIL <captive.g.aaplimg.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:16 ipfire unbound: [32411:0] error: SERVFAIL <e9338.d.akamaiedge.net. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:16 ipfire unbound: [32411:0] error: SERVFAIL <e673.dsce9.akamaiedge.net. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:17 ipfire unbound: [32411:0] error: SERVFAIL <gsp10-ssl.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:17 ipfire unbound: [32411:0] error: SERVFAIL <gsp10-ssl.ls.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:18 ipfire unbound: [32411:0] error: SERVFAIL <gsp10-ssl.ls-apple.com.akadns.net. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:22 ipfire unbound: [32411:0] error: SERVFAIL <configuration.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:40 ipfire unbound: [32411:0] error: SERVFAIL <gsp64-ssl.ls.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:40 ipfire unbound: [32411:0] error: SERVFAIL <gsp64-ssl.ls-apple.com.akadns.net. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
I have tried restarting unbound, I have tried enabling a different set of DNS servers, I primarily use openDNS servers, but after having issues I turned on my AT&T DNS servers and 1.1.1.1. Any suggestions?
I added the entry for 8.8.8.8. IPFire 2.25 update 153. Another bizare issue that is going on, /var/log/messages has grown to be extremely large, as of Jan 4 14:39 the file is 822 MB. There was a regular process that was taking old mail and messages log files and compressing them as .gz files, but the last mail and messages files are date stamped from August 23. Something is not running right to compress old messages and mail files and zip them up. What process is running that procedure or how can I clean this up?
[root@ipfire log]# logrotate -f /etc/logrotate.conf
[root@ipfire log]# logrotate -v /etc/logrotate.conf
Ignoring /etc/logrotate.conf because it’s empty.
Reading state from file: /var/lib/logrotate.status
Allocating hash table for state file, size 64 entries
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
I moved the empty file to .broken, then copied yours in and when running logrotate -f /etc/logrotate.conf I get [root@ipfire log]# logrotate -f /etc/logrotate.conf
error: /etc/logrotate.conf:3 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:6 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:9 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:12 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:15 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:17 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:21, unexpected text after }
error: /etc/logrotate.conf:22 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:23 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:29, unexpected text after }
error: /etc/logrotate.conf:30 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:31 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:42, unexpected text after }
error: /etc/logrotate.conf:43 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:44 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:49, unexpected text after }
error: /etc/logrotate.conf:50 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:51 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:57, unexpected text after }
error: /etc/logrotate.conf:58 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:59 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:69, unexpected text after }
error: /etc/logrotate.conf:70 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:71 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:78, unexpected text after }
error: /etc/logrotate.conf:79 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:80 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:87, unexpected text after }
error: /etc/logrotate.conf:88 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:89 lines must begin with a keyword or a filename (possibly in double quotes)
logrotate_script: $’\r’: command not found
logrotate_script: $’\r’: command not found
/bin/find: missing argument to -exec' /bin/find: missing argument to -exec’
logrotate_script: $’\r’: command not found
/bin/find: missing argument to -exec' /bin/find: missing argument to -exec’
error: stat of /var/log/dhcpcd.log failed: No such file or directory
[root@ipfire log]#
The first time around I copied the text to notepad on windows 10, then used winscp to copy it over, so that’s probably where the line separator issue came in. I used wget to pull in the file directly to ipfire, then put it in /etc, then from /etc/ I ran
logrotate -v logrotate.conf:
[root@ipfire etc]# logrotate -v logrotate.conf
reading config file logrotate.conf
including /etc/logrotate.d
Ignoring .empty because it's empty.
Reading state from file: /var/lib/logrotate.status
Allocating hash table for state file, size 64 entries
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Handling 9 logs
rotating pattern: /var/log/wtmp weekly (1 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/wtmp
Now: 2021-01-05 08:36
Last rotated at 2021-01-05 08:34
log does not need rotating (log has already been rotated)
rotating pattern: /var/log/httpd/access_log /var/log/httpd/error_log /var/log/ht tpd/ssl_request_log /var/log/httpd/ssl_engine_log weekly (52 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/httpd/access_log
Now: 2021-01-05 08:36
Last rotated at 2021-01-05 08:34
log does not need rotating (log has already been rotated)
considering log /var/log/httpd/error_log
Now: 2021-01-05 08:36
Last rotated at 2021-01-05 08:34
log does not need rotating (log has already been rotated)
considering log /var/log/httpd/ssl_request_log
Now: 2021-01-05 08:36
Last rotated at 2021-01-05 08:34
log does not need rotating (log has already been rotated)
considering log /var/log/httpd/ssl_engine_log
log /var/log/httpd/ssl_engine_log does not exist -- skipping
not running postrotate script, since no logs were rotated
rotating pattern: /var/log/suricata/*.log weekly (52 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/suricata/fast.log
Now: 2021-01-05 08:36
Last rotated at 2021-01-05 08:34
log does not need rotating (log has already been rotated)
considering log /var/log/suricata/stats.log
Now: 2021-01-05 08:36
Last rotated at 2021-01-05 08:34
log does not need rotating (log has already been rotated)
rotating pattern: /var/log/squid/access.log /var/log/squid/user_agent.log /var/l og/squid/referer.log weekly (52 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/squid/access.log
Now: 2021-01-05 08:36
Last rotated at 2021-01-05 08:34
log does not need rotating (log has already been rotated)
considering log /var/log/squid/user_agent.log
Now: 2021-01-05 08:36
Last rotated at 2021-01-05 08:34
log does not need rotating (log has already been rotated)
considering log /var/log/squid/referer.log
log /var/log/squid/referer.log does not exist -- skipping
rotating pattern: /var/log/squid/cache.log weekly (3 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/squid/cache.log
Now: 2021-01-05 08:36
Last rotated at 2021-01-05 08:34
log does not need rotating (log has already been rotated)
rotating pattern: /var/log/squid/store.log weekly (3 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/squid/store.log
log /var/log/squid/store.log does not exist -- skipping
rotating pattern: /var/log/messages /var/log/bootlog /var/log/dhcpcd.log /var/lo g/mail weekly (52 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/messages
Now: 2021-01-05 08:36
Last rotated at 2021-01-05 08:34
log does not need rotating (log has already been rotated)
considering log /var/log/bootlog
Now: 2021-01-05 08:36
Last rotated at 2021-01-05 08:34
log does not need rotating (log has already been rotated)
considering log /var/log/dhcpcd.log
error: stat of /var/log/dhcpcd.log failed: No such file or directory
considering log /var/log/mail
Now: 2021-01-05 08:36
Last rotated at 2021-01-05 08:34
log does not need rotating (log has already been rotated)
rotating pattern: /var/log/squidGuard/*.log weekly (4 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/squidGuard/squidGuard.log
Now: 2021-01-05 08:36
Last rotated at 2021-01-05 08:34
log does not need rotating (log has already been rotated)
rotating pattern: /var/log/updatexlrator/*.log weekly (4 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/updatexlrator/cache.log
Now: 2021-01-05 08:36
Last rotated at 2021-01-05 08:34
log does not need rotating (log has already been rotated)
considering log /var/log/updatexlrator/checkup.log
Now: 2021-01-05 08:36
Last rotated at 2021-01-04 16:30
log does not need rotating (log has been rotated at 2021-01-04 16:30, which is less than a week ago)
considering log /var/log/updatexlrator/download.log
Now: 2021-01-05 08:36
Last rotated at 2021-01-04 16:30
log does not need rotating (log has been rotated at 2021-01-04 16:30, which is less than a week ago)
then:
[root@ipfire etc]# logrotate -f logrotate.conf
error: stat of /var/log/dhcpcd.log failed: No such file or directory
[root@ipfire etc]#
Yet, messages is still a large file:
-rw-rw-r-- 1 root syslogd 20 Aug 2 00:01 mail.26.gz
-rw-rw-r-- 1 root syslogd 2.6M Aug 9 00:00 messages.26.gz
-rw-rw-r-- 1 root syslogd 20 Aug 9 00:01 mail.25.gz
-rw-rw-r-- 1 root syslogd 2.5M Aug 16 00:01 messages.25.gz
-rw-rw-r-- 1 root syslogd 20 Aug 16 00:01 mail.24.gz
-rw-rw-r-- 1 root syslogd 2.4M Aug 23 00:00 messages.24.gz
-rw-rw-r-- 1 root syslogd 20 Aug 23 00:01 mail.23.gz
drwxr-xr-x 4 root root 4.0K Aug 26 16:07 rrd
-rw-r--r-- 1 root root 44 Aug 31 15:14 setup.log
drwxr-xr-x 2 root root 4.0K Sep 1 10:01 bootlog-archive
drwxr-xr-x 16 root root 4.0K Oct 23 09:41 ..
-rw-r--r-- 1 root root 57K Nov 15 23:03 bootlog.old
-rw-r--r-- 1 root root 56K Dec 14 11:43 bootlog
drwxr-xr-x 2 root root 4.0K Dec 18 04:41 openvpn
-rw-rw-r-- 1 root syslogd 1.3K Dec 21 08:58 mail
drwxr-xr-x 2 root root 4.0K Dec 22 09:56 dhcpcd
drwxr-xr-x 2 root root 4.0K Dec 22 14:28 pakfire
drwxr-xr-x 18 squid squid 4.0K Jan 4 06:51 cache
-rw-rw-r-- 1 root utmp 295K Jan 4 15:54 lastlog
-rw------- 1 root root 81K Jan 4 16:28 btmp
drwxr-xr-x 2 root root 4.0K Jan 5 00:05 logwatch
-rw-rw-r-- 1 root utmp 20 Jan 5 08:34 wtmp.1.gz
drwxr-xr-x 2 squid squid 4.0K Jan 5 08:34 squidGuard
-rw-rw-r-- 1 root utmp 0 Jan 5 08:37 wtmp
drwxr-xr-x 3 root root 12K Jan 5 08:37 httpd
drwxr-xr-x 5 suricata suricata 4.0K Jan 5 08:37 suricata
drwxr-xr-x 2 squid squid 4.0K Jan 5 08:37 squid
drwxr-xr-x 20 root root 12K Jan 5 08:37 .
drwxr-xr-x 2 squid squid 4.0K Jan 5 08:37 updatexlrator
drwxr-xr-x 2 root root 4.0K Jan 5 08:38 vnstat
-rw-rw-r-- 1 root syslogd 825M Jan 5 08:38 messages
I tested on a VM (core 153). The BEFORE (8 .gz files) shows messages as 111844 bytes, then logrotate, the AFTER (9 .gz files) shows messages as 280 bytes. It created messages.1.gz (7520 byes) and pushed all the other compressed files one down (the .2 became .3, etc).
Can you reboot your ipfire, wait an hour for messages to be populated and then logrotate -f?
I’m still getting a lot of DNS server failures showing up in /var/log/messages. This part of the question was not answered yet. Looking for answers. I liked the information on the log rotate and how it worked, how to debug that problem. Good explanations.
