SERVFAIL all the configured stub or forward servers failed at zone

cwensink · 29 December 2020 19:29

I’m getting a lot of DNS server failures showing up in /var/log/messages over the last two days. Here’s a sample from just the last few hours:

Dec 29 10:58:31 ipfire unbound: [32411:0] error: SERVFAIL <gateway.icloud.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 10:58:31 ipfire unbound: [32411:0] error: SERVFAIL <gateway.fe.apple-dns.net. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 10:59:52 ipfire unbound: [32411:0] error: SERVFAIL <iphone-ld.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 10:59:52 ipfire unbound: [32411:0] error: SERVFAIL <e9338.d.akamaiedge.net. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:00:05 ipfire unbound: [32411:0] error: SERVFAIL <weather-data.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:00:05 ipfire unbound: [32411:0] error: SERVFAIL <a2047.dscb.akamai.net. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:00:29 ipfire unbound: [32411:0] error: SERVFAIL <gsp85-ssl.ls.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:00:38 ipfire unbound: [32411:0] error: SERVFAIL <bag.itunes.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:00:38 ipfire unbound: [32411:0] error: SERVFAIL <e673.dsce9.akamaiedge.net. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:00:39 ipfire unbound: [32411:0] error: SERVFAIL <mesu.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:00:39 ipfire unbound: [32411:0] error: SERVFAIL <mesu.g.aaplimg.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:04:43 ipfire unbound: [32411:0] error: SERVFAIL <gspe35-ssl.ls-apple.com.akadns.net. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:04:44 ipfire unbound: [32411:0] error: SERVFAIL <e6987.e9.akamaiedge.net. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:04:46 ipfire unbound: [32411:0] error: SERVFAIL <iphone-ld.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:04:46 ipfire unbound: [32411:0] error: SERVFAIL <e9338.d.akamaiedge.net. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:04:47 ipfire unbound: [32411:0] error: SERVFAIL <reports.crashlytics.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:16 ipfire unbound: [32411:0] error: SERVFAIL <captive.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:16 ipfire unbound: [32411:0] error: SERVFAIL <iphone-ld.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:16 ipfire unbound: [32411:0] error: SERVFAIL <init.itunes.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:16 ipfire unbound: [32411:0] error: SERVFAIL <api-glb-chi.smoot.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:16 ipfire unbound: [32411:0] error: SERVFAIL <smoot-api-glb-chi.v.aaplimg.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:16 ipfire unbound: [32411:0] error: SERVFAIL <captive.g.aaplimg.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:16 ipfire unbound: [32411:0] error: SERVFAIL <e9338.d.akamaiedge.net. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:16 ipfire unbound: [32411:0] error: SERVFAIL <e673.dsce9.akamaiedge.net. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:17 ipfire unbound: [32411:0] error: SERVFAIL <gsp10-ssl.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:17 ipfire unbound: [32411:0] error: SERVFAIL <gsp10-ssl.ls.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:18 ipfire unbound: [32411:0] error: SERVFAIL <gsp10-ssl.ls-apple.com.akadns.net. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:22 ipfire unbound: [32411:0] error: SERVFAIL <configuration.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:40 ipfire unbound: [32411:0] error: SERVFAIL <gsp64-ssl.ls.apple.com. TYPE65 IN>: all the configured stub or forward servers failed, at zone .
Dec 29 11:05:40 ipfire unbound: [32411:0] error: SERVFAIL <gsp64-ssl.ls-apple.com.akadns.net. TYPE65 IN>: all the configured stub or forward servers failed, at zone .

I have tried restarting unbound, I have tried enabling a different set of DNS servers, I primarily use openDNS servers, but after having issues I turned on my AT&T DNS servers and 1.1.1.1. Any suggestions?

Chris

jon · 9 January 2021 00:16

~~Try the google DNS at 8.8.8.8.~~
Try the Quad9 DNS at 9.9.9.9

What version of IPFire are you using?

You may want to add a screen shot of the DNS Servers at menu Network > Domain Name System

cwensink · 4 January 2021 20:42

I added the entry for 8.8.8.8. IPFire 2.25 update 153. Another bizare issue that is going on, /var/log/messages has grown to be extremely large, as of Jan 4 14:39 the file is 822 MB. There was a regular process that was taking old mail and messages log files and compressing them as .gz files, but the last mail and messages files are date stamped from August 23. Something is not running right to compress old messages and mail files and zip them up. What process is running that procedure or how can I clean this up?

pavlos · 4 January 2021 20:52

there should be a syslogd process running which rotates messages and other files for 52 weeks.

logrotate -v /etc/logrotate.conf should give you information about the rotate schedule.

logrotate -f /etc/logrotate.conf will force rotation even if it thinks it is not necessary.

cwensink · 4 January 2021 21:44

[root@ipfire log]# logrotate -f /etc/logrotate.conf
[root@ipfire log]# logrotate -v /etc/logrotate.conf
Ignoring /etc/logrotate.conf because it’s empty.
Reading state from file: /var/lib/logrotate.status
Allocating hash table for state file, size 64 entries
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state

Handling 0 logs

pavlos · 4 January 2021 21:54

[root@ipfire ~]# ll /etc/logrotate.conf 
-rw-r--r-- 1 root root 1900 Oct 22  2019 /etc/logrotate.conf

it should be 1900 bytes, no idea why yours is empty. here’s mine … https://termbin.com/3oxr

cwensink · 4 January 2021 22:32

I moved the empty file to .broken, then copied yours in and when running logrotate -f /etc/logrotate.conf I get [root@ipfire log]# logrotate -f /etc/logrotate.conf
error: /etc/logrotate.conf:3 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:6 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:9 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:12 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:15 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:17 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:21, unexpected text after }
error: /etc/logrotate.conf:22 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:23 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:29, unexpected text after }
error: /etc/logrotate.conf:30 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:31 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:42, unexpected text after }
error: /etc/logrotate.conf:43 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:44 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:49, unexpected text after }
error: /etc/logrotate.conf:50 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:51 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:57, unexpected text after }
error: /etc/logrotate.conf:58 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:59 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:69, unexpected text after }
error: /etc/logrotate.conf:70 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:71 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:78, unexpected text after }
error: /etc/logrotate.conf:79 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:80 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:87, unexpected text after }
error: /etc/logrotate.conf:88 lines must begin with a keyword or a filename (possibly in double quotes)
error: /etc/logrotate.conf:89 lines must begin with a keyword or a filename (possibly in double quotes)
logrotate_script: $’\r’: command not found
logrotate_script: $’\r’: command not found
/bin/find: missing argument to -exec' /bin/find: missing argument to -exec’
logrotate_script: $’\r’: command not found
/bin/find: missing argument to -exec' /bin/find: missing argument to -exec’
error: stat of /var/log/dhcpcd.log failed: No such file or directory
[root@ipfire log]#

[root@ipfire log]# ll /etc/logrotate.conf
-rw-r–r-- 1 root root 1994 Jan 4 16:27 /etc/logrotate.conf

jon · 4 January 2021 23:25

Maybe the file copied incorrectly from termbin to Chris. Here is the logrotate.conf from GitHub:

https://raw.githubusercontent.com/ipfire/ipfire-2.x/master/config/etc/logrotate.conf
-or-

github.com

ipfire/ipfire-2.x/blob/master/config/etc/logrotate.conf

# rotate log files weekly
weekly

# keep 52 weeks worth of backlogs
rotate 52

# create new (empty) log files after rotating old ones
create

# uncomment this if you want your log files compressed
compress

# packages drop log rotation information into this directory
include /etc/logrotate.d

# wtmp
/var/log/wtmp {
    weekly
    create 0664 root utmp
    rotate 1

This file has been truncated. show original

bbitsch · 4 January 2021 23:43

You should copy in original “Linux mode” ( line separator is newline \n ) not “Windows mode” ( line separator is \r\n ).

cwensink · 5 January 2021 14:53

The first time around I copied the text to notepad on windows 10, then used winscp to copy it over, so that’s probably where the line separator issue came in. I used wget to pull in the file directly to ipfire, then put it in /etc, then from /etc/ I ran

logrotate -v logrotate.conf:

[root@ipfire etc]# logrotate -v logrotate.conf
reading config file logrotate.conf
including /etc/logrotate.d
Ignoring .empty because it's empty.
Reading state from file: /var/lib/logrotate.status
Allocating hash table for state file, size 64 entries
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state
Creating new state

Handling 9 logs

rotating pattern: /var/log/wtmp  weekly (1 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/wtmp
  Now: 2021-01-05 08:36
  Last rotated at 2021-01-05 08:34
  log does not need rotating (log has already been rotated)

rotating pattern: /var/log/httpd/access_log /var/log/httpd/error_log /var/log/ht                                                         tpd/ssl_request_log /var/log/httpd/ssl_engine_log  weekly (52 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/httpd/access_log
  Now: 2021-01-05 08:36
  Last rotated at 2021-01-05 08:34
  log does not need rotating (log has already been rotated)
considering log /var/log/httpd/error_log
  Now: 2021-01-05 08:36
  Last rotated at 2021-01-05 08:34
  log does not need rotating (log has already been rotated)
considering log /var/log/httpd/ssl_request_log
  Now: 2021-01-05 08:36
  Last rotated at 2021-01-05 08:34
  log does not need rotating (log has already been rotated)
considering log /var/log/httpd/ssl_engine_log
  log /var/log/httpd/ssl_engine_log does not exist -- skipping
not running postrotate script, since no logs were rotated

rotating pattern: /var/log/suricata/*.log  weekly (52 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/suricata/fast.log
  Now: 2021-01-05 08:36
  Last rotated at 2021-01-05 08:34
  log does not need rotating (log has already been rotated)
considering log /var/log/suricata/stats.log
  Now: 2021-01-05 08:36
  Last rotated at 2021-01-05 08:34
  log does not need rotating (log has already been rotated)

rotating pattern: /var/log/squid/access.log /var/log/squid/user_agent.log /var/l                                                         og/squid/referer.log  weekly (52 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/squid/access.log
  Now: 2021-01-05 08:36
  Last rotated at 2021-01-05 08:34
  log does not need rotating (log has already been rotated)
considering log /var/log/squid/user_agent.log
  Now: 2021-01-05 08:36
  Last rotated at 2021-01-05 08:34
  log does not need rotating (log has already been rotated)
considering log /var/log/squid/referer.log
  log /var/log/squid/referer.log does not exist -- skipping

rotating pattern: /var/log/squid/cache.log  weekly (3 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/squid/cache.log
  Now: 2021-01-05 08:36
  Last rotated at 2021-01-05 08:34
  log does not need rotating (log has already been rotated)

rotating pattern: /var/log/squid/store.log  weekly (3 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/squid/store.log
  log /var/log/squid/store.log does not exist -- skipping

rotating pattern: /var/log/messages /var/log/bootlog /var/log/dhcpcd.log /var/lo                                                         g/mail  weekly (52 rotations)
empty log files are rotated, old logs are removed
considering log /var/log/messages
  Now: 2021-01-05 08:36
  Last rotated at 2021-01-05 08:34
  log does not need rotating (log has already been rotated)
considering log /var/log/bootlog
  Now: 2021-01-05 08:36
  Last rotated at 2021-01-05 08:34
  log does not need rotating (log has already been rotated)
considering log /var/log/dhcpcd.log
error: stat of /var/log/dhcpcd.log failed: No such file or directory
considering log /var/log/mail
  Now: 2021-01-05 08:36
  Last rotated at 2021-01-05 08:34
  log does not need rotating (log has already been rotated)

rotating pattern: /var/log/squidGuard/*.log  weekly (4 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/squidGuard/squidGuard.log
  Now: 2021-01-05 08:36
  Last rotated at 2021-01-05 08:34
  log does not need rotating (log has already been rotated)

rotating pattern: /var/log/updatexlrator/*.log  weekly (4 rotations)
empty log files are not rotated, old logs are removed
considering log /var/log/updatexlrator/cache.log
  Now: 2021-01-05 08:36
  Last rotated at 2021-01-05 08:34
  log does not need rotating (log has already been rotated)
considering log /var/log/updatexlrator/checkup.log
  Now: 2021-01-05 08:36
  Last rotated at 2021-01-04 16:30
  log does not need rotating (log has been rotated at 2021-01-04 16:30, which is                                                          less than a week ago)
considering log /var/log/updatexlrator/download.log
  Now: 2021-01-05 08:36
  Last rotated at 2021-01-04 16:30
  log does not need rotating (log has been rotated at 2021-01-04 16:30, which is                                                          less than a week ago)

then:
[root@ipfire etc]# logrotate -f logrotate.conf
error: stat of /var/log/dhcpcd.log failed: No such file or directory
[root@ipfire etc]#

Yet, messages is still a large file:

-rw-rw-r--  1 root     syslogd    20 Aug  2 00:01 mail.26.gz
-rw-rw-r--  1 root     syslogd  2.6M Aug  9 00:00 messages.26.gz
-rw-rw-r--  1 root     syslogd    20 Aug  9 00:01 mail.25.gz
-rw-rw-r--  1 root     syslogd  2.5M Aug 16 00:01 messages.25.gz
-rw-rw-r--  1 root     syslogd    20 Aug 16 00:01 mail.24.gz
-rw-rw-r--  1 root     syslogd  2.4M Aug 23 00:00 messages.24.gz
-rw-rw-r--  1 root     syslogd    20 Aug 23 00:01 mail.23.gz
drwxr-xr-x  4 root     root     4.0K Aug 26 16:07 rrd
-rw-r--r--  1 root     root       44 Aug 31 15:14 setup.log
drwxr-xr-x  2 root     root     4.0K Sep  1 10:01 bootlog-archive
drwxr-xr-x 16 root     root     4.0K Oct 23 09:41 ..
-rw-r--r--  1 root     root      57K Nov 15 23:03 bootlog.old
-rw-r--r--  1 root     root      56K Dec 14 11:43 bootlog
drwxr-xr-x  2 root     root     4.0K Dec 18 04:41 openvpn
-rw-rw-r--  1 root     syslogd  1.3K Dec 21 08:58 mail
drwxr-xr-x  2 root     root     4.0K Dec 22 09:56 dhcpcd
drwxr-xr-x  2 root     root     4.0K Dec 22 14:28 pakfire
drwxr-xr-x 18 squid    squid    4.0K Jan  4 06:51 cache
-rw-rw-r--  1 root     utmp     295K Jan  4 15:54 lastlog
-rw-------  1 root     root      81K Jan  4 16:28 btmp
drwxr-xr-x  2 root     root     4.0K Jan  5 00:05 logwatch
-rw-rw-r--  1 root     utmp       20 Jan  5 08:34 wtmp.1.gz
drwxr-xr-x  2 squid    squid    4.0K Jan  5 08:34 squidGuard
-rw-rw-r--  1 root     utmp        0 Jan  5 08:37 wtmp
drwxr-xr-x  3 root     root      12K Jan  5 08:37 httpd
drwxr-xr-x  5 suricata suricata 4.0K Jan  5 08:37 suricata
drwxr-xr-x  2 squid    squid    4.0K Jan  5 08:37 squid
drwxr-xr-x 20 root     root      12K Jan  5 08:37 .
drwxr-xr-x  2 squid    squid    4.0K Jan  5 08:37 updatexlrator
drwxr-xr-x  2 root     root     4.0K Jan  5 08:38 vnstat
-rw-rw-r--  1 root     syslogd  825M Jan  5 08:38 messages

pavlos · 5 January 2021 15:42

what’s the output of grep syslogd /var/log/messages ? I have:

[root@ipfire ~]# grep syslogd /var/log/messages
Jan  3 00:01:01 ipfire syslogd 1.5.1: restart (remote reception).

cwensink · 5 January 2021 15:43

[root@ipfire ~]# grep syslogd /var/log/messages
Aug 30 00:01:03 ipfire syslogd 1.5.1: restart (remote reception).
Aug 31 13:16:38 ipfire syslogd 1.5.1: restart (remote reception).
Sep 8 18:42:37 ipfire syslogd 1.5.1: restart (remote reception).
Sep 21 19:53:05 ipfire syslogd 1.5.1: restart (remote reception).
Sep 21 20:01:19 ipfire syslogd 1.5.1: restart (remote reception).
Nov 15 23:03:13 ipfire syslogd 1.5.1: restart (remote reception).
Dec 14 11:43:32 ipfire syslogd 1.5.1: restart (remote reception).
Dec 30 10:34:02 ipfire syslogd 1.5.1: restart (remote reception).
Dec 30 10:34:05 ipfire syslogd 1.5.1: restart (remote reception).
Jan 4 14:42:52 ipfire syslogd 1.5.1: restart (remote reception).
Jan 4 14:42:55 ipfire syslogd 1.5.1: restart (remote reception).
Jan 4 14:42:58 ipfire syslogd 1.5.1: restart (remote reception).
Jan 4 14:43:08 ipfire syslogd 1.5.1: restart (remote reception).
Jan 4 14:43:10 ipfire syslogd 1.5.1: restart (remote reception).
[root@ipfire ~]#

pavlos · 5 January 2021 16:27

I tested on a VM (core 153). The BEFORE (8 .gz files) shows messages as 111844 bytes, then logrotate, the AFTER (9 .gz files) shows messages as 280 bytes. It created messages.1.gz (7520 byes) and pushed all the other compressed files one down (the .2 became .3, etc).

Can you reboot your ipfire, wait an hour for messages to be populated and then logrotate -f?

agibson · 5 January 2021 16:37

I’m still getting a lot of DNS server failures showing up in /var/log/messages. This part of the question was not answered yet. Looking for answers. I liked the information on the log rotate and how it worked, how to debug that problem. Good explanations.

cwensink · 5 January 2021 16:55

I’ll reboot after hours and see what happens.

pavlos · 5 January 2021 17:01

which DNS do you use? I get those SERVFAIL as well but not that many.

agibson · 9 January 2021 00:09

@pavlos
I use
8.8.8.8
8.8.4.4
75.75.75.75
1.1.1.1
I am going to trying bumping that up 3 more DNS servers and see what that does.