SERVFAIL storm last night

Networking DNS
1

I am looking for help with SERVFAIL errors.

Last night I experienced a SERVFAIL storm. Between 1:30 AM and 7:18 AM the message log filled with 32000+ SERVFAIL messages.

Here is a small sample of the messages log. At 7:17 AM the IPFire box stopped working.

May 27 01:30:06 ipfire unbound: [1514:0] error: SERVFAIL <api.darksky.net. A IN>: all the configured stub or forward servers failed, at zone .
May 27 01:30:06 ipfire unbound: [1514:0] error: SERVFAIL <api.weather.gov. A IN>: all the configured stub or forward servers failed, at zone .
. . .
May 27 07:17:28 ipfire unbound: [1514:0] error: SERVFAIL <HIDDEN_URL. A IN>: all the configured stub or forward servers failed, at zone .
May 27 07:17:28 ipfire unbound: [1514:0] error: SERVFAIL <HIDDEN_URL. A IN>: all the configured stub or forward servers failed, at zone .
May 27 07:17:28 ipfire unbound: [1514:0] error: SERVFAIL <HIDDEN_URL. A IN>: all the configured stub or forward servers failed, at zone .
May 27 07:17:28 ipfire unbound: [1514:0] error: SERVFAIL <HIDDEN_URL. SRV IN>: all the configured stub or forward servers failed, at zone .
May 27 07:17:28 ipfire unbound: [1514:0] error: SERVFAIL <HIDDEN_URL. A IN>: all the configured stub or forward servers failed, at zone .
May 27 07:17:28 ipfire collectd[3088]: rrdtool plugin: Shutting down the queue thread.
May 27 07:17:28 ipfire unbound: [1514:0] error: SERVFAIL <HIDDEN_URL. A IN>: all the configured stub or forward servers failed, at zone .
May 27 07:17:28 ipfire suricata: Signal Received.  Stopping engine.
May 27 07:17:29 ipfire unbound: [1514:0] error: SERVFAIL <HIDDEN_URL. A IN>: all the configured stub or forward servers failed, at zone .
May 27 07:17:29 ipfire unbound: [1514:0] error: SERVFAIL <HIDDEN_URL. A IN>: all the configured stub or forward servers failed, at zone .
May 27 07:17:29 ipfire suricata: (W-NFQ#0) Treated: Pkts 44, Bytes 14535, Errors 0
May 27 07:17:29 ipfire suricata: (W-NFQ#0) Verdict: Accepted 44, Dropped 0, Replaced 0
May 27 07:17:29 ipfire suricata: (W-NFQ#1) Treated: Pkts 1, Bytes 104, Errors 0
May 27 07:17:29 ipfire suricata: (W-NFQ#1) Verdict: Accepted 1, Dropped 0, Replaced 0
May 27 07:17:30 ipfire suricata: (W-NFQ#2) Treated: Pkts 5, Bytes 388, Errors 0
May 27 07:17:30 ipfire suricata: (W-NFQ#2) Verdict: Accepted 5, Dropped 0, Replaced 0
May 27 07:17:30 ipfire suricata: (W-NFQ#3) Treated: Pkts 0, Bytes 0, Errors 0
May 27 07:17:30 ipfire suricata: (W-NFQ#3) Verdict: Accepted 0, Dropped 0, Replaced 0
May 27 07:17:30 ipfire unbound: [1514:0] error: SERVFAIL <HIDDEN_URL. A IN>: all the configured stub or forward servers failed, at zone .
May 27 07:17:31 ipfire dhcpcd[728]: sending commands to dhcpcd process
May 27 07:19:38 ipfire kernel: igb 0000:03:00.0 orange0: igb: orange0 NIC Link is Down
May 27 07:19:39 ipfire kernel: igb 0000:03:00.0 orange0: igb: orange0 NIC Link is Up 100 Mbps Full Duplex, Flow Control: RX/TX

May 27 11:09:25 ipfire syslogd 1.5.1: restart (remote reception).

Near 8:00 AM I tried to access the IPFire WebUI and could not. Then I force powered down the IPFire box and rebooted - no joy. The IPFire box would not boot.

I saw a patch when searching for SERVFAIL. I didn’t know if this is the same issue or not.

EDIT:
My system info:

IPFire version IPFire 2.25 (x86_64) - core144
Pakfire version 2.25-x86_64
Kernel version Linux ipfire.localdomain 4.14.173-ipfire #1 SMP Thu Mar 12 12:07:28 GMT 2020 x86_64 Intel(R) Celeron(R) CPU J1900 @ 1.99GHz GenuineIntel GNU/Linux
2

I have SERVFAIL lines in /var/log/messages but not that many.

3

Hi,

I am getting the exact same error after an upgrade, only with a few more layers of heartbreak.

First, check if your ipfire networks are up and if there isn’t something blocking access.

I’m betting the server can’t reach the forwsrd zones and once you fix your interfaces, everything will be gold again

4

I have gotten a lot of SERVFAILs over the year.
One thing that I did, (not the fixall), is to add more DNS servers. I went from 4 to 7 DNS servers. Hope this is helpful to you and others that stumble on to this.

Update: new total 568,466 on Jan 19, 2021 (It appears that the count is not increasing as fast. 7 DNS servers appears to be better than 4 DNS servers in this situation.)