C203 - DNS down

Network DNS
1

IPfire with C203. There was about 5 minutes power outage in the whole city yesterday evening and I am an ignorant without UPS, that means all my computers were down including IPfire.

I found today that IPfire cannot resolved DNS, it returns SERVFAIL. This IPfire is still in a test mode, so I have just one PC behind it, so it took me longer to notice the issue…

DNS is configured in UDP mode and it was working on Monday.

I checked DNS status on IPfire and DNS service is down…

image
image1024×786 68.6 KB

DNS servers status is Broken.

[root@ipfire ~]# /etc/init.d/knot-resolver status
/usr/bin/knot-resolver is not running but /var/run/knot-resolver.pid exists.

Filesystem has more than 100GB of free space.

It looks like some DB was corrupted and knot resolver is in trouble:

/var/log/messages:

Jul 28 23:33:47 ipfire kresd[1610]: [cache ] LMDB error: MDB_PAGE_FULL: Internal error - page has no more space 
Jul 28 23:33:47 ipfire last message buffered 1 times
Jul 28 23:33:47 ipfire kresd[14272]: [cache ] LMDB error: MDB_PAGE_FULL: Internal error - page has no more space 
Jul 28 23:33:47 ipfire last message buffered 1 times
Jul 28 23:34:08 ipfire kresd[1610]: [cache ] LMDB error: MDB_PAGE_FULL: Internal error - page has no more space 
Jul 28 23:34:08 ipfire kresd[14272]: [cache ] LMDB error: MDB_PAGE_FULL: Internal error - page has no more space

Power outage was about at 19:45. This is /var/log/messages and I see that the issue is there after boot. The first LMDB related error is at 22:55:

Jul 28 19:51:26 ipfire kernel: EXT4-fs (sda4): re-mounted 39901c8f-b588-41cd-807b-677f1afd6f8a.
Jul 28 19:51:26 ipfire kernel: EXT4-fs (sda4): re-mounted 39901c8f-b588-41cd-807b-677f1afd6f8a r/w.
Jul 28 19:51:26 ipfire kernel: EXT4-fs (sda1): mounted filesystem 824a2458-da0f-45e2-a096-cc31e4274ea8 r/w with ordered data mode. Quota mode: none.
Jul 28 19:51:26 ipfire kernel: Adding 918264k swap on /dev/sda3.  Priority:1 extents:1 across:918264k SS
Jul 28 19:51:31 ipfire acpid: starting up with netlink and the input layer
Jul 28 19:51:31 ipfire acpid: 1 rule loaded
Jul 28 19:51:31 ipfire acpid: waiting for events: event logging is off
Jul 28 19:51:34 ipfire knot_resolver.manager.server: Loading configuration from '/etc/knot-resolver/config.yaml' file.
Jul 28 19:51:34 ipfire knot_resolver.manager.logger: Changing logging level to 'INFO'
Jul 28 19:51:34 ipfire knot_resolver.controller: Starting service manager auto-selection...
Jul 28 19:51:34 ipfire knot_resolver.controller: Available subprocess controllers are ('supervisord',)
Jul 28 19:51:34 ipfire knot_resolver.controller: Selected controller 'supervisord'
Jul 28 19:51:34 ipfire knot_resolver.controller.supervi: We want supervisord to restart us when needed, we will therefore exec() it and let it start us 
again.
Jul 28 19:51:34 ipfire knot_resolver.manager.server: Exec requested with arguments: ['/usr/bin/supervisord', 'supervisord', '--configuration', '/run/kno
t-resolver/supervisord.conf']


Jul 28 19:51:34 ipfire knot_resolver.manager.server: Exec requested with arguments: ['/usr/bin/supervisord', 'supervisord', '--configuration', '/run/kno
t-resolver/supervisord.conf']
Jul 28 19:51:36 ipfire supervisord: RPC interface 'patch_logger' initialized
Jul 28 19:51:36 ipfire supervisord: RPC interface 'manager_integration' initialized
Jul 28 19:51:36 ipfire supervisord: RPC interface 'sd_notify' initialized
Jul 28 19:51:36 ipfire supervisord: RPC interface 'supervisor' initialized
Jul 28 19:51:36 ipfire supervisord: RPC interface 'fast' initialized
Jul 28 19:51:36 ipfire supervisord: Server 'unix_http_server' running without any HTTP authentication checking
Jul 28 19:51:36 ipfire supervisord: supervisord started with pid 1554
Jul 28 19:51:36 ipfire supervisord: notify: injected $NOTIFY_SOCKET into event loop
Jul 28 19:51:37 ipfire supervisord: spawned: 'manager' with pid 1600
Jul 28 19:51:40 ipfire knot_resolver.manager.server: Loading configuration from '/etc/knot-resolver/config.yaml' file.
Jul 28 19:51:40 ipfire knot_resolver.manager.logger: Changing logging level to 'INFO'
Jul 28 19:51:40 ipfire knot_resolver.controller: Starting service manager auto-selection...
Jul 28 19:51:40 ipfire knot_resolver.controller: Available subprocess controllers are ('supervisord',)
Jul 28 19:51:40 ipfire knot_resolver.controller: Selected controller 'supervisord'
Jul 28 19:51:40 ipfire knot_resolver.controller.supervi: Supervisord is already running, we will just update its config...
Jul 28 19:51:41 ipfire supervisord: spawned: 'policy-loader' with pid 1605
Jul 28 19:51:41 ipfire supervisord: success: policy-loader entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
Jul 28 19:51:41 ipfire kresd[1605]: [cache ] incompatible cache database detected, purging 
Jul 28 19:51:41 ipfire kresd[1605]: [cache ] Cache top initialized using existing data (generic). 
Jul 28 19:51:41 ipfire supervisord: exited: policy-loader (exit status 0; expected)
Jul 28 19:51:42 ipfire supervisord: spawned: 'kresd0' with pid 1608
Jul 28 19:51:42 ipfire kresd[1608]: [system] path = /run/knot-resolver/control/0 
Jul 28 19:51:42 ipfire kresd[1608]: [cache ] Cache top initialized using existing data (generic). 
Jul 28 19:51:42 ipfire kresd[1608]: [defer ] Initializing defer... 
Jul 28 19:51:42 ipfire kresd[1608]: [defer ] Defer initialized (generic). 
Jul 28 19:51:42 ipfire kresd[1608]: [defer ] Defer configuration:   Expected cpus/procs:     2   Max waiting requests:   64.0 MiB   Request timeout:         1.0 s    Idle:                    1.0 ms   UDP phase:             400.0 us   Non-UDP phase:         400.0 us   Priority levels:        25 (4 main levels, 8 sublevels) + UDP   KRU capacity:          524.3 k (4.2 MiB)   Decay:                   0.012 % per ms (32-bit max: 524280)     Half-life:             5.7 s      Priority rise in:     22.7 s      Counter reset in:     90.8 s    Rate limits for crossing priority levels as single CPU utilization:                             1             2             3           max          v4/32 :        0.012 %       0.195 %       3.125 %      49.999 %          v4/24 :        0.391 %       6.250 %     100.000 %    1599.976 %          v4/20 :        3.125 %      50.000 %     800.000 %   12799.805 %          v4/18 :        9.375 %     150.000 %    2400.000 %   38399.414 %          v6/128:        0.012 %       0.195 %       3.125 %      49.999 %          v6/64 :        0.024 %       0.391 %       6.250 %      99.998 %          v6/56 :        0.037 %       0.586 %       9.375 %     149.998 %          v6/48 :        0.049 %       0.781 %      12.500 %     199.997 %          v6/32 :        0.781 %      12.500 %     200.000 %    3199.951 %   Instant limits for crossing priority levels as CPU time:                             1             2             3           max          v4/32 :          1.0 ms       16.0 ms      256.0 ms        4.1 s           v4/24 :         32.0 ms      512.0 ms        8.2 s       131.1 s           v4/20 :        256.0 ms        4.1 s        65.5 s      1048.6 s           v4/18 :        768.0 ms       12.3 s       196.6 s      3145.7 s           v6/128:          1.0 ms       16.0 ms      256.0 ms        4.1 s           v6/64 :          2.0 ms       32.0 ms      512.0 ms        8.2 s           v6/56 :          3.0 ms       48.0 ms      768.0 ms       12.3 s           v6/48 :          4.0 ms       64.0 ms   
Jul 28 19:51:42 ipfire supervisord: success: kresd0 entered RUNNING state, process sent notification via $NOTIFY_SOCKET

...

Jul 28 22:55:22 ipfire kresd[1609]: [cache ] LMDB error: MDB_CURSOR_FULL: Internal error - cursor stack limit reached 
Jul 28 22:55:22 ipfire kresd[1609]: [system] assertion "ret == kr_error(ENOENT)" failed in peek_nosync@../lib/cache/peek.c:134 
Jul 28 22:55:22 ipfire supervisord: exited: kresd0 (terminated by SIGABRT; not expected)
Jul 28 22:55:23 ipfire supervisord: spawned: 'kresd0' with pid 14045
Jul 28 22:55:23 ipfire kresd[14045]: [system] path = /run/knot-resolver/control/0 
Jul 28 22:55:23 ipfire kresd[14045]: [cache ] Cache top initialized using existing data (generic). 
Jul 28 22:55:23 ipfire kresd[14045]: [defer ] Using existing defer data (generic). 
Jul 28 22:55:23 ipfire supervisord: success: kresd0 entered RUNNING state, process sent notification via $NOTIFY_SOCKET
Jul 28 22:55:23 ipfire kresd[14045]: [taupd ] refreshing TA for . 
Jul 28 22:55:23 ipfire kresd[14045]: [cache ] LMDB error: MDB_CURSOR_FULL: Internal error - cursor stack limit reached 
Jul 28 22:55:23 ipfire kresd[14045]: [system] assertion "ret == kr_error(ENOENT)" failed in peek_nosync@../lib/cache/peek.c:134 
Jul 28 22:55:23 ipfire supervisord: exited: kresd0 (terminated by SIGABRT; not expected)
Jul 28 22:55:24 ipfire supervisord: spawned: 'kresd0' with pid 14046
Jul 28 22:55:24 ipfire kresd[14046]: [system] path = /run/knot-resolver/control/0 
Jul 28 22:55:25 ipfire kresd[14046]: [cache ] Cache top initialized using existing data (generic). 
Jul 28 22:55:25 ipfire kresd[14046]: [defer ] Using existing defer data (generic). 
Jul 28 22:55:25 ipfire supervisord: success: kresd0 entered RUNNING state, process sent notification via $NOTIFY_SOCKET
...

Jul 28 23:33:47 ipfire kresd[1610]: [cache ] LMDB error: MDB_PAGE_FULL: Internal error - page has no more space 
Jul 28 23:33:47 ipfire last message buffered 1 times
Jul 28 23:33:47 ipfire kresd[14272]: [cache ] LMDB error: MDB_PAGE_FULL: Internal error - page has no more space 
Jul 28 23:33:47 ipfire last message buffered 1 times
Jul 28 23:34:08 ipfire kresd[1610]: [cache ] LMDB error: MDB_PAGE_FULL: Internal error - page has no more space 
Jul 28 23:34:08 ipfire kresd[14272]: [cache ] LMDB error: MDB_PAGE_FULL: Internal error - page has no more space 
Jul 28 23:34:15 ipfire supervisord: captured stdio output from kresd1[1610] (stderr): mdb.c:3356: Assertion 'len >= 0 && id <= env->me_pglast' failed in
 mdb_freelist_save()
Jul 28 23:34:15 ipfire supervisord: exited: kresd1 (terminated by SIGABRT; not expected)
Jul 28 23:34:15 ipfire supervisord: captured stdio output from kresd0[14272] (stderr): mdb.c:3356: Assertion 'len >= 0 && id <= env->me_pglast' failed i
n mdb_freelist_save()
Jul 28 23:34:15 ipfire supervisord: spawned: 'kresd1' with pid 16462
Jul 28 23:34:15 ipfire supervisord: exited: kresd0 (terminated by SIGABRT; not expected)
Jul 28 23:34:15 ipfire supervisord: spawned: 'kresd0' with pid 16463
Jul 28 23:34:15 ipfire kresd[16462]: [system] path = /run/knot-resolver/control/1 
Jul 28 23:34:15 ipfire supervisord: captured stdio output from kresd1[16462] (stderr): mdb.c:3356: Assertion 'len >= 0 && id <= env->me_pglast' failed i
n mdb_freelist_save()
Jul 28 23:34:15 ipfire kresd[16463]: [system] path = /run/knot-resolver/control/0 
Jul 28 23:34:15 ipfire supervisord: exited: kresd1 (terminated by SIGABRT; not expected)
Jul 28 23:34:15 ipfire supervisord: captured stdio output from kresd0[16463] (stderr): mdb.c:3356: Assertion 'len >= 0 && id <= env->me_pglast' failed i
n mdb_freelist_save()

...
Jul 28 23:34:19 ipfire supervisord: captured stdio output from kresd1[16467] (stderr): mdb.c:3356: Assertion 'len >= 0 && id <= env->me_pglast' failed i
n mdb_freelist_save()
Jul 28 23:34:19 ipfire supervisord: exited: kresd1 (terminated by SIGABRT; not expected)
Jul 28 23:34:22 ipfire supervisord: spawned: 'kresd0' with pid 16468
Jul 28 23:34:22 ipfire supervisord: spawned: 'kresd1' with pid 16469
Jul 28 23:34:22 ipfire kresd[16468]: [system] path = /run/knot-resolver/control/0 
Jul 28 23:34:22 ipfire supervisord: captured stdio output from kresd0[16468] (stderr): mdb.c:3356: Assertion 'len >= 0 && id <= env->me_pglast' failed i
n mdb_freelist_save()
Jul 28 23:34:22 ipfire kresd[16469]: [system] path = /run/knot-resolver/control/1 
Jul 28 23:34:22 ipfire supervisord: exited: kresd0 (terminated by SIGABRT; not expected)
Jul 28 23:34:22 ipfire supervisord: captured stdio output from kresd1[16469] (stderr): mdb.c:3356: Assertion 'len >= 0 && id <= env->me_pglast' failed i
n mdb_freelist_save()
Jul 28 23:34:22 ipfire supervisord: gave up: kresd0 entered FATAL state, too many start retries too quickly
Jul 28 23:34:22 ipfire supervisord: exited: kresd1 (terminated by SIGABRT; not expected)
Jul 28 23:34:22 ipfire supervisord: gave up: kresd1 entered FATAL state, too many start retries too quickly
Jul 28 23:34:23 ipfire hostapd: blue0: AP-STA-DISCONNECTED 10:30:47:f9:88:93
Jul 28 23:34:23 ipfire knot_resolver.manager.manager: Subprocess 'kresd:kresd0' is in FATAL state!
Jul 28 23:34:23 ipfire knot_resolver.manager.manager: Subprocess 'kresd:kresd1' is in FATAL state!
Jul 28 23:34:23 ipfire knot_resolver.manager.manager: Instability detected. Dropping known list of workers and reloading it from the system.
Jul 28 23:34:23 ipfire knot_resolver.manager.manager: Workers reloaded. Applying old config....
Jul 28 23:34:23 ipfire supervisord: spawned: 'policy-loader' with pid 16470
Jul 28 23:34:23 ipfire supervisord: success: policy-loader entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
Jul 28 23:34:23 ipfire supervisord: captured stdio output from policy-loader[16470] (stderr): mdb.c:3356: Assertion 'len >= 0 && id <= env->me_pglast' f
ailed in mdb_freelist_save()
Jul 28 23:34:23 ipfire supervisord: exited: policy-loader (terminated by SIGABRT; not expected)
Jul 28 23:34:24 ipfire knot_resolver.manager.manager: Failed attempting to fix an error. Forcefully shutting down. Traceback (most recent call last):   
File "/usr/lib/python3.10/site-packages/knot_resolver/manager/manager.py", line 401, in _instability_handler     await self._config_store.renew()   File
 "/usr/lib/python3.10/site-packages/knot_resolver/manager/config_store.py", line 42, in renew     await self.update(self._config, force)   File "/usr/li
b/python3.10/site-packages/knot_resolver/manager/config_store.py", line 31, in update     raise KresManagerBaseError("Configuration validation failed. T
he reasons are:\n - " + "\n - ".join(errs)) knot_resolver.manager.exceptions.KresManagerBaseError: Configuration validation failed. The reasons are:  - 
kresd 'policy-loader' process exited unexpectedly. The configuration may be invalid. Please check the log.
Jul 28 23:34:24 ipfire knot_resolver.manager.manager: Collecting all remaining workers...
Jul 28 23:34:24 ipfire knot_resolver.manager.manager: Terminating...
Jul 28 23:34:24 ipfire supervisord: ignoring ready notification from unregistered PID=1600
Jul 28 23:34:24 ipfire knot_resolver.manager.server: Stopping API service...
Jul 28 23:34:24 ipfire knot_resolver.manager.server: Stopping kresd manager...
Jul 28 23:34:24 ipfire supervisord: waiting for cache-gc, manager to die
Jul 28 23:34:24 ipfire knot_resolver.manager.server: The manager run for 13364 seconds...
Jul 28 23:34:24 ipfire knot_resolver.manager.server: Received SIGINT while already shutting down. Ignoring. If you want to forcefully stop the manager r
ight now, use SIGTERM.
Jul 28 23:34:24 ipfire supervisord: stopped: manager (exit status 1)
Jul 28 23:34:25 ipfire supervisord: stopped: cache-gc (exit status 0)

I attach charts with CPU, memory usage and red traffic.

image
image979×605 75 KB

image
image971×579 74.3 KB

image
image967×552 61.5 KB

I assume that some of these files is a troubelmaker:

[root@ipfire ~]# ls -ltr /var/cache/knot-resolver/
total 262164
-rw------- 1 knot-resolver root  16777472 Jul 28 23:34 top
-rw-r----- 1 knot-resolver root      8192 Jul 28 23:34 lock.mdb
-rw-r----- 1 knot-resolver root 251654144 Jul 28 23:34 data.mdb

or here:

[root@ipfire ~]# ls -ltr /run/knot-resolver/ruledb/
total 932
-rw-r----- 1 knot-resolver root 950272 Jul 28 23:24 data.mdb
-rw-r----- 1 knot-resolver root   8192 Jul 28 23:34 lock.mdb

SERVFAIL is returned by Linux desktop. When I ask directly IPfire, I received error (DNS is down)

root@linux:~# host ipfire.org 192.168.22.1
;; communications error to 192.168.22.1#53: connection refused
;; communications error to 192.168.22.1#53: connection refused
;; no servers could be reached
2

I cannot start it. I activated debug mode (/etc/knot-resolver/config.yaml) but I do not see any clue in the log

[root@ipfire ~]# /etc/init.d/knot-resolver start

/var/log/messages

Jul 29 17:33:21 ipfire knot_resolver.manager.server: Loading configuration from '/etc/knot-resolver/config.yaml' file.
Jul 29 17:33:21 ipfire knot_resolver.manager.logger: Changing logging level to 'DEBUG'
Jul 29 17:33:21 ipfire knot_resolver.controller: Starting service manager auto-selection...
Jul 29 17:33:21 ipfire knot_resolver.controller: Available subprocess controllers are ('supervisord',)
Jul 29 17:33:21 ipfire knot_resolver.controller: Selected controller 'supervisord'
Jul 29 17:33:21 ipfire knot_resolver.controller.supervi: We want supervisord to restart us when needed, we will therefore exec() it and let it start us again.
Jul 29 17:33:21 ipfire knot_resolver.manager.server: Exec requested with arguments: ['/usr/bin/supervisord', 'supervisord', '--configuration', '/run/knot-resolver/supervisord.conf']
Jul 29 17:33:23 ipfire supervisord: RPC interface 'patch_logger' initialized
Jul 29 17:33:23 ipfire supervisord: RPC interface 'manager_integration' initialized
Jul 29 17:33:23 ipfire supervisord: RPC interface 'sd_notify' initialized
Jul 29 17:33:23 ipfire supervisord: RPC interface 'supervisor' initialized
Jul 29 17:33:23 ipfire supervisord: RPC interface 'fast' initialized
Jul 29 17:33:23 ipfire supervisord: Server 'unix_http_server' running without any HTTP authentication checking
Jul 29 17:33:23 ipfire supervisord: supervisord started with pid 16763
Jul 29 17:33:23 ipfire supervisord: notify: injected $NOTIFY_SOCKET into event loop
Jul 29 17:33:24 ipfire supervisord: spawned: 'manager' with pid 16765
Jul 29 17:33:25 ipfire kernel: DROP_CTINVALID IN=blue0 OUT=red0 MAC=3c:77:e6:02:71:99:0e:d5:f3:2c:94:bc:08:00 SRC=192.168.23.101 DST=142.251.209.3 LEN=76 TOS=0x00 PREC=0x00 TTL=63 ID=18353 DF PROTO=TCP SPT=42966 DPT=443 WINDOW=300 RES=0x00 ACK PSH FIN URGP=0 
Jul 29 17:33:27 ipfire knot_resolver.manager.server: Loading configuration from '/etc/knot-resolver/config.yaml' file.
Jul 29 17:33:27 ipfire knot_resolver.manager.logger: Changing logging level to 'DEBUG'
Jul 29 17:33:27 ipfire knot_resolver.controller: Starting service manager auto-selection...
Jul 29 17:33:27 ipfire knot_resolver.controller: Available subprocess controllers are ('supervisord',)
Jul 29 17:33:27 ipfire knot_resolver.controller: Selected controller 'supervisord'
Jul 29 17:33:27 ipfire knot_resolver.controller.supervi: Supervisord is already running, we will just update its config...
Jul 29 17:33:28 ipfire supervisord: spawned: 'policy-loader' with pid 16772
Jul 29 17:33:28 ipfire supervisord: success: policy-loader entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
Jul 29 17:33:28 ipfire supervisord: captured stdio output from policy-loader[16772] (stderr): mdb.c:3356: Assertion 'len >= 0 && id <= env->me_pglast' failed in mdb_freelist_save()
Jul 29 17:33:28 ipfire supervisord: exited: policy-loader (terminated by SIGABRT; not expected)
Jul 29 17:33:29 ipfire knot_resolver.manager.server: Initial config verification failed with error: kresd 'policy-loader' process exited unexpectedly. The configuration may be invalid. Please check the log.
Jul 29 17:33:29 ipfire supervisord: exited: manager (exit status 1; not expected)
Jul 29 17:33:30 ipfire supervisord: spawned: 'manager' with pid 16774
Jul 29 17:33:33 ipfire knot_resolver.manager.server: Loading configuration from '/etc/knot-resolver/config.yaml' file.
Jul 29 17:33:33 ipfire knot_resolver.manager.logger: Changing logging level to 'DEBUG'
Jul 29 17:33:33 ipfire knot_resolver.controller: Starting service manager auto-selection...
Jul 29 17:33:33 ipfire knot_resolver.controller: Available subprocess controllers are ('supervisord',)
Jul 29 17:33:33 ipfire knot_resolver.controller: Selected controller 'supervisord'
Jul 29 17:33:33 ipfire knot_resolver.controller.supervi: Supervisord is already running, we will just update its config...
Jul 29 17:33:34 ipfire supervisord: spawned: 'policy-loader' with pid 16776
Jul 29 17:33:34 ipfire supervisord: success: policy-loader entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
Jul 29 17:33:34 ipfire supervisord: captured stdio output from policy-loader[16776] (stderr): mdb.c:3356: Assertion 'len >= 0 && id <= env->me_pglast' failed in mdb_freelist_save()
Jul 29 17:33:34 ipfire supervisord: exited: policy-loader (terminated by SIGABRT; not expected)
Jul 29 17:33:35 ipfire knot_resolver.manager.server: Initial config verification failed with error: kresd 'policy-loader' process exited unexpectedly. The configuration may be invalid. Please check the log.
Jul 29 17:33:35 ipfire supervisord: exited: manager (exit status 1; not expected)
Jul 29 17:33:37 ipfire supervisord: spawned: 'manager' with pid 16778
Jul 29 17:33:40 ipfire knot_resolver.manager.server: Loading configuration from '/etc/knot-resolver/config.yaml' file.
Jul 29 17:33:40 ipfire knot_resolver.manager.logger: Changing logging level to 'DEBUG'
Jul 29 17:33:40 ipfire knot_resolver.controller: Starting service manager auto-selection...
Jul 29 17:33:40 ipfire knot_resolver.controller: Available subprocess controllers are ('supervisord',)
Jul 29 17:33:40 ipfire knot_resolver.controller: Selected controller 'supervisord'
Jul 29 17:33:40 ipfire knot_resolver.controller.supervi: Supervisord is already running, we will just update its config...
Jul 29 17:33:41 ipfire supervisord: spawned: 'policy-loader' with pid 16780
Jul 29 17:33:41 ipfire supervisord: success: policy-loader entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
Jul 29 17:33:41 ipfire supervisord: captured stdio output from policy-loader[16780] (stderr): mdb.c:3356: Assertion 'len >= 0 && id <= env->me_pglast' failed in mdb_freelist_save()
Jul 29 17:33:41 ipfire supervisord: exited: policy-loader (terminated by SIGABRT; not expected)
Jul 29 17:33:42 ipfire knot_resolver.manager.server: Initial config verification failed with error: kresd 'policy-loader' process exited unexpectedly. The configuration may be invalid. Please check the log.
Jul 29 17:33:42 ipfire supervisord: exited: manager (exit status 1; not expected)
Jul 29 17:33:45 ipfire supervisord: spawned: 'manager' with pid 16783
Jul 29 17:33:48 ipfire knot_resolver.manager.server: Loading configuration from '/etc/knot-resolver/config.yaml' file.
Jul 29 17:33:48 ipfire knot_resolver.manager.logger: Changing logging level to 'DEBUG'
Jul 29 17:33:48 ipfire knot_resolver.controller: Starting service manager auto-selection...
Jul 29 17:33:48 ipfire knot_resolver.controller: Available subprocess controllers are ('supervisord',)
Jul 29 17:33:48 ipfire knot_resolver.controller: Selected controller 'supervisord'
Jul 29 17:33:48 ipfire knot_resolver.controller.supervi: Supervisord is already running, we will just update its config...
Jul 29 17:33:49 ipfire supervisord: spawned: 'policy-loader' with pid 16785
Jul 29 17:33:49 ipfire supervisord: success: policy-loader entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
Jul 29 17:33:49 ipfire supervisord: captured stdio output from policy-loader[16785] (stderr): mdb.c:3356: Assertion 'len >= 0 && id <= env->me_pglast' failed in mdb_freelist_save()
Jul 29 17:33:49 ipfire supervisord: exited: policy-loader (terminated by SIGABRT; not expected)
Jul 29 17:33:50 ipfire knot_resolver.manager.server: Initial config verification failed with error: kresd 'policy-loader' process exited unexpectedly. The configuration may be invalid. Please check the log.
Jul 29 17:33:50 ipfire supervisord: exited: manager (exit status 1; not expected)
Jul 29 17:33:51 ipfire supervisord: manager process entered FATAL state! Shutting down
Jul 29 17:33:51 ipfire supervisord: gave up: manager entered FATAL state, too many start retries too quickly

Files in directory /run/knot-resolver/ruledb/ were not updated:

[root@ipfire ~]# ls -ltr /var/cache/knot-resolver/
total 262164
-rw------- 1 knot-resolver root  16777472 Jul 28 23:34 top
-rw-r----- 1 knot-resolver root      8192 Jul 29 17:33 lock.mdb
-rw-r----- 1 knot-resolver root 251654144 Jul 29 17:33 data.mdb

[root@ipfire ~]# ls -ltr /run/knot-resolver/ruledb/
total 932
-rw-r----- 1 knot-resolver root 950272 Jul 28 23:24 data.mdb
-rw-r----- 1 knot-resolver root   8192 Jul 28 23:34 lock.mdb
3

I deleted mdb files in /var/cache/knot-resolver/

rm /var/cache/knot-resolver/*mdb

and started resolver

[root@ipfire ~]# /etc/init.d/knot-resolver start

It looks like I fixed the problem.
That looks like troublemaker was file /var/cache/knot-resolver/data.mdb

This is the current status of DB files:

[root@ipfire ~]# ls -ltr /var/cache/knot-resolver/*
-rw-r----- 1 knot-resolver root      8192 Jul 29 18:01 /var/cache/knot-resolver/lock.mdb
-rw------- 1 knot-resolver root  16777472 Jul 29 18:01 /var/cache/knot-resolver/top
-rw-r----- 1 knot-resolver root 251654144 Jul 29 18:01 /var/cache/knot-resolver/data.mdb

[root@ipfire ~]# ls -ltr /run/knot-resolver/ruledb/
total 932
-rw-r----- 1 knot-resolver root   8192 Jul 29 17:56 lock.mdb
-rw-r----- 1 knot-resolver root 950272 Jul 29 17:56 data.mdb
4

Just updated system, memory and traffic graphs after one week later.

image
image977×599 79 KB

image
image971×581 73.7 KB

image
image975×555 59 KB

I just noticed small detail, CPU graph doesn’t have extra frame like other graphs. That chart frame helps me to cut chart from the screenshot and it will be nice to fix CPU chart to be drawn in the same way…

/srv/web/ipfire/cgi-bin/memory.cgi

&Header::showhttpheaders();
&Header::openpage($Lang::tr{'memory information'}, 1, '');
&Header::openbigbox('100%', 'left');

&Header::openbox('100%', 'center', "Memory $Lang::tr{'graph'}");
&Graphs::makegraphbox("memory.cgi","memory","day");
&Header::closebox();

/srv/web/ipfire/cgi-bin/system.cgi

CPU page is created in a simpler way, it miss call of openbigbox, openbox, charts are created with &Header::graph but charts in all other status pages are created with &Graphs::makegraphbox.

&Header::showhttpheaders();

&Header::openpage($Lang::tr{'status information'}, 1, '');

# Processor Graph
&Header::graph("$Lang::tr{'processors'}", "system.cgi", "cpu", "day");

A patch. Please, review, clean, improve and schedule for the next release:

[root@ipfire cgi-bin]# diff -u system0.cgi system.cgi 
--- system0.cgi	2026-07-14 14:37:53.126036013 +0200
+++ system.cgi	2026-08-04 10:28:58.066360949 +0200
@@ -34,16 +34,28 @@
 &Header::showhttpheaders();
 
 &Header::openpage($Lang::tr{'status information'}, 1, '');
+&Header::openbigbox('100%', 'left');
 
 # Processor Graph
-&Header::graph("$Lang::tr{'processors'}", "system.cgi", "cpu", "day");
+&Header::openbox('100%', 'center', "$Lang::tr{'processors'}");
+&Graphs::makegraphbox("system.cgi","cpu","day");
+&Header::closebox();
 
 # CPU Frequency
 if ( -e "$mainsettings{'RRDLOG'}/collectd/localhost/cpufreq-0/cpufreq.rrd"){
-	&Header::graph("$Lang::tr{'cpu frequency'}", "system.cgi", "cpufreq", "day");
+        &Header::openbox('100%', 'center', "$Lang::tr{'cpu frequency'}");
+        &Graphs::makegraphbox("system.cgi","cpufreq","day");
+        &Header::closebox();
 }
 
 # Load Average
-&Header::graph("$Lang::tr{'load average'}", "system.cgi", "load", "day");
+&Header::openbox('100%', 'center', "$Lang::tr{'load average'}");
+&Graphs::makegraphbox("system.cgi","load","day");
+&Header::closebox();
 
+&Header::closebigbox();
 &Header::closepage();

system.cgi page with the “patch”, compare how nice it is:

image
image971×641 74.5 KB

When I search “Header::graph”, I assume that page ids.cgi should be “fixed” too:

[root@ipfire ~]# grep -l "Header::graph" /srv/web/ipfire/cgi-bin/*.cgi
/srv/web/ipfire/cgi-bin/ids.cgi
grep: /srv/web/ipfire/cgi-bin/logs.cgi: Is a directory
/srv/web/ipfire/cgi-bin/system.cgi