High memory consumption: DNS Proxy Server

I have on a Proxmox version: IPFire 2.29 (x86_64) - Core-Update 201 Development Build: master/a5909296 (I don’t know if this same thing happened in previous versions) but the DNS Proxy Server consumption seems excessive to me.

DNS Proxy Server RUNNING 1.73 GB RSS

Sometimes, even more, and I have to reduce the amount of memory consumed by running /etc/init.d/unbound restart.

I have 4GB of RAM allocated to this Proxmox, which seems sufficient to me, but in this case, it appears that it isn’t…

Today I found myself unable to browse the internet because the DNS Proxy Server was down. I had to restart it via SSH.

Is this happening to anyone else?.

The question is how many lists do you have selected in your DNS firewall, after selecting all of them IpFire with 8GB RAM uses over 50%

Could that be it?

Thanks, I’ll try disabling them all to see if it lowers the usage.

If you disable all of them, RAM usage should return to the level before enabling them. Enabling the DNS firewall significantly uses RAM. In my opinion, IPFire should currently have 8GB if you want to use the DNS firewall.

It should also work on 4GB, but it will start using swap. For me, a good setup is one that doesn’t use swap.

Not that I have any direct issues, but I can also add the additional ram consumtion as a fact… it jumped from 5% to about 15% …

After enabling three DNS Firewall categories (Malware, Phishing, and Violence), memory usage on IPFire jumped from approximately 5.9% to 14.3% on a system with 16GB RAM. That’s roughly a 10 percentage point increase, translating to about 1.4GB of additional memory consumed by the RPZ zone data.

The Phishing list (458K domains) is likely responsible for the bulk of that, given its size relative to Malware (253K domains) and Violence (234 domains).

Buffered memory dropped from 6.87% to 1.81% and cached memory from 18.49% to 4.79% post-enablement, which is expected β€” unbound holds the RPZ zone data in its own memory space, showing as β€œused” rather than cache/buffers.

With three categories enabled, the system still sits at 79% free memory. Extrapolating from this, enabling all 14 categories (approximately 1.47 million domains) would likely bring usage to around 30-35%, which remains comfortable for my dedicated firewall, but perhaps a bit much for others.

This only looks at the memory necessary to hold the lists.
My experience is, that after changes to the lists and reload the old memory isn’t freed but consumption is old+new. This means that after some time ( and modifications ) memory is full. The time of this event is determined by memory size. With your 16GB this may be acceptable, maybe it doesn’t occur because of core updates occur more early than the memory fill-up.
With smaller memory sizes this is not acceptable.

Which effectively means that many small RAM solutions will be unable to run this, I am thinking Raspberrys, and similar low-power SBCs that are otherwise perfectly capable IPFire boxes.

On my 16GB system, just three categories (Malware, Phishing, Violence) consumed approximately 1.4GB. A Raspberry Pi 4 with 2GB would be at the edge with even a single category enabled, and at serious risk of the OOM killer taking out unbound β€” which means no DNS, which means no internet.

A warning in the WUI based on available RAM before applying categories would go a long way.

I find the situation odd, I would never give a firewall lean resources, yet so many seem to shop around for just that, small SBC’s and small power impact, and I fear many are not going to be happy with their earlier hardware decisions.
And all the IPFire low resources firewall argumentation is no longer valid.

The warning exits in the wiki. :wink:

Bernhard, I know, I have read it, but you know as well as I that warnings in the WUI, the factual interface, will be seen, not so much those in the wiki.

How many Threads the system has? && If you force unbound to use less threads & less cache can you reduce that memory consumption?

See my post in here my unbound (with 12 RPZ lists) consumed 3,3-3,4 GB: I created a test version of a RPZ add-on and I am looking for feedback - Network / DNS - IPFire Community

I could of course try, but I am very reluctant to mess with the base system so you would have to elaborate on how to do that and I can review its impact on my system and get back with results, if I decide to do it.

You mean this?:

> /etc/unbound/local.d/memory-fix.conf 
> Manual override to limit Unbound memory consumption and threading overhead in Core 200.
> This file prevents hardware-based auto-tuning from inflating RAM usage when using large RPZ lists.
> server:
>     num-threads: 1
>     msg-cache-slabs: 1
>     rrset-cache-slabs: 1
>     infra-cache-slabs: 1
>     key-cache-slabs: 1
>     # Optional: Reduce cache sizes if 1 thread is still too high
>     rrset-cache-size: 128m
>     msg-cache-size: 64m

This behaviour has another side effect.
Because of the high memory consumption of unbound, other processes request swap space. At a first glance this are squid, hostapd, qos, …
I don’t think this is a good system state ( I fear this swap usage isn’t reduced ).

I can confirm,

I’m using a Raspberry Pi 4 with 2GB of RAM as a backup at night to keep my IoT devices connected.
I tried enabling three DNS firewall lists (Malware, Phishing, Piracy).
It worked for three days, but after restarting last night, I had a memory crash and lost unbound connections, so no internet, and therefore no IoT devices this morning.

May 12 21:41:02 ipfire2 kernel: basename invoked oom-killer: gfp_mask=0x140dca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO|__GFP_COMP), order=0, oom_score_adj=0
May 12 21:41:02 ipfire2 kernel: CPU: 2 UID: 0 PID: 17623 Comm: basename Tainted: G         C          6.18.7-ipfire #1 PREEMPT(voluntary) 
May 12 21:41:02 ipfire2 kernel: Tainted: [C]=CRAP
May 12 21:41:02 ipfire2 kernel: Hardware name: Raspberry Pi 4 Model B Rev 1.2 (DT)
May 12 21:41:02 ipfire2 kernel: Call trace:
May 12 21:41:02 ipfire2 kernel:  show_stack+0x34/0xd8 (C)
May 12 21:41:02 ipfire2 kernel:  dump_stack_lvl+0x64/0xa0
May 12 21:41:02 ipfire2 kernel:  dump_stack+0x1c/0x38
May 12 21:41:02 ipfire2 kernel:  dump_header+0x4c/0x160
May 12 21:41:02 ipfire2 kernel:  oom_kill_process+0x2e0/0x368
May 12 21:41:02 ipfire2 kernel:  out_of_memory+0x248/0x5a0
May 12 21:41:02 ipfire2 kernel:  __alloc_frozen_pages_noprof+0xb20/0xf40
May 12 21:41:02 ipfire2 kernel:  alloc_pages_mpol+0x60/0x168
May 12 21:41:02 ipfire2 kernel:  vma_alloc_folio_noprof+0x8c/0x110
May 12 21:41:02 ipfire2 kernel:  vma_alloc_zeroed_movable_folio+0x40/0x70
May 12 21:41:02 ipfire2 kernel:  __handle_mm_fault+0x810/0x1108
May 12 21:41:02 ipfire2 kernel:  handle_mm_fault+0x9c/0x2d8
May 12 21:41:02 ipfire2 kernel:  do_page_fault+0x104/0x788
May 12 21:41:02 ipfire2 kernel:  do_translation_fault+0x78/0xb8
May 12 21:41:02 ipfire2 kernel:  do_mem_abort+0x4c/0xd0
May 12 21:41:02 ipfire2 kernel:  el0_da+0x30/0xb8
May 12 21:41:02 ipfire2 kernel:  el0t_64_sync_handler+0xf0/0x110
May 12 21:41:02 ipfire2 kernel:  el0t_64_sync+0x1a4/0x1a8
May 12 21:41:02 ipfire2 kernel: Mem-Info:
May 12 21:41:02 ipfire2 kernel: active_anon:246 inactive_anon:445309 isolated_anon:0\x0a active_file:1794 inactive_file:194 isolated_file:30\x0a unevictable:0 dirty:0 writeback:0\x0a slab_reclaimable:1857 slab_unreclaimable:21315\x0a mapped:1563 shmem:5760 pagetables:1705\x0a sec_pagetables:0 bounce:0\x0a kernel_misc_reclaimable:0\x0a free:3658 free_pcp:25 free_cma:48
May 12 21:41:02 ipfire2 kernel: Node 0 active_anon:984kB inactive_anon:1781236kB active_file:7176kB inactive_file:776kB unevictable:0kB isolated(anon):0kB isolated(file):120kB mapped:6252kB dirty:0kB writeback:0kB shmem:23040kB kernel_stack:3120kB shadow_call_stack:792kB pagetables:6820kB sec_pagetables:0kB all_unreclaimable? no Balloon:0kB
May 12 21:41:02 ipfire2 kernel: Node 0 DMA free:9336kB boost:8836kB min:12768kB low:13748kB high:14728kB reserved_highatomic:0KB free_highatomic:0KB active_anon:148kB inactive_anon:859088kB active_file:1512kB inactive_file:744kB unevictable:0kB writepending:0kB zspages:0kB present:1021952kB managed:927756kB mlocked:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:192kB
May 12 21:41:02 ipfire2 kernel: lowmem_reserve[]: 0 980 980 980
May 12 21:41:02 ipfire2 kernel: Node 0 DMA32 free:5296kB boost:8192kB min:12448kB low:13512kB high:14576kB reserved_highatomic:0KB free_highatomic:0KB active_anon:836kB inactive_anon:922148kB active_file:5664kB inactive_file:32kB unevictable:0kB writepending:0kB zspages:0kB present:1048576kB managed:1004248kB mlocked:0kB bounce:0kB free_pcp:104kB local_pcp:0kB free_cma:0kB
May 12 21:41:02 ipfire2 kernel: lowmem_reserve[]: 0 0 0 0
May 12 21:41:02 ipfire2 kernel: Node 0 DMA: 190*4kB (UMC) 163*8kB (UMEC) 114*16kB (UMEC) 34*32kB (UC) 7*64kB (UE) 5*128kB (UE) 1*256kB (E) 2*512kB (UE) 2*1024kB (UE) 0*2048kB 0*4096kB = 9392kB
May 12 21:41:02 ipfire2 kernel: Node 0 DMA32: 109*4kB (UM) 49*8kB (UE) 31*16kB (U) 47*32kB (UE) 16*64kB (UE) 8*128kB (UE) 2*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 5388kB
May 12 21:41:02 ipfire2 kernel: 7811 total pagecache pages
May 12 21:41:02 ipfire2 kernel: 0 pages in swap cache
May 12 21:41:02 ipfire2 kernel: Free swap  = 0kB
May 12 21:41:02 ipfire2 kernel: Total swap = 0kB
May 12 21:41:02 ipfire2 kernel: 517632 pages RAM
May 12 21:41:02 ipfire2 kernel: 0 pages HighMem/MovableOnly
May 12 21:41:02 ipfire2 kernel: 34631 pages reserved
May 12 21:41:02 ipfire2 kernel: 16384 pages cma reserved
May 12 21:41:02 ipfire2 kernel: Tasks state (memory values in pages):
May 12 21:41:02 ipfire2 kernel: [  pid  ]   uid  tgid total_vm      rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
May 12 21:41:02 ipfire2 kernel: [    535]     0   535     8116      594      474      120         0    65536        0             0 udevd
May 12 21:41:02 ipfire2 kernel: [  11471]     0 11471     1360      329      220      109         0    45056        0             0 vnstatd
May 12 21:41:02 ipfire2 kernel: [  11482]     0 11482      839       54        0       54         0    40960        0             0 syslogd
May 12 21:41:02 ipfire2 kernel: [  11535]    99 11535   483405   420362   420268       94         0  3469312        0             0 unbound
May 12 21:41:02 ipfire2 kernel: [  11546]     0 11546      741       75        0       75         0    40960        0             0 acpid
May 12 21:41:02 ipfire2 kernel: [  11565]   999 11565     1131       79       32       47         0    45056        0             0 dbus-daemon
May 12 21:41:02 ipfire2 kernel: [  11580]     0 11580     3383      210      177       33         0    61440        0             0 hostapd
May 12 21:41:02 ipfire2 kernel: [  16998]    23 16998     5242      550      500       50         0    73728        0             0 squid
May 12 21:41:02 ipfire2 kernel: [  17001]    23 17001     6964     2385     2112      273         0    90112        0             0 squid
May 12 21:41:02 ipfire2 kernel: [  17006]    23 17006    22143     2099     2080       19         0   126976        0             0 python3
May 12 21:41:02 ipfire2 kernel: [  17008]    23 17008    21936     2165     2112       53         0   131072        0             0 python3
May 12 21:41:02 ipfire2 kernel: [  17112]     0 17112   153585     1293     1290        3         0   163840        0             0 collectd
May 12 21:41:02 ipfire2 kernel: [  17220]     0 17220     1244      101       32       69         0    45056        0             0 saslauthd
May 12 21:41:02 ipfire2 kernel: [  17221]     0 17221     1244       78       33       45         0    45056        0             0 saslauthd
May 12 21:41:02 ipfire2 kernel: [  17303]     0 17303    19900      437      219      218         0    61440        0             0 ntpd
May 12 21:41:02 ipfire2 kernel: [  17348]     0 17348     2370      960      960        0         0    49152        0             0 dhcpd
May 12 21:41:02 ipfire2 kernel: [  17357]     0 17357    41695     2272     2177       95         0    90112        0             0 unbound-dhcp-le
May 12 21:41:02 ipfire2 kernel: [  17370]     0 17370     2749      354      290       64         0    53248        0           -16 sshd
May 12 21:41:02 ipfire2 kernel: [  17384]     0 17384     4724      677      519       62        96    73728        0             0 httpd
May 12 21:41:02 ipfire2 kernel: [  17385]    99 17385     4513      598      521       77         0    65536        0             0 httpd
May 12 21:41:02 ipfire2 kernel: [  17387]    99 17387   336837      573      553       20         0   237568        0             0 httpd
May 12 21:41:02 ipfire2 kernel: [  17437]     0 17437     4259      272      207       65         0    73728        0             0 keepalived
May 12 21:41:02 ipfire2 kernel: [  17438]     0 17438     6307      338      271       67         0    77824        0             0 keepalived
May 12 21:41:02 ipfire2 kernel: [  17449]     0 17449     1060      137       32      105         0    40960        0             0 fcron
May 12 21:41:02 ipfire2 kernel: [  17502]     0 17502     1249      115       32       83         0    49152        0             0 agetty
May 12 21:41:02 ipfire2 kernel: [  17503]     0 17503      961       88       32       56         0    40960        0             0 agetty
May 12 21:41:02 ipfire2 kernel: [  17504]     0 17504     1121       53        0       53         0    45056        0             0 agetty
May 12 21:41:02 ipfire2 kernel: [  17505]     0 17505      855       33        0       33         0    40960        0             0 agetty
May 12 21:41:02 ipfire2 kernel: [  17506]     0 17506     1217      104       32       72         0    45056        0             0 agetty
May 12 21:41:02 ipfire2 kernel: [  17507]     0 17507      876       93       32       61         0    45056        0             0 agetty
May 12 21:41:02 ipfire2 kernel: [  17616]     0 17616     1060       56       36       20         0    40960        0             0 fcron
May 12 21:41:02 ipfire2 kernel: [  17617]     0 17617     1642      100       64       36         0    40960        0             0 bash
May 12 21:41:02 ipfire2 kernel: [  17618]     0 17618     8588     5505     4640      865         0   102400        0             0 update-ipblockl
May 12 21:41:02 ipfire2 kernel: [  17619]     0 17619     1060       55       36       19         0    40960        0             0 fcron
May 12 21:41:02 ipfire2 kernel: [  17620]     0 17620     1619      223       32      191         0    45056        0             0 run-parts
May 12 21:41:02 ipfire2 kernel: [  17623]     0 17623      776      231        0      231         0    40960        0             0 basename
May 12 21:41:02 ipfire2 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=unbound,pid=11535,uid=99
May 12 21:41:02 ipfire2 kernel: Out of memory: Killed process 11535 (unbound) total-vm:1933620kB, anon-rss:1681072kB, file-rss:376kB, shmem-rss:0kB, UID:99 pgtables:3388kB oom_score_adj:0

Edit : It’s as if Unbound consumes more memory with each restart

Every night around midnight unbound grabs another 50-100 MB from memory.
I did a test: I’ve reloaded unbound 5 times: every time the RSS memory increased although the rpz lists stayed the same (1,8 millions entries, ~69MB)

Apparently this behavior was seen in previous versions as well: Memory leak on rpz zone reloads when unbound is listening on an external interface Β· Issue #318 Β· NLnetLabs/unbound

Copied from there: (without any interfaces configured) unbound is listening on 0.0.0.0:53. same thing happens: unbound’s memory allocation increases substantially with each rpz zone reload.

Those people exhausted 32GB RAM by loading 100MB rpz zones from their own server.
Therefore I am happy to report that current version of unbound stays below 3,5 GB RAM for 69 MB rpz zone files.

Yup, leak is growing.


(The image is from my Grafana monitoring that keeps track of all my systems)

Even on a 16GB system this is not something I want to see.

And from last hour after restarting the unbound service:

[root@surtveggur ~]# /etc/init.d/unbound restart
Stopping Unbound DNS Proxy...  [  OK  ]
Starting Unbound DNS Proxy...  [  OK  ]

[root@surtveggur ~]# free -m
               total        used        free      shared  buff/cache   available
Mem:           15821        1675       12557           4        1752       14146
Swap:           1023           0        1023

[root@surtveggur ~]# for i in $(seq 1 30); do echo "$(date) $(free -m | awk '/^Mem/{...}')"; sleep 60; done
09:29:28 Used:1659 Free:12573 Cache:1752
09:30:28 Used:1676 Free:12556 Cache:1752
09:31:28 Used:1674 Free:12558 Cache:1753
09:32:28 Used:1673 Free:12558 Cache:1753

I am actually going to put a cron restart on that service until there is a fix for this leakage. I do not have to, but I do not want any uncontrolled resource hogging if I can avoid it.

I have tested unbound memory consumption after each reload: CU200, single rpz list loaded (malware), single client in the network (reduce numbers of dns queries). Test machine is PCEngines APU2D4, red-green config, single LAN client connected to green.

Default config test: how much RAM is consumed after restart (not reload, because reload does not release the memory)

/etc/init.d/unbound stop; sleep 5; /etc/init.d/unbound start; sleep 10; ~/check_unbound.sh
Stopping Unbound DNS Proxy…                                                                                                                                                                                                        [  OK  ]
Starting Unbound DNS Proxy…                                                                                                                                                                                                        [  OK  ]--- RSS MEMORY ---
  PID   RSS COMMAND
 4442 51592 /usr/sbin/unbound

--- SLABS & THREADS ---
    msg-cache-slabs   : 4
    rrset-cache-slabs : 4
    infra-cache-slabs : 4
    key-cache-slabs   : 4
    num-threads   : 4
    infra-cache-numhosts   : 10000
    interface-automatic   : yes
    interface   : 0.0.0.0

--- INTERNAL CACHE USAGE ---
mem.cache.rrset : 75722
mem.cache.message : 69419
mem.mod.validator : 70838
[root@silver-x86-64 ~]# ~/check_unbound.sh 
--- RSS MEMORY ---
  PID   RSS COMMAND
 4442 51616 /usr/sbin/unbound

--- SLABS & THREADS ---
    msg-cache-slabs   : 4
    rrset-cache-slabs : 4
    infra-cache-slabs : 4
    key-cache-slabs   : 4
    num-threads   : 4
    infra-cache-numhosts   : 10000
    interface-automatic   : yes
    interface   : 0.0.0.0

--- INTERNAL CACHE USAGE ---
mem.cache.rrset : 80349
mem.cache.message : 72656
mem.mod.validator : 71959



Then 10 times reload followed by report of memory consumed and cache:

for i in {1..10}; do echo β€œβ€” Iteration: ${i}” β€”; unbound-control reload && sleep 5 && ~/check_unbound.sh; done
β€” Iteration: 1 β€”
ok
β€” RSS MEMORY β€”
PID   RSS COMMAND
4442 51656 /usr/sbin/unbound

β€” SLABS & THREADS β€”
msg-cache-slabs   : 4
rrset-cache-slabs : 4
infra-cache-slabs : 4
key-cache-slabs   : 4
num-threads   : 4
infra-cache-numhosts   : 10000
interface-automatic   : yes
interface   : 0.0.0.0

β€” INTERNAL CACHE USAGE β€”
mem.cache.rrset : 73006
mem.cache.message : 68294
mem.mod.validator : 70104
β€” Iteration: 2 β€”
ok
β€” RSS MEMORY β€”
PID   RSS COMMAND
4442 51772 /usr/sbin/unbound

β€” SLABS & THREADS β€”
msg-cache-slabs   : 4
rrset-cache-slabs : 4
infra-cache-slabs : 4
key-cache-slabs   : 4
num-threads   : 4
infra-cache-numhosts   : 10000
interface-automatic   : yes
interface   : 0.0.0.0

β€” INTERNAL CACHE USAGE β€”
mem.cache.rrset : 73006
mem.cache.message : 68294
mem.mod.validator : 70104
β€” Iteration: 3 β€”
ok
β€” RSS MEMORY β€”
PID   RSS COMMAND
4442 52032 /usr/sbin/unbound

β€” SLABS & THREADS β€”
msg-cache-slabs   : 4
rrset-cache-slabs : 4
infra-cache-slabs : 4
key-cache-slabs   : 4
num-threads   : 4
infra-cache-numhosts   : 10000
interface-automatic   : yes
interface   : 0.0.0.0

β€” INTERNAL CACHE USAGE β€”
mem.cache.rrset : 78568
mem.cache.message : 71144
mem.mod.validator : 71578
β€” Iteration: 4 β€”
ok
β€” RSS MEMORY β€”
PID   RSS COMMAND
4442 52048 /usr/sbin/unbound

β€” SLABS & THREADS β€”
msg-cache-slabs   : 4
rrset-cache-slabs : 4
infra-cache-slabs : 4
key-cache-slabs   : 4
num-threads   : 4
infra-cache-numhosts   : 10000
interface-automatic   : yes
interface   : 0.0.0.0

β€” INTERNAL CACHE USAGE β€”
mem.cache.rrset : 73006
mem.cache.message : 68294
mem.mod.validator : 70104
β€” Iteration: 5 β€”
ok
β€” RSS MEMORY β€”
PID   RSS COMMAND
4442 52072 /usr/sbin/unbound

β€” SLABS & THREADS β€”
msg-cache-slabs   : 4
rrset-cache-slabs : 4
infra-cache-slabs : 4
key-cache-slabs   : 4
num-threads   : 4
infra-cache-numhosts   : 10000
interface-automatic   : yes
interface   : 0.0.0.0

β€” INTERNAL CACHE USAGE β€”
mem.cache.rrset : 73006
mem.cache.message : 68294
mem.mod.validator : 70104
β€” Iteration: 6 β€”
ok
β€” RSS MEMORY β€”
PID   RSS COMMAND
4442 52100 /usr/sbin/unbound

β€” SLABS & THREADS β€”
msg-cache-slabs   : 4
rrset-cache-slabs : 4
infra-cache-slabs : 4
key-cache-slabs   : 4
num-threads   : 4
infra-cache-numhosts   : 10000
interface-automatic   : yes
interface   : 0.0.0.0

β€” INTERNAL CACHE USAGE β€”
mem.cache.rrset : 73006
mem.cache.message : 68294
mem.mod.validator : 70104
β€” Iteration: 7 β€”
ok
β€” RSS MEMORY β€”
PID   RSS COMMAND
4442 52100 /usr/sbin/unbound

β€” SLABS & THREADS β€”
msg-cache-slabs   : 4
rrset-cache-slabs : 4
infra-cache-slabs : 4
key-cache-slabs   : 4
num-threads   : 4
infra-cache-numhosts   : 10000
interface-automatic   : yes
interface   : 0.0.0.0

β€” INTERNAL CACHE USAGE β€”
mem.cache.rrset : 73006
mem.cache.message : 68294
mem.mod.validator : 70104
β€” Iteration: 8 β€”
ok
β€” RSS MEMORY β€”
PID   RSS COMMAND
4442 52112 /usr/sbin/unbound

β€” SLABS & THREADS β€”
msg-cache-slabs   : 4
rrset-cache-slabs : 4
infra-cache-slabs : 4
key-cache-slabs   : 4
num-threads   : 4
infra-cache-numhosts   : 10000
interface-automatic   : yes
interface   : 0.0.0.0

β€” INTERNAL CACHE USAGE β€”
mem.cache.rrset : 73006
mem.cache.message : 68294
mem.mod.validator : 70104
β€” Iteration: 9 β€”
ok
β€” RSS MEMORY β€”
PID   RSS COMMAND
4442 52112 /usr/sbin/unbound

β€” SLABS & THREADS β€”
msg-cache-slabs   : 4
rrset-cache-slabs : 4
infra-cache-slabs : 4
key-cache-slabs   : 4
num-threads   : 4
infra-cache-numhosts   : 10000
interface-automatic   : yes
interface   : 0.0.0.0

β€” INTERNAL CACHE USAGE β€”
mem.cache.rrset : 73006
mem.cache.message : 68294
mem.mod.validator : 70104
β€” Iteration: 10 β€”
ok
β€” RSS MEMORY β€”
PID   RSS COMMAND
4442 52124 /usr/sbin/unbound

β€” SLABS & THREADS β€”
msg-cache-slabs   : 4
rrset-cache-slabs : 4
infra-cache-slabs : 4
key-cache-slabs   : 4
num-threads   : 4
infra-cache-numhosts   : 10000
interface-automatic   : yes
interface   : 0.0.0.0

β€” INTERNAL CACHE USAGE β€”
mem.cache.rrset : 73006
mem.cache.message : 68294
mem.mod.validator : 70104

So in default config the system had 1 MB gained after 10 reloads for 51 MB initial memory consumption

Test 2: custom config added in /etc/unbound/local.d/

cat memory-fix.conf

server:

num-threads: 1
msg-cache-slabs: 1
rrset-cache-slabs: 1
infra-cache-slabs: 1
key-cache-slabs: 1

Memory consumed after restart with the custom config


/etc/init.d/unbound stop; sleep 5; /etc/init.d/unbound start; sleep 10; ~/check_unbound.sh
Stopping Unbound DNS Proxy…                                                                                                                                                                                                        [  OK  ]
Starting Unbound DNS Proxy…                                                                                                                                                                                                        [  OK  ]
β€” RSS MEMORY β€”
PID   RSS COMMAND
5150 17264 /usr/sbin/unbound

β€” SLABS & THREADS β€”
msg-cache-slabs   : 1
rrset-cache-slabs : 1
infra-cache-slabs : 1
key-cache-slabs   : 1
num-threads   : 1
infra-cache-numhosts   : 10000
interface-automatic   : yes
interface   : 0.0.0.0

β€” INTERNAL CACHE USAGE β€”
mem.cache.rrset : 23446
mem.cache.message : 18734
mem.mod.validator : 20544

So the initial memory dropped from 51 MB to 17 MB!

Now the reload iterations:


for i in {1..10}; do echo β€œβ€” Iteration: ${i}” β€”; unbound-control reload && sleep 5 && ~/check_unbound.sh; done
β€” Iteration: 1 β€”
ok
β€” RSS MEMORY β€”
PID   RSS COMMAND
5150 19660 /usr/sbin/unbound

β€” SLABS & THREADS β€”
msg-cache-slabs   : 1
rrset-cache-slabs : 1
infra-cache-slabs : 1
key-cache-slabs   : 1
num-threads   : 1
infra-cache-numhosts   : 10000
interface-automatic   : yes
interface   : 0.0.0.0

β€” INTERNAL CACHE USAGE β€”
mem.cache.rrset : 23446
mem.cache.message : 18734
mem.mod.validator : 20544
β€” Iteration: 2 β€”
ok
β€” RSS MEMORY β€”
PID   RSS COMMAND
5150 19940 /usr/sbin/unbound

β€” SLABS & THREADS β€”
msg-cache-slabs   : 1
rrset-cache-slabs : 1
infra-cache-slabs : 1
key-cache-slabs   : 1
num-threads   : 1
infra-cache-numhosts   : 10000
interface-automatic   : yes
interface   : 0.0.0.0

β€” INTERNAL CACHE USAGE β€”
mem.cache.rrset : 23446
mem.cache.message : 18734
mem.mod.validator : 20544
β€” Iteration: 3 β€”
ok
β€” RSS MEMORY β€”
PID   RSS COMMAND
5150 19948 /usr/sbin/unbound

β€” SLABS & THREADS β€”
msg-cache-slabs   : 1
rrset-cache-slabs : 1
infra-cache-slabs : 1
key-cache-slabs   : 1
num-threads   : 1
infra-cache-numhosts   : 10000
interface-automatic   : yes
interface   : 0.0.0.0

β€” INTERNAL CACHE USAGE β€”
mem.cache.rrset : 23446
mem.cache.message : 18734
mem.mod.validator : 20544
β€” Iteration: 4 β€”
ok
β€” RSS MEMORY β€”
PID   RSS COMMAND
5150 20076 /usr/sbin/unbound

β€” SLABS & THREADS β€”
msg-cache-slabs   : 1
rrset-cache-slabs : 1
infra-cache-slabs : 1
key-cache-slabs   : 1
num-threads   : 1
infra-cache-numhosts   : 10000
interface-automatic   : yes
interface   : 0.0.0.0

β€” INTERNAL CACHE USAGE β€”
mem.cache.rrset : 23446
mem.cache.message : 18734
mem.mod.validator : 20544
β€” Iteration: 5 β€”
ok
β€” RSS MEMORY β€”
PID   RSS COMMAND
5150 20576 /usr/sbin/unbound

β€” SLABS & THREADS β€”
msg-cache-slabs   : 1
rrset-cache-slabs : 1
infra-cache-slabs : 1
key-cache-slabs   : 1
num-threads   : 1
infra-cache-numhosts   : 10000
interface-automatic   : yes
interface   : 0.0.0.0

β€” INTERNAL CACHE USAGE β€”
mem.cache.rrset : 29008
mem.cache.message : 21584
mem.mod.validator : 22018
β€” Iteration: 6 β€”
ok
β€” RSS MEMORY β€”
PID   RSS COMMAND
5150 20576 /usr/sbin/unbound

β€” SLABS & THREADS β€”
msg-cache-slabs   : 1
rrset-cache-slabs : 1
infra-cache-slabs : 1
key-cache-slabs   : 1
num-threads   : 1
infra-cache-numhosts   : 10000
interface-automatic   : yes
interface   : 0.0.0.0

β€” INTERNAL CACHE USAGE β€”
mem.cache.rrset : 23446
mem.cache.message : 18734
mem.mod.validator : 20544
β€” Iteration: 7 β€”
ok
β€” RSS MEMORY β€”
PID   RSS COMMAND
5150 20580 /usr/sbin/unbound

β€” SLABS & THREADS β€”
msg-cache-slabs   : 1
rrset-cache-slabs : 1
infra-cache-slabs : 1
key-cache-slabs   : 1
num-threads   : 1
infra-cache-numhosts   : 10000
interface-automatic   : yes
interface   : 0.0.0.0

β€” INTERNAL CACHE USAGE β€”
mem.cache.rrset : 23446
mem.cache.message : 18734
mem.mod.validator : 20544
β€” Iteration: 8 β€”
ok
β€” RSS MEMORY β€”
PID   RSS COMMAND
5150 20580 /usr/sbin/unbound

β€” SLABS & THREADS β€”
msg-cache-slabs   : 1
rrset-cache-slabs : 1
infra-cache-slabs : 1
key-cache-slabs   : 1
num-threads   : 1
infra-cache-numhosts   : 10000
interface-automatic   : yes
interface   : 0.0.0.0

β€” INTERNAL CACHE USAGE β€”
mem.cache.rrset : 23446
mem.cache.message : 18734
mem.mod.validator : 20544
β€” Iteration: 9 β€”
ok
β€” RSS MEMORY β€”
PID   RSS COMMAND
5150 20588 /usr/sbin/unbound

β€” SLABS & THREADS β€”
msg-cache-slabs   : 1
rrset-cache-slabs : 1
infra-cache-slabs : 1
key-cache-slabs   : 1
num-threads   : 1
infra-cache-numhosts   : 10000
interface-automatic   : yes
interface   : 0.0.0.0

β€” INTERNAL CACHE USAGE β€”
mem.cache.rrset : 26162
mem.cache.message : 19859
mem.mod.validator : 21278
β€” Iteration: 10 β€”
ok
β€” RSS MEMORY β€”
PID   RSS COMMAND
5150 20588 /usr/sbin/unbound

β€” SLABS & THREADS β€”
msg-cache-slabs   : 1
rrset-cache-slabs : 1
infra-cache-slabs : 1
key-cache-slabs   : 1
num-threads   : 1
infra-cache-numhosts   : 10000
interface-automatic   : yes
interface   : 0.0.0.0

β€” INTERNAL CACHE USAGE β€”
mem.cache.rrset : 23446
mem.cache.message : 18734
mem.mod.validator : 20544

Unbound ended up with 20,5 MB memory consumed after 10 reload iterations from an initial 17 MB amount.
First reload added the biggest memory consumption

From above test I see that num-threads and *-cache-slabs play a role only to the initial memory allocation but does not stop the memory leakage. Percentage wise, the custom config seems worse: the memory increase was >15% while default config added ~0,5% memory increase (same numbers of reloads, same rpz list loaded).

The internal cache stays pretty much the same in my test - but the test machine had only one Linux client connected (and in Linux was started just ssh to ipfire and Firefox toward this webpage)

Note: custom config with num-threads and *-cache-slabs set to 1 reduces the overall response capacity of the dns. It is not recommended for systems that need to resolve a large amount of queries. Find your suitable config based on your network demands.

Hello everyone,

as you know from the wiki page, we are aware that Unbound is using an amount of memory that is simply unwarranted for the functionality. Loading an RPZ into memory will definitely consume more space than the plain text file on disk, but even an initial load will consume about 10x of the data. It does not even need today’s RAM prices to make that unnecessarily expensive. That Unbound has a bug open for six years now not fixing this is a problem, and the list of those is very long.

We have plans to entirely replace Unbound for that and other reasons in IPFire. It has been given us a lot of reason for frustration and it simply isn’t keeping up with what we need.

A network with bad DNS is a bad network and therefore we will now change this.

Since the DNS resolver is deeply integrated into IPFire touching with the DHCP server, supports a lot of things like the new DNS Firewall, TLS for upstream servers, Safe Search, static hosts, qname minimisation, this will be a long road and I am hoping for some support.

In the meantime, @hjkl, would reverting this commit ease the pain? git.ipfire.org Git - ipfire-2.x.git/commitdiff

For my case it makes no difference: I see that a custom config files allows me to change threads and slabs according to my needs (w/p touch main unbound.conf as it might bet overwritten by future updates and my custom config gets lost)

But what I can’t change the interface:0.0.0.0 part - unless I remove it/changed in unbound.conf
I want to test the case where unbound listen only to a specific list of interfaces (like green, blue). The 2020 post from git unbound issues indicate that interface: 0.0.0.0 ( and interface-automatic: yes) do play a role in the memory leak (?! - I need to test it with current version; that post is 6 years old)

Therefore, for me would be helpful if these 2 parameters can be changed by a custom config file, which, at the moment, it seems not being a solution (at least for interface: it simply add additional interfaces I’ve put in the custom conf file, while 0.0.0.0 stays in the interface list)

Late edit: I understand and support the config with interface: 0.0.0.0 - it covers all possible configs a user can run: w/ or w/o WG, OpenVPN, green, blue, etc… I am not challenging the config! I simply want to test if unbound listening to one interface only does keep its memory in control.

After experimenting with some parameters, using the testing version CU202, I know definitely that the number of processes doesn’t change the bug. Neither the so-reuseports setting has any effect.

BTW my Virtualbox testsystem shows the effect drastically. I use 2GB of RAM only ( this is okay with no users connected ). With 3 RPZ zones activated, including the big phishing file, less than 50% of memory is used. But each reload increases memory usage, so it is easy to drive the system into swap.