IPFire went down last night, can't find cause

cwensink · 7 November 2022 20:29

Well, the problem continues, same exact circumstances. I had a reboot of IPFire unexpectedly today (when I was out of the building at lunch), and from the logs I see that it restarted at 5:25 PM last night as well. I have plugged the power supply of the hardware appliance directly into a Smart APC 2200 power supply, I have swapped the hardware appliance out, I have swapped out the power supply itself with another duplicate model. IPFire 171 and 170 units both had the same problems. On the backup unit at my desk I saw decreased hard drive performance compared to the live up and working unit, but it was still online.

Memory Graph:

System:

/var/log/messages shows normal DROP_INPUT traffic up until the line of failure which just shows ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ @Nov 7 13:10:05 ipfire syslogd 1.5.1: restart (remote reception) .

Could this be a bad batch of power supplies, with capacitors going out all at the same time? Bad batch of motherboards? Bad Ram? Bug in IPFire that affects all hardware no matter what it runs on?

Sunday night the system seems to have just rebooted then came back up on it’s own, today I was out of the office when it happened and my boss pulled the plug to reset it.

Thoughts?

cwensink · 7 November 2022 21:25

One other strange thing that I noticed. I wondered if openvpn taking up a lot of resources has anything to do with this at all. I noticed that the openvpn authentication services was taking up a lot of CPU power. I stopped the Openvpn service from the GUI, yet in top in bash I could see that the openvpn authenticator service was still running and taking up a lot of CPU. It seems like the service stopping in the GUI did not kill the service. I killed the service in the shell, then the CPU usage went back down to normal.

Here’s the openvpn server.conf
[root@ipfire ovpn]# cat server.conf
#OpenVPN Server conf

daemon openvpnserver
writepid /var/run/openvpn.pid
#DAN prepare OpenVPN for listening on blue and orange
;local
dev tun
proto udp
port 1194
script-security 3
ifconfig-pool-persist /var/ipfire/ovpn/ovpn-leases.db 3600
client-config-dir /var/ipfire/ovpn/ccd
tls-server
ca /var/ipfire/ovpn/ca/cacert.pem
cert /var/ipfire/ovpn/certs/servercert.pem
key /var/ipfire/ovpn/certs/serverkey.pem
dh /var/ipfire/ovpn/ca/dh1024.pem
server 255.255.255.0
tun-mtu 1400
mssfix
keepalive 10 60
status-version 1
status /var/run/ovpnserver.log 30
ncp-disable
cipher AES-256-CBC
auth SHA512
tls-version-min 1.2
push “dhcp-option DNS ”
push “dhcp-option WINS ”
max-clients 100
tls-verify /usr/lib/openvpn/verify
crl-verify /var/ipfire/ovpn/crls/cacrl.pem
auth-user-pass-optional
reneg-sec 86400
user nobody
group nobody
persist-key
persist-tun
verb 4

Log clients connecting/disconnecting

client-connect “/usr/sbin/openvpn-metrics client-connect”
client-disconnect “/usr/sbin/openvpn-metrics client-disconnect”

Enable Management Socket

management /var/run/openvpn.sock unix
management-client-auth

See any concerns there?

bbitsch · 8 November 2022 08:25

Maybe my questions are answered before, nevertheless I’ll pose them.

Did you try with direct power connection? If your power source is reliable, this test should show whether the UPSs are the source of the problem.
Did you check your HD? The ^@s suggest there is a problem in the file system. ^@ is the value 0, which can exist after the end of a file. But if it is listed as contents, the EOF seems to be incorrect.

Just two thoughts after reading your last posts.

cwensink · 8 November 2022 14:11

Are You referring to running fsck, or hdarm, or some other ipfire plugin?

bbitsch · 8 November 2022 14:17

Yes. Just to be sure.

cwensink · 8 November 2022 15:04

Regarding power I had the connection going directly to the wall, before when I was having this issue. Initially I thought that was the reason why I was having issues, so I moved it to a power strip on one battery backup. I had the issue again so I moved it to the other identical battery backup, plugging directly into it, not through a strip, and the issue is still happening.

Chris

cwensink · 9 November 2022 15:27

Last night I rebooted and checked the file system, everything came back clean without any errors. The memory graph shows the restart at 6 PM last night.

There is definitely something strange going on with the openvpn authenticator. This is what I observed this morning when coming into the office:

Huge memory spokes started around 6:00 AM, for no clear reasons.
I stopped the openvpn service in the WUI:

The Memory activity did not go down, it continued to climb at a steady pace. There are no active openvpn connections, nobody is attempting to connect in.

In the shell running top this is what I’m observing:

Even though in the WUI I stopped the openvpn service, on the back end after 5 minutes the openvpn authenticator service was still running and consuming a ton of resources.

I used kill to stop that thread, then resources went back to normal.

Has anyone experienced anything like this? Is this related to the bug report mentioned higher up in th is thread?

bonnietwin · 9 November 2022 17:00

It might be but I am not sure. That bug is related to where the used memory (ie the blue part of the memory graph) goes up to 100% because of the OpenVPN Authenticator and then the OOM Killer kicks in (Out Of Memory) and kills the OpenVPN Authenticator.

Your graph is showing the yellow portion of the memory increasing but this is cached memory where IPFire caches stuff from the Disk Drive to be able to access quicker if needed again but if the memory is needed for real usage then the yellow is freed up for the blue. It might be that if you left it running the OpenVPN Authenticator would continue to increase it’s memory consumption and eventually hit the 100% actual used level.

I don’t see the above at all. I don’t see any blue memory peaks and also the OpenVPN Authenticator is not using CPU time on my system.
From the bug it looks like this is occurring when the bug reporter created a new Host/Root certificate in OpenVPN while on CU170.

I plan to test this out with a vm test machine to see if it is the creation of a new Host/Root certificate that starts OpenVPN Authenticator using high amounts of CPU time and eventually memory but I won’t be able to do this for a day or so as I was away and am now busy with updates of python and consequently all python modules and programs using python for submission as a patch set for CU172.

I will report back here as well as into the bug report on if I am able to replicate the effect.

jon · 9 November 2022 17:01

My Memory charts looks like this. And all is A-OK.

So I am not sure your Memory charts are indicating a problem. The openvpn-auth does look “odd” but I don’t know why…

EDIT: This chart with the fast rising blue indicates a problem:

jon · 9 November 2022 17:27

Chris - In the image above in Post 84

258c1fd2fed8b41b5b537385658ce50bd9686abe_2_690x319

and in Post 88

the network seems to be busy between midnight and 7:30 AM.

I am curious:

how many devices are attached to your GREEN network?
what is going on at between 12 and 7:30 AM?

This is just my curiosity and me going off on an odd tangent!.

cwensink · 9 November 2022 17:46

Network is busy because backups are occurring, we have about 100 PC’s and laptops plus printers, barcode scanners, etc.

Chris

cwensink · 9 November 2022 17:47

Jon,

Every time you update to a new core version do you destroy and generate new certificates for openvpn and ipsec, then re-create VPN connections, or do you keep the same ones from version to the next?

jon · 9 November 2022 18:04

every time “no”. sometimes “yes”. There have been security related updates to OpenVPN or IPsec over the years that I want to take advantage of. So I will trash everything related to VPN and rebuild.

It is sometimes related to Diffie-Hellman parameter length.

Something like this:
https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=46803376d54f7e42d0eefe25155de82672bff41e

or in the near future, like this:
https://lists.ipfire.org/pipermail/development/2022-November/014601.html

bonnietwin · 11 November 2022 17:05

I have now cloned a vm running CU171, with a confirmed working OpenVPN system.
Running htop it showed openvpn-authenticator running at 0% cpu and 1.1% of the 1GB memory.

I then removed the x509 certificate set, which also removed all the client connection details.

I then created a new Root and Host certificate set using the same details as used previously. Then created a new client connection which was then transferred to my Android phone and I was able to successfully run a tunnel to a vm client in the vm testbed network just as previously done.

Running htop it showed openvpn-authenticator at 0% cpu and 1.0% of the 1GB memory.

I left the system running for 2 hours and nothing changed in the openvpn authenticator values of cpu and memory usage.

I then rebooted the IPFire vm and ran it again for 2 hours and again htop showed 0% cpu and 1.0% memory.

I will leave this running overnight but so far I have not been able to duplicate the effect that has been reported of openvpn-authenticator having a large cpu% and with memory climbing up till it uses all free memory, even with a fresh install of the Root/Host certificate set.

Will report back tomorrow if anything changes after running overnight.

bonnietwin · 11 November 2022 17:34

I got clarification from @schami23 that the openvpn-authenticator cpu and memory usage resulting in the oom killer only occurs when the OpenVPN server is stopped.

I turned off the OpenVPN Server and openvpn-authenticator went to something like 89% cpu% and memory spiked very quickly and triggered swap to be used to nearly 100% and then openvpn-authenticator was removed by the system - no longer in the htop table at all.

Looking in the kernel logs I have the oom killer message

18:17:20 kernel: Out of memory: Killed process 9525 (openvpn-authent) total-vm:831900kB, anon-rss :741952kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:1556kB oom_score_adj:0

18:17:20 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,glob al_oom,task_memcg=/,task=openvpn-authent,pid=9525,uid=0

So, yes I can duplicate the oom killer problem but it only seems to occur when the OpenVPN Server is turned off. It would appear that openvpn-authenticator doesn’t like to be running without the OpenVPN server turned on. Maybe we need to turn off the openvpn-authenticator when the OpenVPN server is stopped.

I will do the same test on a clone with the original Root/Host certificate to see if that has a different response or not.

I will also have a ook at the openvpn-authenticator code to see if I can spot anything obvious with regard to this effect.

Also after the openvpn-authenticator was stopped by the oom killer IPFire continued to run normally and cpu and memory % went back to normal. So this did not duplicate the whole IPFire shutdown or reboot that @cwensink is experiencing.

tphz · 11 November 2022 17:46

I can confirm this behavior
edit

After stopping the openvpn (WUI) server, I see high CPU load.
After restarting the openvpn setwer (WUI), the load does not decrease.

bonnietwin · 11 November 2022 17:49

I have also confirmed that the same thing happens on my original clone vm which has an old Root/Host certificate from many many CU’s ago.

So the problem looks not to be related to creating a new Root/Host certificate but just on openvpn-authenticator not liking to be running when OpenVPN Server is stopped.

Unfortunately the openvpn-authenticator is python code which I am not familiar with at all, so it will take a bit of work to look through it, but will give it a go.

bonnietwin · 11 November 2022 18:26

The problem looks to not be the openvpn-authenticator python code but the openvpnctrl c code.

There is a command stopAuthenticator which is part of the stopDaemon command but it is using /sbin/killall to stop the authenticator and in sbin there is only killall5. killall is in /bin so the /sbin/killall command will end up not doing anything because it can’t be found and so the openvpn-authenticator is not stopped.

I will edit the code and do a build and install of IPFire on my vm system to test this hypothesis out.

schami23 · 15 November 2022 14:26

Is there something new? Have you already found the problem?

Thanks
Jürgen

cwensink · 15 November 2022 15:08

I’m still having the problem. Our Production Machine rebooted on it’s own last night at 03:53:54 AM,

[root@ipfire ~]# uptime
 08:05:55 up  4:10,  1 user,  load average: 0.30, 0.44, 0.37
[root@ipfire ~]# top
top - 08:45:03 up  4:49,  1 user,  load average: 0.25, 0.33, 0.38
Tasks: 130 total,   1 running, 129 sleeping,   0 stopped,   0 zombie
%Cpu0  :   1.4/1.4     3[||                                                                                        ]   %Cpu1  :   0.7/3.5     4[||||                                                                                      ]
%Cpu2  :   0.7/15.0   16[||||||||||||||                                                                            ]   %Cpu3  :   8.7/10.7   19[||||||||||||||||||                                                                        ]
GiB Mem :  5.8/7.7      [                                                                                          ]   GiB Swap:  0.0/1.0      [                                                                                          ]

  PID USER      PR  NI    VIRT    RES  %CPU  %MEM     TIME+ S COMMAND
    1 root      20   0    2.4m   1.6m   0.0   0.0   0:03.52 S init [3]
  634 root      20   0    4.0m   0.1m   0.0   0.0   0:00.00 S  `- /usr/sbin/lvmetad
  653 root      20   0   18.8m   3.2m   0.0   0.0   0:00.35 S  `- udevd --daemon
  890 root      20   0    9.9m   0.4m   0.0   0.0   0:00.00 S  `- /usr/sbin/rngd -r /dev/hwrng --quiet
 3576 root      20   0    5.3m   3.6m   0.0   0.0   0:10.79 S  `- /usr/sbin/vnstatd -d --alwaysadd
 3587 root      20   0    9.5m   8.5m   0.0   0.1   0:08.50 S  `- klogd -c 1
 3594 root      20   0    2.4m   0.1m   0.7   0.0   0:11.65 S  `- syslogd -m 0 -r
 3743 nobody    20   0   23.9m  21.6m   0.0   0.3   0:48.25 S  `- /usr/sbin/unbound
 3754 root      20   0    2.5m   0.1m   0.0   0.0   0:00.00 S  `- /usr/sbin/acpid
 5212 suricata  20   0  481.8m  72.7m  20.5   0.9  27:10.45 S  `- /usr/bin/suricata -c /etc/suricata/suricata.yaml -D -q 0:3
 5402 root      20   0   28.3m  14.0m   0.0   0.2   0:03.28 S  `- /usr/bin/perl -w /usr/local/bin/qosd red0
 5405 root      20   0   28.3m  14.2m   0.0   0.2   0:03.19 S  `- /usr/bin/perl -w /usr/local/bin/qosd imq0
 6221 root      20   0   21.6m   6.3m   0.0   0.1   0:00.00 S  `- /usr/sbin/squid
 6224 squid     20   0   45.0m  34.6m   0.0   0.4   0:30.59 S      `- (squid-1) --kid squid-1
 6229 squid     20   0   15.9m  11.2m   0.0   0.1   0:01.36 S          `- /usr/bin/perl /usr/sbin/redirect_wrapper
 6256 squid     20   0    4.7m   3.5m   0.0   0.0   0:00.62 S              `- /usr/bin/squidGuard
 6258 squid     20   0    8.7m   6.4m   0.0   0.1   0:00.85 S              `- /usr/bin/perl /usr/sbin/updxlrator
 6230 squid     20   0   15.9m  11.3m   0.0   0.1   0:00.37 S          `- /usr/bin/perl /usr/sbin/redirect_wrapper
 6254 squid     20   0    4.5m   3.4m   0.0   0.0   0:00.03 S              `- /usr/bin/squidGuard
 6255 squid     20   0    8.7m   6.5m   0.0   0.1   0:00.08 S              `- /usr/bin/perl /usr/sbin/updxlrator
 6231 squid     20   0   15.9m  11.3m   0.0   0.1   0:00.29 S          `- /usr/bin/perl /usr/sbin/redirect_wrapper
 6257 squid     20   0    4.5m   2.0m   0.0   0.0   0:00.00 S              `- /usr/bin/squidGuard
 6259 squid     20   0    8.7m   6.5m   0.0   0.1   0:00.05 S              `- /usr/bin/perl /usr/sbin/updxlrator
 6233 squid     20   0   15.9m  11.5m   0.0   0.1   0:00.27 S          `- /usr/bin/perl /usr/sbin/redirect_wrapper
 6260 squid     20   0    4.5m   2.1m   0.0   0.0   0:00.00 S              `- /usr/bin/squidGuard
 6261 squid     20   0    8.7m   6.4m   0.0   0.1   0:00.05 S              `- /usr/bin/perl /usr/sbin/updxlrator
 6333 root      20   0    4.6m   1.7m   0.0   0.0   0:00.00 S  `- /usr/libexec/ipsec/starter --daemon charon
 6334 root      20   0 1165.9m  11.8m   0.0   0.2   1:08.80 S      `- /usr/libexec/ipsec/charon --use-syslog
 6372 nobody    20   0    7.4m   5.8m   0.0   0.1   0:05.43 S  `- /usr/sbin/openvpn --config /var/ipfire/ovpn/server.conf
 6435 root      20   0   15.7m  10.6m   0.0   0.1   0:00.04 S  `- /usr/bin/python3 /usr/sbin/openvpn-authenticator -d
 6550 root      20   0    2.8m   0.1m   0.0   0.0   0:00.00 S  `- /usr/sbin/saslauthd -n 2 -a pam
 6551 root      20   0    2.8m   0.1m   0.0   0.0   0:00.00 S      `- /usr/sbin/saslauthd -n 2 -a pam
 6636 root      20   0   72.4m   3.8m   0.0   0.0   0:02.45 S  `- /usr/bin/ntpd -Ap /var/run/ntpd.pid
 6689 root      20   0  232.6m   4.1m   0.0   0.1   0:31.34 S  `- /usr/sbin/collectd -C /etc/collectd.conf
 6717 root      20   0    9.1m   6.9m   0.0   0.1   0:01.28 S  `- /usr/sbin/dhcpd -q green0 blue0
 6727 root      20   0   17.6m  12.9m   0.0   0.2   0:37.47 S  `- /usr/bin/python3 /usr/sbin/unbound-dhcp-leases-bridge -d
 6738 root      20   0    6.5m   2.2m   0.0   0.0   0:00.00 S  `- sshd: /usr/sbin/sshd [listener] 0 of 5-5 startups
 1408 root      20   0    6.9m   5.5m   0.0   0.1   0:00.99 S      `- sshd: root@pts/0
 1424 root      20   0    7.6m   4.3m   0.0   0.1   0:00.02 S          `- -bash
 1777 root      20   0    8.3m   4.8m   1.3   0.1   0:21.14 R              `- top
 6752 root      20   0   11.2m   7.2m   0.0   0.1   0:01.86 S  `- /usr/sbin/httpd -k start
 6753 nobody    20   0   10.4m   3.7m   0.7   0.0   0:19.31 S      `- /usr/sbin/httpd -k start
 6756 nobody    20   0 1307.4m  13.4m   0.7   0.2   4:15.54 S      `- /usr/sbin/httpd -k start
 6802 root      20   0    3.5m   1.7m   0.0   0.0   0:00.41 S  `- /usr/sbin/fcron -y
 6805 root      20   0    2.9m   2.2m   0.0   0.0   0:00.02 S  `- /sbin/agetty console --noclear
 6806 root      20   0    2.9m   2.1m   0.0   0.0   0:00.00 S  `- /sbin/agetty tty2
 6807 root      20   0    2.9m   2.2m   0.0   0.0   0:00.00 S  `- /sbin/agetty tty3
 6808 root      20   0    2.9m   2.2m   0.0   0.0   0:00.00 S  `- /sbin/agetty tty4
 6809 root      20   0    2.9m   2.1m   0.0   0.0   0:00.00 S  `- /sbin/agetty tty5
 6810 root      20   0    2.9m   2.2m   0.0   0.0   0:00.00 S  `- /sbin/agetty tty6
    2 root      20   0    0.0m   0.0m   0.0   0.0   0:00.01 S [kthreadd]
    3 root       0 -20    0.0m   0.0m   0.0   0.0   0:00.00 I  `- [rcu_gp]
    4 root       0 -20    0.0m   0.0m   0.0   0.0   0:00.00 I  `- [rcu_par_gp]
    5 root       0 -20    0.0m   0.0m   0.0   0.0   0:00.00 I  `- [slub_flushwq]
    6 root       0 -20    0.0m   0.0m   0.0   0.0   0:00.00 I  `- [netns]
    8 root       0 -20    0.0m   0.0m   0.0   0.0   0:00.00 I  `- [kworker/0:0H-events_highpri]
[root@ipfire ~]# cd /var/log
[root@ipfire log]# ls
10-19-22-1500.txt  bootlog.1.gz  bootlog.8.gz    counter           dhcpcd.log.1.gz  dhcpcd.log.8.gz  mail.10.gz  mail.3.gz  messages               messages.14.gz  messages.8.gz     sa             wtmp
bootlog            bootlog.2.gz  bootlog.9.gz    dhcpcd.log        dhcpcd.log.2.gz  dhcpcd.log.9.gz  mail.11.gz  mail.4.gz  messages.1             messages.2.gz   messages.9.gz     setup.log      wtmp.1.gz
bootlog.10.gz      bootlog.3.gz  bootlog.old     dhcpcd.log.10.gz  dhcpcd.log.3.gz  haproxy          mail.12.gz  mail.5.gz  messages.10.24.22.txt  messages.3.gz   messags.10.27.22  squid
bootlog.11.gz      bootlog.4.gz  btmp            dhcpcd.log.11.gz  dhcpcd.log.4.gz  httpd            mail.13.gz  mail.6.gz  messages.10.gz         messages.4.gz   openvpn           squidGuard
bootlog.12.gz      bootlog.5.gz  cache           dhcpcd.log.12.gz  dhcpcd.log.5.gz  lastlog          mail.14.gz  mail.7.gz  messages.11.gz         messages.5.gz   pakfire           suricata
bootlog.13.gz      bootlog.6.gz  calamaris       dhcpcd.log.13.gz  dhcpcd.log.6.gz  logwatch         mail.1.gz   mail.8.gz  messages.12.gz         messages.6.gz   pakfire.log       updatexlrator
bootlog.14.gz      bootlog.7.gz  connect-errors  dhcpcd.log.14.gz  dhcpcd.log.7.gz  mail             mail.2.gz   mail.9.gz  messages.13.gz         messages.7.gz   rrd               vnstat
[root@ipfire log]# vi messages
Nov 15 03:53:53 ipfire kernel: DROP_INPUT IN=red0 OUT= MAC=ac:1f:6b:10:cb:a0:84:bb:69:d2:b8:b0:08:00 SRC=24.38.71.10 DST=<external-ip> LEN=84 TOS=0x00 PREC=0x00 TTL=108 ID=37763 PROTO=UDP SPT=55166 DPT=57892 LEN=64 MARK=0x80000000
Nov 15 03:53:53 ipfire kernel: DROP_INPUT IN=red0 OUT= MAC=ac:1f:6b:10:cb:a0:84:bb:69:d2:b8:b0:08:00 SRC=68.105.124.157 DST=<external-ip> LEN=52 TOS=0x00 PREC=0x00 TTL=108 ID=19915 DF PROTO=TCP SPT=54395 DPT=7680 WINDOW=64240 RES=0x00 SYN URGP=0 MARK=0x8000d200
Nov 15 03:53:53 ipfire kernel: DROP_INPUT IN=red0 OUT= MAC=ac:1f:6b:10:cb:a0:84:bb:69:d2:b8:b0:08:00 SRC=24.38.71.10 DST=<external-ip> LEN=84 TOS=0x00 PREC=0x00 TTL=108 ID=37765 PROTO=UDP SPT=55166 DPT=57892 LEN=64 MARK=0x80000000
Nov 15 03:53:53 ipfire kernel: DROP_INPUT IN=red0 OUT= MAC=ac:1f:6b:10:cb:a0:84:bb:69:d2:b8:b0:08:00 SRC=24.38.71.10 DST=<external-ip> LEN=80 TOS=0x00 PREC=0x00 TTL=108 ID=37764 PROTO=UDP SPT=57441 DPT=57892 LEN=60 MARK=0x80000000
Nov 15 03:53:54 ipfire kernel: DROP_INPUT IN=red0 OUT= MAC=ac:1f:6b:10:cb:a0:84:bb:69:d2:b8:b0:08:00 SRC=144.121.168.196 DST=<external-ip> LEN=52 TOS=0x00 PREC=0x00 TTL=113 ID=10999 DF PROTO=TCP SPT=9915 DPT=7680 WINDOW=64240 RES=0x00 SYN URGP=0 MARK=0x8000d200
Nov 15 03:53:54 ipfire kernel: DROP_INPUT IN=red0 OUT= MAC=ac:1f:6b:10:cb:a0:84:bb:69:d2:b8:b0:08:00 SRC=100.11.124.120 DST=<external-ip> LEN=80 TOS=0x00 PREC=0x00 TTL=115 ID=58849 PROTO=UDP SPT=63583 DPT=55701 LEN=60 MARK=0x80000000
Nov 15 03:53:54 ipfire kernel: DROP_INPUT IN=red0 OUT= MAC=ac:1f:6b:10:cb:a0:84:bb:69:d2:b8:b0:08:00 SRC=52.162.82.248 DST=<external-ip> LEN=84 TOS=0x00 PREC=0x00 TTL=111 ID=32375 PROTO=UDP SPT=3544 DPT=55701 LEN=64 MARK=0x80000000
Nov 15 03:53:54 ipfire kernel: DROP_INPUT IN=red0 OUT= MAC=ac:1f:6b:10:cb:a0:84:bb:69:d2:b8:b0:08:00 SRC=50.200.106.246 DST=<external-ip> LEN=80 TOS=0x00 PREC=0x00 TTL=112 ID=52472 PROTO=UDP SPT=62808 DPT=55701 LEN=60 MARK=0x80000000
Nov 15 03:53:54 ipfire kernel: DROP_INPUT IN=red0 OUT= MAC=ac:1f:6b:10:cb:a0:84:bb:69:d2:b8:b0:08:00 SRC=52.162.82.248 DST=<external-ip> LEN=84 TOS=0x00 PREC=0x00 TTL=110 ID=28103 PROTO=UDP SPT=3544 DPT=55701 LEN=64 MARK=0x80000000
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@Nov 15 03:56:11 ipfire syslogd 1.5.1: restart (remote reception).
Nov 15 03:56:11 ipfire kernel: pps pps0: new PPS source ptp0
Nov 15 03:56:11 ipfire kernel: igb 0000:02:00.0: added PHC on eth0
Nov 15 03:56:11 ipfire kernel: igb 0000:02:00.0: Intel(R) Gigabit Ethernet Network Connection
Nov 15 03:56:11 ipfire kernel: igb 0000:02:00.0: eth0: (PCIe:2.5Gb/s:Width x1) ac:1f:6b:10:cb:a0
Nov 15 03:56:11 ipfire kernel: igb 0000:02:00.0: eth0: PBA No: 012700-000
Nov 15 03:56:11 ipfire kernel: igb 0000:02:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
Nov 15 03:56:11 ipfire kernel: pps pps1: new PPS source ptp1
Nov 15 03:56:11 ipfire kernel: igb 0000:05:00.0: added PHC on eth1
Nov 15 03:56:11 ipfire kernel: igb 0000:05:00.0: Intel(R) Gigabit Ethernet Network Connection
Nov 15 03:56:11 ipfire kernel: igb 0000:05:00.0: eth1: (PCIe:2.5Gb/s:Width x1) ac:1f:6b:10:cb:a1
Nov 15 03:56:11 ipfire kernel: igb 0000:05:00.0: eth1: PBA No: 012700-000
Nov 15 03:56:11 ipfire kernel: igb 0000:05:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
Nov 15 03:56:11 ipfire kernel: pps pps2: new PPS source ptp2
Nov 15 03:56:11 ipfire kernel: igb 0000:06:00.0: added PHC on eth2
Nov 15 03:56:11 ipfire kernel: igb 0000:06:00.0: Intel(R) Gigabit Ethernet Network Connection
Nov 15 03:56:11 ipfire kernel: igb 0000:06:00.0: eth2: (PCIe:2.5Gb/s:Width x1) ac:1f:6b:10:cb:a2
Nov 15 03:56:11 ipfire kernel: igb 0000:06:00.0: eth2: PBA No: 012700-000
Nov 15 03:56:11 ipfire kernel: igb 0000:06:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
Nov 15 03:56:11 ipfire kernel: pps pps3: new PPS source ptp3
Nov 15 03:56:11 ipfire kernel: igb 0000:07:00.0: added PHC on eth3
Nov 15 03:56:11 ipfire kernel: igb 0000:07:00.0: Intel(R) Gigabit Ethernet Network Connection
Nov 15 03:56:11 ipfire kernel: igb 0000:07:00.0: eth3: (PCIe:2.5Gb/s:Width x1) ac:1f:6b:10:cb:a3
Nov 15 03:56:11 ipfire kernel: igb 0000:07:00.0: eth3: PBA No: 012700-000
Nov 15 03:56:11 ipfire kernel: igb 0000:07:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
Nov 15 03:56:11 ipfire kernel: iTCO_vendor_support: vendor-support=0
Nov 15 03:56:11 ipfire kernel: iTCO_wdt iTCO_wdt.1.auto: Found a Braswell SoC TCO device (Version=3, TCOBASE=0x0460)
Nov 15 03:56:11 ipfire kernel: iTCO_wdt iTCO_wdt.1.auto: initialized. heartbeat=30 sec (nowayout=1)
Nov 15 03:56:11 ipfire kernel: at24 0-0050: 256 byte spd EEPROM, read-only
Nov 15 03:56:11 ipfire kernel: at24 0-0051: 256 byte spd EEPROM, read-only
Nov 15 03:56:11 ipfire kernel: igb 0000:06:00.0 blue0: renamed from eth2
Nov 15 03:56:11 ipfire kernel: igb 0000:05:00.0 green0: renamed from eth1
Nov 15 03:56:11 ipfire kernel: snd_hda_intel 0000:00:1b.0: couldn't bind with audio component
Nov 15 03:56:11 ipfire kernel: igb 0000:02:00.0 red0: renamed from eth0
Nov 15 03:56:11 ipfire kernel: igb 0000:07:00.0 orange0: renamed from eth3
Nov 15 03:56:11 ipfire kernel: ipmi_si IPI0001:00: IPMI kcs interface initialized
-- VISUAL --                                                                                                                                                                                                     10        1408799,97    95%
-----------------------------------------

Graphs:

I also noticed something odd, over time we have staff turn over just like any business, but I can see in /var/log/messages that there are entries that say ipfire openvpnserver[6299]: ifconfig_pool_read(), in='olduser,. There are a few dozen entries like this, a mixture of existing users that do have openvpn profiles in the WUI, and ones that do not.

Nov 13 22:57:08 ipfire openvpnserver[6299]: IFCONFIG POOL LIST
Nov 13 22:57:08 ipfire openvpnserver[6299]: charlie,10.8.1.4,
Nov 13 22:57:08 ipfire openvpnserver[6299]: dylan,10.8.1.8,
Nov 13 22:57:08 ipfire openvpnserver[6299]: rebecca2,10.8.1.12,
Nov 13 22:57:08 ipfire openvpnserver[6299]: paul,10.8.1.16,
Nov 13 22:57:08 ipfire openvpnserver[6299]: benj,10.8.1.20,
Nov 13 22:57:08 ipfire openvpnserver[6299]: brian,10.8.1.24,
Nov 13 22:57:08 ipfire openvpnserver[6299]: steve,10.8.1.28,
Nov 13 22:57:08 ipfire openvpnserver[6299]: al,10.8.1.32,
Nov 13 22:57:08 ipfire openvpnserver[6299]: chris,10.8.1.36,
Nov 13 22:57:08 ipfire openvpnserver[6299]: leah,10.8.1.40,
Nov 13 22:57:08 ipfire openvpnserver[6299]: dave,10.8.1.44,
Nov 13 22:57:08 ipfire openvpnserver[6299]: scott,10.8.1.48,
Nov 13 22:57:08 ipfire openvpnserver[6299]: wayne,10.8.1.52,
Nov 13 22:57:08 ipfire openvpnserver[6299]: cwi,10.8.1.56,
Nov 13 22:57:08 ipfire openvpnserver[6299]: elizabeth,10.8.1.60,
Nov 13 22:57:08 ipfire openvpnserver[6299]: spmobile,10.8.1.64,
Nov 13 22:57:08 ipfire openvpnserver[6299]: rebeccasi,10.8.1.68,
Nov 13 22:57:08 ipfire openvpnserver[6299]: chrisk,10.8.1.72,
Nov 13 22:57:08 ipfire openvpnserver[6299]: dee,10.8.1.76,
Nov 13 22:57:08 ipfire openvpnserver[6299]: timdewanz2,10.8.1.80,
Nov 13 22:57:08 ipfire openvpnserver[6299]: waynep,10.8.1.84,
Nov 13 22:57:08 ipfire openvpnserver[6299]: stevep,10.8.1.88,
Nov 13 22:57:08 ipfire openvpnserver[6299]: alb,10.8.1.92,

In this example old users ‘dee’, ‘rebeccasi’, ‘leah’ that do not have WUI profiles are still coming up in the listing. I don’t want them to be there, they should not be there. How do I fully remove them from all configurations?

Is this somehow a factor in the high ram usage creep that is seem in the graphs?