IPFire went down last night, can't find cause

bonnietwin · 9 November 2022 17:00

It might be but I am not sure. That bug is related to where the used memory (ie the blue part of the memory graph) goes up to 100% because of the OpenVPN Authenticator and then the OOM Killer kicks in (Out Of Memory) and kills the OpenVPN Authenticator.

Your graph is showing the yellow portion of the memory increasing but this is cached memory where IPFire caches stuff from the Disk Drive to be able to access quicker if needed again but if the memory is needed for real usage then the yellow is freed up for the blue. It might be that if you left it running the OpenVPN Authenticator would continue to increase it’s memory consumption and eventually hit the 100% actual used level.

I don’t see the above at all. I don’t see any blue memory peaks and also the OpenVPN Authenticator is not using CPU time on my system.
From the bug it looks like this is occurring when the bug reporter created a new Host/Root certificate in OpenVPN while on CU170.

I plan to test this out with a vm test machine to see if it is the creation of a new Host/Root certificate that starts OpenVPN Authenticator using high amounts of CPU time and eventually memory but I won’t be able to do this for a day or so as I was away and am now busy with updates of python and consequently all python modules and programs using python for submission as a patch set for CU172.

I will report back here as well as into the bug report on if I am able to replicate the effect.

jon · 9 November 2022 17:01

My Memory charts looks like this. And all is A-OK.

So I am not sure your Memory charts are indicating a problem. The openvpn-auth does look “odd” but I don’t know why…

EDIT: This chart with the fast rising blue indicates a problem:

jon · 9 November 2022 17:27

Chris - In the image above in Post 84

258c1fd2fed8b41b5b537385658ce50bd9686abe_2_690x319

and in Post 88

the network seems to be busy between midnight and 7:30 AM.

I am curious:

how many devices are attached to your GREEN network?
what is going on at between 12 and 7:30 AM?

This is just my curiosity and me going off on an odd tangent!.

cwensink · 9 November 2022 17:46

Network is busy because backups are occurring, we have about 100 PC’s and laptops plus printers, barcode scanners, etc.

Chris

cwensink · 9 November 2022 17:47

Jon,

Every time you update to a new core version do you destroy and generate new certificates for openvpn and ipsec, then re-create VPN connections, or do you keep the same ones from version to the next?

jon · 9 November 2022 18:04

every time “no”. sometimes “yes”. There have been security related updates to OpenVPN or IPsec over the years that I want to take advantage of. So I will trash everything related to VPN and rebuild.

It is sometimes related to Diffie-Hellman parameter length.

Something like this:
https://git.ipfire.org/?p=ipfire-2.x.git;a=commit;h=46803376d54f7e42d0eefe25155de82672bff41e

or in the near future, like this:
https://lists.ipfire.org/pipermail/development/2022-November/014601.html

bonnietwin · 11 November 2022 17:05

I have now cloned a vm running CU171, with a confirmed working OpenVPN system.
Running htop it showed openvpn-authenticator running at 0% cpu and 1.1% of the 1GB memory.

I then removed the x509 certificate set, which also removed all the client connection details.

I then created a new Root and Host certificate set using the same details as used previously. Then created a new client connection which was then transferred to my Android phone and I was able to successfully run a tunnel to a vm client in the vm testbed network just as previously done.

Running htop it showed openvpn-authenticator at 0% cpu and 1.0% of the 1GB memory.

I left the system running for 2 hours and nothing changed in the openvpn authenticator values of cpu and memory usage.

I then rebooted the IPFire vm and ran it again for 2 hours and again htop showed 0% cpu and 1.0% memory.

I will leave this running overnight but so far I have not been able to duplicate the effect that has been reported of openvpn-authenticator having a large cpu% and with memory climbing up till it uses all free memory, even with a fresh install of the Root/Host certificate set.

Will report back tomorrow if anything changes after running overnight.

bonnietwin · 11 November 2022 17:34

I got clarification from @schami23 that the openvpn-authenticator cpu and memory usage resulting in the oom killer only occurs when the OpenVPN server is stopped.

I turned off the OpenVPN Server and openvpn-authenticator went to something like 89% cpu% and memory spiked very quickly and triggered swap to be used to nearly 100% and then openvpn-authenticator was removed by the system - no longer in the htop table at all.

Looking in the kernel logs I have the oom killer message

18:17:20 kernel: Out of memory: Killed process 9525 (openvpn-authent) total-vm:831900kB, anon-rss :741952kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:1556kB oom_score_adj:0

18:17:20 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,glob al_oom,task_memcg=/,task=openvpn-authent,pid=9525,uid=0

So, yes I can duplicate the oom killer problem but it only seems to occur when the OpenVPN Server is turned off. It would appear that openvpn-authenticator doesn’t like to be running without the OpenVPN server turned on. Maybe we need to turn off the openvpn-authenticator when the OpenVPN server is stopped.

I will do the same test on a clone with the original Root/Host certificate to see if that has a different response or not.

I will also have a ook at the openvpn-authenticator code to see if I can spot anything obvious with regard to this effect.

Also after the openvpn-authenticator was stopped by the oom killer IPFire continued to run normally and cpu and memory % went back to normal. So this did not duplicate the whole IPFire shutdown or reboot that @cwensink is experiencing.

tphz · 11 November 2022 17:46

I can confirm this behavior
edit

After stopping the openvpn (WUI) server, I see high CPU load.
After restarting the openvpn setwer (WUI), the load does not decrease.

bonnietwin · 11 November 2022 17:49

I have also confirmed that the same thing happens on my original clone vm which has an old Root/Host certificate from many many CU’s ago.

So the problem looks not to be related to creating a new Root/Host certificate but just on openvpn-authenticator not liking to be running when OpenVPN Server is stopped.

Unfortunately the openvpn-authenticator is python code which I am not familiar with at all, so it will take a bit of work to look through it, but will give it a go.

bonnietwin · 11 November 2022 18:26

The problem looks to not be the openvpn-authenticator python code but the openvpnctrl c code.

There is a command stopAuthenticator which is part of the stopDaemon command but it is using /sbin/killall to stop the authenticator and in sbin there is only killall5. killall is in /bin so the /sbin/killall command will end up not doing anything because it can’t be found and so the openvpn-authenticator is not stopped.

I will edit the code and do a build and install of IPFire on my vm system to test this hypothesis out.

schami23 · 15 November 2022 14:26

Is there something new? Have you already found the problem?

Thanks
Jürgen

cwensink · 15 November 2022 15:08

I’m still having the problem. Our Production Machine rebooted on it’s own last night at 03:53:54 AM,

[root@ipfire ~]# uptime
 08:05:55 up  4:10,  1 user,  load average: 0.30, 0.44, 0.37
[root@ipfire ~]# top
top - 08:45:03 up  4:49,  1 user,  load average: 0.25, 0.33, 0.38
Tasks: 130 total,   1 running, 129 sleeping,   0 stopped,   0 zombie
%Cpu0  :   1.4/1.4     3[||                                                                                        ]   %Cpu1  :   0.7/3.5     4[||||                                                                                      ]
%Cpu2  :   0.7/15.0   16[||||||||||||||                                                                            ]   %Cpu3  :   8.7/10.7   19[||||||||||||||||||                                                                        ]
GiB Mem :  5.8/7.7      [                                                                                          ]   GiB Swap:  0.0/1.0      [                                                                                          ]

  PID USER      PR  NI    VIRT    RES  %CPU  %MEM     TIME+ S COMMAND
    1 root      20   0    2.4m   1.6m   0.0   0.0   0:03.52 S init [3]
  634 root      20   0    4.0m   0.1m   0.0   0.0   0:00.00 S  `- /usr/sbin/lvmetad
  653 root      20   0   18.8m   3.2m   0.0   0.0   0:00.35 S  `- udevd --daemon
  890 root      20   0    9.9m   0.4m   0.0   0.0   0:00.00 S  `- /usr/sbin/rngd -r /dev/hwrng --quiet
 3576 root      20   0    5.3m   3.6m   0.0   0.0   0:10.79 S  `- /usr/sbin/vnstatd -d --alwaysadd
 3587 root      20   0    9.5m   8.5m   0.0   0.1   0:08.50 S  `- klogd -c 1
 3594 root      20   0    2.4m   0.1m   0.7   0.0   0:11.65 S  `- syslogd -m 0 -r
 3743 nobody    20   0   23.9m  21.6m   0.0   0.3   0:48.25 S  `- /usr/sbin/unbound
 3754 root      20   0    2.5m   0.1m   0.0   0.0   0:00.00 S  `- /usr/sbin/acpid
 5212 suricata  20   0  481.8m  72.7m  20.5   0.9  27:10.45 S  `- /usr/bin/suricata -c /etc/suricata/suricata.yaml -D -q 0:3
 5402 root      20   0   28.3m  14.0m   0.0   0.2   0:03.28 S  `- /usr/bin/perl -w /usr/local/bin/qosd red0
 5405 root      20   0   28.3m  14.2m   0.0   0.2   0:03.19 S  `- /usr/bin/perl -w /usr/local/bin/qosd imq0
 6221 root      20   0   21.6m   6.3m   0.0   0.1   0:00.00 S  `- /usr/sbin/squid
 6224 squid     20   0   45.0m  34.6m   0.0   0.4   0:30.59 S      `- (squid-1) --kid squid-1
 6229 squid     20   0   15.9m  11.2m   0.0   0.1   0:01.36 S          `- /usr/bin/perl /usr/sbin/redirect_wrapper
 6256 squid     20   0    4.7m   3.5m   0.0   0.0   0:00.62 S              `- /usr/bin/squidGuard
 6258 squid     20   0    8.7m   6.4m   0.0   0.1   0:00.85 S              `- /usr/bin/perl /usr/sbin/updxlrator
 6230 squid     20   0   15.9m  11.3m   0.0   0.1   0:00.37 S          `- /usr/bin/perl /usr/sbin/redirect_wrapper
 6254 squid     20   0    4.5m   3.4m   0.0   0.0   0:00.03 S              `- /usr/bin/squidGuard
 6255 squid     20   0    8.7m   6.5m   0.0   0.1   0:00.08 S              `- /usr/bin/perl /usr/sbin/updxlrator
 6231 squid     20   0   15.9m  11.3m   0.0   0.1   0:00.29 S          `- /usr/bin/perl /usr/sbin/redirect_wrapper
 6257 squid     20   0    4.5m   2.0m   0.0   0.0   0:00.00 S              `- /usr/bin/squidGuard
 6259 squid     20   0    8.7m   6.5m   0.0   0.1   0:00.05 S              `- /usr/bin/perl /usr/sbin/updxlrator
 6233 squid     20   0   15.9m  11.5m   0.0   0.1   0:00.27 S          `- /usr/bin/perl /usr/sbin/redirect_wrapper
 6260 squid     20   0    4.5m   2.1m   0.0   0.0   0:00.00 S              `- /usr/bin/squidGuard
 6261 squid     20   0    8.7m   6.4m   0.0   0.1   0:00.05 S              `- /usr/bin/perl /usr/sbin/updxlrator
 6333 root      20   0    4.6m   1.7m   0.0   0.0   0:00.00 S  `- /usr/libexec/ipsec/starter --daemon charon
 6334 root      20   0 1165.9m  11.8m   0.0   0.2   1:08.80 S      `- /usr/libexec/ipsec/charon --use-syslog
 6372 nobody    20   0    7.4m   5.8m   0.0   0.1   0:05.43 S  `- /usr/sbin/openvpn --config /var/ipfire/ovpn/server.conf
 6435 root      20   0   15.7m  10.6m   0.0   0.1   0:00.04 S  `- /usr/bin/python3 /usr/sbin/openvpn-authenticator -d
 6550 root      20   0    2.8m   0.1m   0.0   0.0   0:00.00 S  `- /usr/sbin/saslauthd -n 2 -a pam
 6551 root      20   0    2.8m   0.1m   0.0   0.0   0:00.00 S      `- /usr/sbin/saslauthd -n 2 -a pam
 6636 root      20   0   72.4m   3.8m   0.0   0.0   0:02.45 S  `- /usr/bin/ntpd -Ap /var/run/ntpd.pid
 6689 root      20   0  232.6m   4.1m   0.0   0.1   0:31.34 S  `- /usr/sbin/collectd -C /etc/collectd.conf
 6717 root      20   0    9.1m   6.9m   0.0   0.1   0:01.28 S  `- /usr/sbin/dhcpd -q green0 blue0
 6727 root      20   0   17.6m  12.9m   0.0   0.2   0:37.47 S  `- /usr/bin/python3 /usr/sbin/unbound-dhcp-leases-bridge -d
 6738 root      20   0    6.5m   2.2m   0.0   0.0   0:00.00 S  `- sshd: /usr/sbin/sshd [listener] 0 of 5-5 startups
 1408 root      20   0    6.9m   5.5m   0.0   0.1   0:00.99 S      `- sshd: root@pts/0
 1424 root      20   0    7.6m   4.3m   0.0   0.1   0:00.02 S          `- -bash
 1777 root      20   0    8.3m   4.8m   1.3   0.1   0:21.14 R              `- top
 6752 root      20   0   11.2m   7.2m   0.0   0.1   0:01.86 S  `- /usr/sbin/httpd -k start
 6753 nobody    20   0   10.4m   3.7m   0.7   0.0   0:19.31 S      `- /usr/sbin/httpd -k start
 6756 nobody    20   0 1307.4m  13.4m   0.7   0.2   4:15.54 S      `- /usr/sbin/httpd -k start
 6802 root      20   0    3.5m   1.7m   0.0   0.0   0:00.41 S  `- /usr/sbin/fcron -y
 6805 root      20   0    2.9m   2.2m   0.0   0.0   0:00.02 S  `- /sbin/agetty console --noclear
 6806 root      20   0    2.9m   2.1m   0.0   0.0   0:00.00 S  `- /sbin/agetty tty2
 6807 root      20   0    2.9m   2.2m   0.0   0.0   0:00.00 S  `- /sbin/agetty tty3
 6808 root      20   0    2.9m   2.2m   0.0   0.0   0:00.00 S  `- /sbin/agetty tty4
 6809 root      20   0    2.9m   2.1m   0.0   0.0   0:00.00 S  `- /sbin/agetty tty5
 6810 root      20   0    2.9m   2.2m   0.0   0.0   0:00.00 S  `- /sbin/agetty tty6
    2 root      20   0    0.0m   0.0m   0.0   0.0   0:00.01 S [kthreadd]
    3 root       0 -20    0.0m   0.0m   0.0   0.0   0:00.00 I  `- [rcu_gp]
    4 root       0 -20    0.0m   0.0m   0.0   0.0   0:00.00 I  `- [rcu_par_gp]
    5 root       0 -20    0.0m   0.0m   0.0   0.0   0:00.00 I  `- [slub_flushwq]
    6 root       0 -20    0.0m   0.0m   0.0   0.0   0:00.00 I  `- [netns]
    8 root       0 -20    0.0m   0.0m   0.0   0.0   0:00.00 I  `- [kworker/0:0H-events_highpri]
[root@ipfire ~]# cd /var/log
[root@ipfire log]# ls
10-19-22-1500.txt  bootlog.1.gz  bootlog.8.gz    counter           dhcpcd.log.1.gz  dhcpcd.log.8.gz  mail.10.gz  mail.3.gz  messages               messages.14.gz  messages.8.gz     sa             wtmp
bootlog            bootlog.2.gz  bootlog.9.gz    dhcpcd.log        dhcpcd.log.2.gz  dhcpcd.log.9.gz  mail.11.gz  mail.4.gz  messages.1             messages.2.gz   messages.9.gz     setup.log      wtmp.1.gz
bootlog.10.gz      bootlog.3.gz  bootlog.old     dhcpcd.log.10.gz  dhcpcd.log.3.gz  haproxy          mail.12.gz  mail.5.gz  messages.10.24.22.txt  messages.3.gz   messags.10.27.22  squid
bootlog.11.gz      bootlog.4.gz  btmp            dhcpcd.log.11.gz  dhcpcd.log.4.gz  httpd            mail.13.gz  mail.6.gz  messages.10.gz         messages.4.gz   openvpn           squidGuard
bootlog.12.gz      bootlog.5.gz  cache           dhcpcd.log.12.gz  dhcpcd.log.5.gz  lastlog          mail.14.gz  mail.7.gz  messages.11.gz         messages.5.gz   pakfire           suricata
bootlog.13.gz      bootlog.6.gz  calamaris       dhcpcd.log.13.gz  dhcpcd.log.6.gz  logwatch         mail.1.gz   mail.8.gz  messages.12.gz         messages.6.gz   pakfire.log       updatexlrator
bootlog.14.gz      bootlog.7.gz  connect-errors  dhcpcd.log.14.gz  dhcpcd.log.7.gz  mail             mail.2.gz   mail.9.gz  messages.13.gz         messages.7.gz   rrd               vnstat
[root@ipfire log]# vi messages
Nov 15 03:53:53 ipfire kernel: DROP_INPUT IN=red0 OUT= MAC=ac:1f:6b:10:cb:a0:84:bb:69:d2:b8:b0:08:00 SRC=24.38.71.10 DST=<external-ip> LEN=84 TOS=0x00 PREC=0x00 TTL=108 ID=37763 PROTO=UDP SPT=55166 DPT=57892 LEN=64 MARK=0x80000000
Nov 15 03:53:53 ipfire kernel: DROP_INPUT IN=red0 OUT= MAC=ac:1f:6b:10:cb:a0:84:bb:69:d2:b8:b0:08:00 SRC=68.105.124.157 DST=<external-ip> LEN=52 TOS=0x00 PREC=0x00 TTL=108 ID=19915 DF PROTO=TCP SPT=54395 DPT=7680 WINDOW=64240 RES=0x00 SYN URGP=0 MARK=0x8000d200
Nov 15 03:53:53 ipfire kernel: DROP_INPUT IN=red0 OUT= MAC=ac:1f:6b:10:cb:a0:84:bb:69:d2:b8:b0:08:00 SRC=24.38.71.10 DST=<external-ip> LEN=84 TOS=0x00 PREC=0x00 TTL=108 ID=37765 PROTO=UDP SPT=55166 DPT=57892 LEN=64 MARK=0x80000000
Nov 15 03:53:53 ipfire kernel: DROP_INPUT IN=red0 OUT= MAC=ac:1f:6b:10:cb:a0:84:bb:69:d2:b8:b0:08:00 SRC=24.38.71.10 DST=<external-ip> LEN=80 TOS=0x00 PREC=0x00 TTL=108 ID=37764 PROTO=UDP SPT=57441 DPT=57892 LEN=60 MARK=0x80000000
Nov 15 03:53:54 ipfire kernel: DROP_INPUT IN=red0 OUT= MAC=ac:1f:6b:10:cb:a0:84:bb:69:d2:b8:b0:08:00 SRC=144.121.168.196 DST=<external-ip> LEN=52 TOS=0x00 PREC=0x00 TTL=113 ID=10999 DF PROTO=TCP SPT=9915 DPT=7680 WINDOW=64240 RES=0x00 SYN URGP=0 MARK=0x8000d200
Nov 15 03:53:54 ipfire kernel: DROP_INPUT IN=red0 OUT= MAC=ac:1f:6b:10:cb:a0:84:bb:69:d2:b8:b0:08:00 SRC=100.11.124.120 DST=<external-ip> LEN=80 TOS=0x00 PREC=0x00 TTL=115 ID=58849 PROTO=UDP SPT=63583 DPT=55701 LEN=60 MARK=0x80000000
Nov 15 03:53:54 ipfire kernel: DROP_INPUT IN=red0 OUT= MAC=ac:1f:6b:10:cb:a0:84:bb:69:d2:b8:b0:08:00 SRC=52.162.82.248 DST=<external-ip> LEN=84 TOS=0x00 PREC=0x00 TTL=111 ID=32375 PROTO=UDP SPT=3544 DPT=55701 LEN=64 MARK=0x80000000
Nov 15 03:53:54 ipfire kernel: DROP_INPUT IN=red0 OUT= MAC=ac:1f:6b:10:cb:a0:84:bb:69:d2:b8:b0:08:00 SRC=50.200.106.246 DST=<external-ip> LEN=80 TOS=0x00 PREC=0x00 TTL=112 ID=52472 PROTO=UDP SPT=62808 DPT=55701 LEN=60 MARK=0x80000000
Nov 15 03:53:54 ipfire kernel: DROP_INPUT IN=red0 OUT= MAC=ac:1f:6b:10:cb:a0:84:bb:69:d2:b8:b0:08:00 SRC=52.162.82.248 DST=<external-ip> LEN=84 TOS=0x00 PREC=0x00 TTL=110 ID=28103 PROTO=UDP SPT=3544 DPT=55701 LEN=64 MARK=0x80000000
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@Nov 15 03:56:11 ipfire syslogd 1.5.1: restart (remote reception).
Nov 15 03:56:11 ipfire kernel: pps pps0: new PPS source ptp0
Nov 15 03:56:11 ipfire kernel: igb 0000:02:00.0: added PHC on eth0
Nov 15 03:56:11 ipfire kernel: igb 0000:02:00.0: Intel(R) Gigabit Ethernet Network Connection
Nov 15 03:56:11 ipfire kernel: igb 0000:02:00.0: eth0: (PCIe:2.5Gb/s:Width x1) ac:1f:6b:10:cb:a0
Nov 15 03:56:11 ipfire kernel: igb 0000:02:00.0: eth0: PBA No: 012700-000
Nov 15 03:56:11 ipfire kernel: igb 0000:02:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
Nov 15 03:56:11 ipfire kernel: pps pps1: new PPS source ptp1
Nov 15 03:56:11 ipfire kernel: igb 0000:05:00.0: added PHC on eth1
Nov 15 03:56:11 ipfire kernel: igb 0000:05:00.0: Intel(R) Gigabit Ethernet Network Connection
Nov 15 03:56:11 ipfire kernel: igb 0000:05:00.0: eth1: (PCIe:2.5Gb/s:Width x1) ac:1f:6b:10:cb:a1
Nov 15 03:56:11 ipfire kernel: igb 0000:05:00.0: eth1: PBA No: 012700-000
Nov 15 03:56:11 ipfire kernel: igb 0000:05:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
Nov 15 03:56:11 ipfire kernel: pps pps2: new PPS source ptp2
Nov 15 03:56:11 ipfire kernel: igb 0000:06:00.0: added PHC on eth2
Nov 15 03:56:11 ipfire kernel: igb 0000:06:00.0: Intel(R) Gigabit Ethernet Network Connection
Nov 15 03:56:11 ipfire kernel: igb 0000:06:00.0: eth2: (PCIe:2.5Gb/s:Width x1) ac:1f:6b:10:cb:a2
Nov 15 03:56:11 ipfire kernel: igb 0000:06:00.0: eth2: PBA No: 012700-000
Nov 15 03:56:11 ipfire kernel: igb 0000:06:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
Nov 15 03:56:11 ipfire kernel: pps pps3: new PPS source ptp3
Nov 15 03:56:11 ipfire kernel: igb 0000:07:00.0: added PHC on eth3
Nov 15 03:56:11 ipfire kernel: igb 0000:07:00.0: Intel(R) Gigabit Ethernet Network Connection
Nov 15 03:56:11 ipfire kernel: igb 0000:07:00.0: eth3: (PCIe:2.5Gb/s:Width x1) ac:1f:6b:10:cb:a3
Nov 15 03:56:11 ipfire kernel: igb 0000:07:00.0: eth3: PBA No: 012700-000
Nov 15 03:56:11 ipfire kernel: igb 0000:07:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
Nov 15 03:56:11 ipfire kernel: iTCO_vendor_support: vendor-support=0
Nov 15 03:56:11 ipfire kernel: iTCO_wdt iTCO_wdt.1.auto: Found a Braswell SoC TCO device (Version=3, TCOBASE=0x0460)
Nov 15 03:56:11 ipfire kernel: iTCO_wdt iTCO_wdt.1.auto: initialized. heartbeat=30 sec (nowayout=1)
Nov 15 03:56:11 ipfire kernel: at24 0-0050: 256 byte spd EEPROM, read-only
Nov 15 03:56:11 ipfire kernel: at24 0-0051: 256 byte spd EEPROM, read-only
Nov 15 03:56:11 ipfire kernel: igb 0000:06:00.0 blue0: renamed from eth2
Nov 15 03:56:11 ipfire kernel: igb 0000:05:00.0 green0: renamed from eth1
Nov 15 03:56:11 ipfire kernel: snd_hda_intel 0000:00:1b.0: couldn't bind with audio component
Nov 15 03:56:11 ipfire kernel: igb 0000:02:00.0 red0: renamed from eth0
Nov 15 03:56:11 ipfire kernel: igb 0000:07:00.0 orange0: renamed from eth3
Nov 15 03:56:11 ipfire kernel: ipmi_si IPI0001:00: IPMI kcs interface initialized
-- VISUAL --                                                                                                                                                                                                     10        1408799,97    95%
-----------------------------------------

Graphs:

I also noticed something odd, over time we have staff turn over just like any business, but I can see in /var/log/messages that there are entries that say ipfire openvpnserver[6299]: ifconfig_pool_read(), in='olduser,. There are a few dozen entries like this, a mixture of existing users that do have openvpn profiles in the WUI, and ones that do not.

Nov 13 22:57:08 ipfire openvpnserver[6299]: IFCONFIG POOL LIST
Nov 13 22:57:08 ipfire openvpnserver[6299]: charlie,10.8.1.4,
Nov 13 22:57:08 ipfire openvpnserver[6299]: dylan,10.8.1.8,
Nov 13 22:57:08 ipfire openvpnserver[6299]: rebecca2,10.8.1.12,
Nov 13 22:57:08 ipfire openvpnserver[6299]: paul,10.8.1.16,
Nov 13 22:57:08 ipfire openvpnserver[6299]: benj,10.8.1.20,
Nov 13 22:57:08 ipfire openvpnserver[6299]: brian,10.8.1.24,
Nov 13 22:57:08 ipfire openvpnserver[6299]: steve,10.8.1.28,
Nov 13 22:57:08 ipfire openvpnserver[6299]: al,10.8.1.32,
Nov 13 22:57:08 ipfire openvpnserver[6299]: chris,10.8.1.36,
Nov 13 22:57:08 ipfire openvpnserver[6299]: leah,10.8.1.40,
Nov 13 22:57:08 ipfire openvpnserver[6299]: dave,10.8.1.44,
Nov 13 22:57:08 ipfire openvpnserver[6299]: scott,10.8.1.48,
Nov 13 22:57:08 ipfire openvpnserver[6299]: wayne,10.8.1.52,
Nov 13 22:57:08 ipfire openvpnserver[6299]: cwi,10.8.1.56,
Nov 13 22:57:08 ipfire openvpnserver[6299]: elizabeth,10.8.1.60,
Nov 13 22:57:08 ipfire openvpnserver[6299]: spmobile,10.8.1.64,
Nov 13 22:57:08 ipfire openvpnserver[6299]: rebeccasi,10.8.1.68,
Nov 13 22:57:08 ipfire openvpnserver[6299]: chrisk,10.8.1.72,
Nov 13 22:57:08 ipfire openvpnserver[6299]: dee,10.8.1.76,
Nov 13 22:57:08 ipfire openvpnserver[6299]: timdewanz2,10.8.1.80,
Nov 13 22:57:08 ipfire openvpnserver[6299]: waynep,10.8.1.84,
Nov 13 22:57:08 ipfire openvpnserver[6299]: stevep,10.8.1.88,
Nov 13 22:57:08 ipfire openvpnserver[6299]: alb,10.8.1.92,

In this example old users ‘dee’, ‘rebeccasi’, ‘leah’ that do not have WUI profiles are still coming up in the listing. I don’t want them to be there, they should not be there. How do I fully remove them from all configurations?

Is this somehow a factor in the high ram usage creep that is seem in the graphs?

cwensink · 15 November 2022 15:16

I should also mention to everyone that over the weekend IPFire stopped communicating with the internal LAN. When I came into the building the appliance was still powered on. I could log into the console, I could communicate and ping public domains and ip addresses, but I could not ping or communicate with anything on the green interface (LAN) other than the green’s IP address.

Before restarting I tried starting / restarting all services I could in /etc/init.d including:

acpid, apache, beep, collectd, console, cyrus-sasl, dhcp, dhcp-relay, fireinfo, firewall, ipsec, localnet, network, ntp, rc, squid, sshd, static-routes,suricata,unbound.

None of this made a difference, only restarting the whole appliance brought it back online Sunday night at 10:00 PM.

Now, I like my job, but I don’t particularly want to be there 24/7 to babysit this ongoing problem.

If this situation comes up again, what other services do you suggest that I restart in order to get the system back up and going again without having to restart the whole router? In an Ideal world I would like to find the service that’s a problem, and then fix the issue so this stops happening, or at least set up a cron job to restart that service at regular intervals to avoid having to babysit the whole system all the time.

Any suggestions?

Chris

schami23 · 7 December 2022 11:47

schami23 · 8 December 2022 07:14

With the patch the client cannot connect to openVPN, get the message ‘push_request’ (status=1)

cwensink · 8 December 2022 15:32

Will this Patch be rolled into Core-Update 172?

jon · 19 December 2022 17:39

~~I think it is~~

~~This is the build info from the 17th:~~

~~See the 4th and 5th “commit” for the patch.~~

To be sure it works you may want to do some testing:

bonnietwin · 19 December 2022 18:12

No it won’t be in CU172. It is being removed due to report from @schami23 in the bug that the OTP aspect no longer worked with the patch in place although it does stop the memory and cpu consumption. So further work is still required on the bug.

See the following post in this thread for the bug link to check latest status
https://community.ipfire.org/t/ipfire-went-down-last-night-cant-find-cause/8570/52

jon · 19 December 2022 18:41

Ugh! my bad!!