IPFire went down last night, can't find cause

Networking Web Proxy
7

/var/log/logwatch/2022-09-14:

root@ipfire logwatch]# cat 2022-09-14
You have old files in your logwatch tmpdir (/var/cache/logwatch):
        logwatch.AjIhVCG1
The directories listed above were most likely created by a
logwatch run that failed to complete successfully.  If so, you
may delete these directories.

*** Error: There is no logfile defined. Do you have a /etc/logwatch/conf/logfiles/secure.conf file ?

 ################### Logwatch 7.6 (01/22/22) ####################
        Processing Initiated: Thu Sep 15 00:05:15 2022
        Date Range Processed: yesterday
                              ( 2022-Sep-14 )
                              Period is day.
        Detail Level of Output: 0
        Type of Output/Format: stdout / text
        Logfiles for Host: ipfire.internal.lan
 ##################################################################

 --------------------- dhcpd Begin ------------------------

 Late or spurious Echo Reply for:
    internal.50: 1 Time(s)
    internal.64: 1 Time(s)
    internal.73: 1 Time(s)
    internal.96: 1 Time(s)
    internal.99: 1 Time(s)
    blue.interface.2: 1 Time(s)

 Unknown Entries:
    5 udp packets in 5 too long - dropped: 1 Time(s)


 ---------------------- dhcpd End -------------------------


 --------------------- Kernel Begin ------------------------


 WARNING:  Out of memory killer killed these executables
    openvpn-authent :  1 Time(s)

 WARNING:  Kernel Errors Present
    Buffer I/O error on dev sda4, log ...:  1 Time(s)
    Buffer I/O error on device sda4,  ...:  30 Time(s)
    EXT4-fs (sda4): previous I/O error to superblock de ...:  1 Time(s)
    EXT4-fs error (device sda4): e ...:  1 Time(s)
    EXT4-fs warning (device sda4): ext4_end_bio:344: I/O error 10 writing to in ...:  7 Time(s)
    blk_update_request: I/O error, dev sda, sector ...:  15 Time(s)
    buffer_io_error: 501 callbacks s ...:  1 Time(s)
    buffer_io_error: 733 callbacks s ...:  1 Time(s)

 ---------------------- Kernel End -------------------------


 --------------------- SSHD Begin ------------------------


 SSHD Killed: 2 Times

 SSHD Started: 3 Times

 Users logging in through sshd:
    root:
       <internal.ip> (me.lan.address): 1 Time

 ---------------------- SSHD End -------------------------


 --------------------- Disk Space Begin ------------------------

 Filesystem      Size  Used Avail Use% Mounted on
 /dev/sda4       116G   16G   95G  14% /
 /dev/sda1       110M   48M   54M  48% /boot
 /dev/sda2        32M  270K   32M   1% /boot/efi


 ---------------------- Disk Space End -------------------------


 --------------------- lm_sensors output Begin ------------------------

 coretemp-isa-0000
 Adapter: ISA adapter
 Core 0:       +46.0 C  (high = +90.0 C, crit = +90.0 C)
 Core 1:       +46.0 C  (high = +90.0 C, crit = +90.0 C)
 Core 2:       +44.0 C  (high = +90.0 C, crit = +90.0 C)
 Core 3:       +44.0 C  (high = +90.0 C, crit = +90.0 C)


 ---------------------- lm_sensors output End -------------------------


 ###################### Logwatch End #########################

[root@ipfire logwatch]#
8

Your logs for the red interface don’t show any problem there.

Your openvpn authentication executable was killed because your system ran out of memory. What does the Memory WUI menu item show for the usage and especially around the time that you had the problem. Either you are running very close to using all the memory up on an ongoing basis, or some program executed and used up all the memory and had a higher priority than the openvpn authentication program.

There are also quite a lot of I/O error which doesn’t look good.
What does the SMART status from the WUI Status - Media menu item say about the disk drive.

Your summary log would make me look at potential hardware problems. That would also tie up with the collectd messages about many values being too old. It is not getting any new data and the cache data is too old My first thought on that was that there was a problem getting the required data for the rrdtool to put into the graphs.

The collectd error messages could also mean that you don’t have any memory data in the graphs for the time that the problem happened.

1 Like
9

There was hardly any memory in use for the time in question, yesterday, there was a spike today between 11 AM and 12 PM but nothing went down at the time:

image
image978×469 27.2 KB

image
image899×866 23.4 KB

image
image1010×888 39.8 KB

10

What do you see in the /var/log/messages file around the 7:50 pm time period that the problem happened.

11

That info is in my first post screenshot. /var/log/messages is a VERY large file: 638 MB, 2,543,435 lines.

Just the log info for Sep 14 for 1900 hours is 30,137 lines.

I’m happy to grep through the file and post output but I don’t precisely know what to look for. I may be looking for a needle in a warehouse full of needles.

12

Hi all,
having here the same log entries from the OOM killer log

Sep 15 19:29:50 ipfire kernel: openvpn-authent invoked oom-killer: gfp_mask=0x1100dca(GFP_HIGHUSER_MOVABLE|__GFP_ZERO), order=0, oom_score_adj=0
Sep 15 19:29:50 ipfire kernel: oom_kill_process.cold+0xb/0x10
Sep 15 19:29:50 ipfire kernel: [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Sep 15 19:29:50 ipfire kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=openvpn-authent,pid=12228,uid=0
Sep 15 19:29:50 ipfire kernel: Out of memory: Killed process 12228 (openvpn-authent) total-vm:1177692kB, anon-rss:854540kB, file-rss:1252kB, shmem-rss:0kB, UID:0 pgtables:2300kB oom_score_adj:0

Sep 15 19:48:30 ipfire kernel: Out of memory: Killed process 12614 (openvpn-authent) total-vm:1876516kB, anon-rss:1438396kB, file-rss:68kB, shmem-rss:0kB, UID:0 pgtables:3624kB oom_score_adj:0

and the memory graph at this time

memory_graph_oom
memory_graph_oom1446×450 40.3 KB

whereby it seems that ‘openvpn-authent’ process reaches the OOM scores limit and invokes the kernel OOM but this should not causes IPFire to get down since the highest scored processes will be suspended/killed…

Nevertheless the OOM seems a bit strange to me ???

Best,

Erik

1 Like
13

That is a big log file.

Looking at your first screenshot, at 19:50:37 something has happened that causes unbound to be unable to get any responses from any of the DNS servers.

None of the entries immediately above the unbound errors would be expected to stop IPFire from working.
The only suggestion I have is to look at the log entries from 19:50:27 back to something like 19:50:25 or 19:50:22 and see if there are any entries that look unusual (not normal firewall drops or forwards).

If there is nothing within a 2 sec to 5 sec max window then anything earlier is unlikely to be relevant to the stoppage. I would expect problems to show in the log within a quite short period of time to unbound showing the errors.

If there is nothing in the logs when you scan them then we are running out of ideas. If it happens again (hopefully not but just in case) then before restarting IPFire check from the console if RED is connected to the ISP or not.As your RED log showed no problems in it for the involved time then that means either that losing the RED connection was not the problem or something happened so quickly with RED that nothing was able to be logged in time.

Sorry I can’t think of anything further at the moment. Maybe others have further ideas.

1 Like
14

I would definitely agree with you Erik.

I have checked my memory graph and I don’t see any peak like that at all over the whole of the last year. Currently running on CU169. The used memory has varied between 8% and a max of 27%

I did a grep in my messages log for “Out of memory” and nothing showed up for over the last 5 days.

15

Hi Adolf,
have yesterday discovered it by checking some OpenVPN development and debugging. Since i run currently OpenVPN-2.6.dev version i first wasn´t sure where it comes from but since the log entries appears also at @cwensink it seems that it has nothing to do with the OpenVPN dev binary. Have checked (clicked around) the OpenVPN WUI and there it seems somehow appears the OOM by calling /usr/sbin/openvpn-authentication.
Checked via script → 

#!/bin/bash
# Displays running processes in descending order of OOM score
printf 'PID\tOOM Score\tOOM Adj\tCommand\n'
while read -r pid comm; do
	[ -f /proc/$pid/oom_score ] && [ $(cat /proc/$pid/oom_score) != 0 ] && printf '%d\t%d\t\t%d\t%s\n' "$pid" "$(cat /proc/$pid/oom_score)" "$(cat /proc/$pid/oom_score_adj)" "$comm";
done < <(ps -e -o pid= -o comm=) | sort -k 2nr

# EOF

the OOM scores of the processes and there openvpn-authent appears at the very top → 

PID	OOM Score	OOM Adj	Command
5272	981		0	openvpn-authent
6830	810		0	openvpn-authent
2018	772		0	Suricata-Main
3195	676		0	pmacctd
5743	673		0	tor
1619	670		0	unbound
2327	669		0	squid
3187	669		0	pmacctd
11963	668		0	httpd
12839	668		0	collectd
4444	668		0	unbound-dhcp-le
7013	668		0	openvpn-authent
7934	668		0	zsh
19707	667		0	haproxy
3240	667		0	snmpd
4406	667		0	dhcpd
8146	667		0	oom_scores.sh
8147	667		0	oom_scores.sh
11958	666		0	httpd
1570	666		0	vnstatd
1630	666		0	acpid
1662	666		0	dbus-daemon
1679	666		0	hostapd
2325	666		0	squid
2472	666		0	saslauthd
2473	666		0	saslauthd
2484	666		0	cupsd
2538	666		0	ntpd
2654	666		0	httpd
2704	666		0	fcron
2715	666		0	nginx
2716	666		0	nginx
3227	666		0	avahi-daemon
3228	666		0	avahi-daemon
3445	666		0	agetty
3446	666		0	agetty
3447	666		0	agetty
3448	666		0	agetty
3449	666		0	agetty
3450	666		0	agetty
5424	666		0	dma
5432	666		0	dma
5445	666		0	dma
5447	666		0	dma
5458	666		0	dma
5479	666		0	dma
5483	666		0	dma
5485	666		0	dma
587	666		0	lvmetad
6977	666		0	openvpn
7932	666		0	sshd
8148	666		0	sort
8949	666		0	dma
8951	666		0	dma
8952	666		0	dma
8954	666		0	dma
8955	666		0	dma
8957	666		0	dma
8958	666		0	dma
8959	666		0	dma
8960	666		0	dma
8961	666		0	dma
8963	666		0	dma
8964	666		0	dma
8965	666		0	dma
8989	666		0	dma
9013	666		0	dma
9016	666		0	dma
9017	666		0	dma
9022	666		0	dma
9025	666		0	dma
9027	666		0	dma
9028	666		0	dma
9029	666		0	dma
9037	666		0	dma
9038	666		0	dma
9039	666		0	dma
9040	666		0	dma
9048	666		0	dma
9049	666		0	dma
9055	666		0	dma
2640	656		-16	sshd

may a possible memory leak ???

But this seems to be another topic won´t make here an OT…

Best,

Erik

P.S.: Have also Core 169 running…

3 Likes
16

I have a very similar problem, but in my case not only OpenVPN is terminated but also clamAV and the Tor Node, the OpenVPN server restarts automatically, the Virus Scanner and the Tor Node do not.
Why is the memory running so full?

By the way, I noticed that I also have an error with unbound DNS, which I noticed in connection with DDNS, maybe this is related.

17

Everyone,

Well, problems continue here with no known cause. As suggested when the system went down, I tried just restarting the red interface, nothing. I restarted unbound, nothing. I restarted the proxy: nothing. Only restarting the whole router brings things back online.

I had to reset on Friday, Saturday, Sunday, and 4 times today already in the middle of the production day. Nothing appears in /var/log/messages other than a whole lot of DNS unbound connections timing out.

I am seeing these unusual messages:

14:50:46 kernel: ll header: 00000000: ff ff ff ff ff ff 9c 8e 99 c1 b4 80 08 06
14:50:46 kernel: IPv4: martian source 10.1.10.1 from 10.1.10.250, on dev green0
14:49:37 kernel: ll header: 00000000: ff ff ff ff ff ff 9c 8e 99 c1 b4 80 08 06
14:49:37 kernel: IPv4: martian source 10.1.10.1 from 10.1.10.250, on dev green0
14:48:28 kernel: ll header: 00000000: ff ff ff ff ff ff 9c 8e 99 c1 b4 80 08 06
14:48:28 kernel: IPv4: martian source 10.1.10.1 from 10.1.10.250, on dev green0
14:47:19 kernel: ll header: 00000000: ff ff ff ff ff ff 9c 8e 99 c1 b4 80 08 06
14:47:19 kernel: IPv4: martian source 10.1.10.1 from 10.1.10.250, on dev green0
14:46:10 kernel: ll header: 00000000: ff ff ff ff ff ff 9c 8e 99 c1 b4 80 08 06
14:46:10 kernel: IPv4: martian source 10.1.10.1 from 10.1.10.250, on dev green0

Our IP Scheme is 10.5.1.X, 10.1.10.X does match the IP scheme of another offsite location, but that’s connected via IPSec point to point tunnel and I’m seeing these in the logs even when the tunnel is down.

Another strange thing that happened. I saw bursts of traffic from one IP address while looking at a tail -f /var/log/messages this morning and I was seeing bursts of messages from one particular IP address, which happened to be the HR Manager’s business iPhone. I had him turn it off after discovering this, then just before lunch he asked if he could turn it back on, he did and then within 30 seconds the network went down again and I had to restart.

So, I factory reset and erased all settings on the iPhone, then updated versions of iOS.

I have not that phone back on the company WiFi yet. I will let it back on after 5 PM to see what the result is.

I also updated to 2.27 Core update 170, but I still had to reset with 170 installed.

I am setting up a complete replacement IPFire router now, on different hardware in case all of this is hardware related, if anyone has any other thoughts, let me know.

18

Here are the logs for about the last 24 hours, filtering for grep for every error, filtering out all unbound messages.

[root@ipfire log]# cat messages | grep "error" | grep -v "unbound"
Sep 18 00:01:48 ipfire openvpnserver[3244]: 200.95.174.12:80 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 00:02:50 ipfire openvpnserver[3244]: 200.95.174.12:80 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 00:03:52 ipfire openvpnserver[3244]: 200.95.174.12:80 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 00:04:54 ipfire openvpnserver[3244]: 200.95.174.12:80 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 00:05:55 ipfire openvpnserver[3244]: 200.95.174.12:80 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 00:06:57 ipfire openvpnserver[3244]: 200.95.174.12:80 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 03:49:10 ipfire pakfire: Giving up: There was no chance to get the file 2.27-x86_64/lists/server-list.db from any available server. There was an error on the way. Please fix it.
Sep 18 03:49:10 ipfire pakfire: Giving up: There was no chance to get the file lists/packages_list.db from any available server. There was an error on the way. Please fix it.
Sep 18 03:49:10 ipfire pakfire: Giving up: There was no chance to get the file lists/core-list.db from any available server. There was an error on the way. Please fix it.
Sep 18 18:14:39 ipfire openvpnserver[3244]: ERROR: Linux route delete command failed: external program exited with error status: 2
Sep 18 18:14:39 ipfire openvpnserver[3244]: Linux ip addr del failed: external program exited with error status: 2
Sep 18 18:19:06 ipfire openvpnserver[3243]: 20.42.201.67:30108 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:19:29 ipfire openvpnserver[3243]: 138.59.208.13:80 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:20:08 ipfire openvpnserver[3243]: 20.42.201.67:30108 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:20:30 ipfire openvpnserver[3243]: 138.59.208.13:80 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:21:10 ipfire openvpnserver[3243]: 20.42.201.67:30108 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:21:32 ipfire openvpnserver[3243]: 138.59.208.13:80 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:22:12 ipfire openvpnserver[3243]: 20.42.201.67:30108 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:22:34 ipfire openvpnserver[3243]: 138.59.208.13:80 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:23:14 ipfire openvpnserver[3243]: 20.42.201.67:30108 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:23:36 ipfire openvpnserver[3243]: 138.59.208.13:80 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:24:16 ipfire openvpnserver[3243]: 20.42.201.67:30108 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:24:38 ipfire openvpnserver[3243]: 138.59.208.13:80 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:25:19 ipfire openvpnserver[3243]: 20.42.201.67:30108 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:25:40 ipfire openvpnserver[3243]: 138.59.208.13:80 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:26:20 ipfire openvpnserver[3243]: 20.42.201.67:30108 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:26:42 ipfire openvpnserver[3243]: 138.59.208.13:80 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:27:22 ipfire openvpnserver[3243]: 20.42.201.67:30108 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:27:44 ipfire openvpnserver[3243]: 138.59.208.13:80 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:28:23 ipfire openvpnserver[3243]: 20.42.201.67:30108 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:28:46 ipfire openvpnserver[3243]: 138.59.208.13:80 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:29:10 ipfire openvpnserver[3243]: 20.42.201.67:30214 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:29:48 ipfire openvpnserver[3243]: 138.59.208.13:80 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:30:12 ipfire openvpnserver[3243]: 20.42.201.67:30214 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:30:50 ipfire openvpnserver[3243]: 138.59.208.13:80 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:31:14 ipfire openvpnserver[3243]: 20.42.201.67:30214 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:31:30 ipfire openvpnserver[3243]: 205.178.182.243:1194 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:31:51 ipfire openvpnserver[3243]: 138.59.208.13:80 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:32:16 ipfire openvpnserver[3243]: 20.42.201.67:30214 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:32:35 ipfire openvpnserver[3243]: 205.178.182.243:1194 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:32:53 ipfire openvpnserver[3243]: 138.59.208.13:80 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:33:18 ipfire openvpnserver[3243]: 20.42.201.67:30214 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:33:39 ipfire openvpnserver[3243]: 205.178.182.243:1194 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:33:56 ipfire openvpnserver[3243]: 138.59.208.13:80 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:34:19 ipfire openvpnserver[3243]: 20.42.201.67:30214 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:34:48 ipfire openvpnserver[3243]: 205.178.182.243:1194 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:35:26 ipfire openvpnserver[3243]: 20.42.201.67:30025 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:35:48 ipfire openvpnserver[3243]: 205.178.182.243:1194 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:36:29 ipfire openvpnserver[3243]: 20.42.201.67:30025 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:36:48 ipfire openvpnserver[3243]: 205.178.182.243:1194 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:37:30 ipfire openvpnserver[3243]: 20.42.201.67:30025 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:37:51 ipfire openvpnserver[3243]: 205.178.182.243:1194 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:38:33 ipfire openvpnserver[3243]: 20.42.201.67:30025 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:38:54 ipfire openvpnserver[3243]: 205.178.182.243:1194 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:39:34 ipfire openvpnserver[3243]: 20.42.201.67:30025 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:39:57 ipfire openvpnserver[3243]: 205.178.182.243:1194 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:40:36 ipfire openvpnserver[3243]: 20.42.201.67:30025 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:40:42 ipfire openvpnserver[3243]: 20.42.201.67:30234 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:41:00 ipfire openvpnserver[3243]: 205.178.182.243:1194 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:41:44 ipfire openvpnserver[3243]: 20.42.201.67:30234 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:42:04 ipfire openvpnserver[3243]: 205.178.182.243:1194 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:42:45 ipfire openvpnserver[3243]: 20.42.201.67:30234 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:43:05 ipfire openvpnserver[3243]: 205.178.182.243:1194 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:43:35 ipfire openvpnserver[3243]: 20.42.201.67:30110 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:44:09 ipfire openvpnserver[3243]: 205.178.182.243:1194 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:44:38 ipfire openvpnserver[3243]: 20.42.201.67:30110 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:45:11 ipfire openvpnserver[3243]: 205.178.182.243:1194 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:45:40 ipfire openvpnserver[3243]: 20.42.201.67:30110 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:46:06 ipfire openvpnserver[3243]: 20.42.201.67:30102 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:46:15 ipfire openvpnserver[3243]: 205.178.182.243:1194 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:46:42 ipfire openvpnserver[3243]: 20.42.201.67:30110 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:47:08 ipfire openvpnserver[3243]: 20.42.201.67:30102 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:47:18 ipfire openvpnserver[3243]: 205.178.182.243:1194 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:47:44 ipfire openvpnserver[3243]: 20.42.201.67:30110 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:48:10 ipfire openvpnserver[3243]: 20.42.201.67:30102 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:48:19 ipfire openvpnserver[3243]: 205.178.182.243:1194 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:48:46 ipfire openvpnserver[3243]: 20.42.201.67:30110 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:49:12 ipfire openvpnserver[3243]: 20.42.201.67:30102 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:49:24 ipfire openvpnserver[3243]: 205.178.182.243:1194 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:49:48 ipfire openvpnserver[3243]: 20.42.201.67:30110 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:50:14 ipfire openvpnserver[3243]: 20.42.201.67:30102 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:50:26 ipfire openvpnserver[3243]: 205.178.182.243:1194 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:50:50 ipfire openvpnserver[3243]: 20.42.201.67:30110 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:51:15 ipfire openvpnserver[3243]: 20.42.201.67:30102 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:51:27 ipfire openvpnserver[3243]: 205.178.182.243:1194 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:51:52 ipfire openvpnserver[3243]: 20.42.201.67:30110 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:52:17 ipfire openvpnserver[3243]: 20.42.201.67:30102 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:52:54 ipfire openvpnserver[3243]: 20.42.201.67:30110 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:53:19 ipfire openvpnserver[3243]: 20.42.201.67:30102 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:53:56 ipfire openvpnserver[3243]: 20.42.201.67:30110 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:54:38 ipfire openvpnserver[3243]: 20.42.201.67:30213 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:55:39 ipfire openvpnserver[3243]: 20.42.201.67:30213 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:56:40 ipfire openvpnserver[3243]: 20.42.201.67:30233 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:56:41 ipfire openvpnserver[3243]: 20.42.201.67:30213 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:57:42 ipfire openvpnserver[3243]: 20.42.201.67:30233 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:57:44 ipfire openvpnserver[3243]: 20.42.201.67:30213 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:58:44 ipfire openvpnserver[3243]: 20.42.201.67:30233 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:58:45 ipfire openvpnserver[3243]: 20.42.201.67:30213 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:59:46 ipfire openvpnserver[3243]: 20.42.201.67:30233 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 18:59:47 ipfire openvpnserver[3243]: 20.42.201.67:30213 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:00:48 ipfire openvpnserver[3243]: 20.42.201.67:30233 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:00:49 ipfire openvpnserver[3243]: 20.42.201.67:30213 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:01:50 ipfire openvpnserver[3243]: 20.42.201.67:30233 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:01:51 ipfire openvpnserver[3243]: 20.42.201.67:30213 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:02:52 ipfire openvpnserver[3243]: 20.42.201.67:30233 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:02:53 ipfire openvpnserver[3243]: 20.42.201.67:30213 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:03:54 ipfire openvpnserver[3243]: 20.42.201.67:30233 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:03:56 ipfire openvpnserver[3243]: 20.42.201.67:30213 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:04:55 ipfire openvpnserver[3243]: 20.42.201.67:30233 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:04:57 ipfire openvpnserver[3243]: 20.42.201.67:30213 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:05:57 ipfire openvpnserver[3243]: 20.42.201.67:30233 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:05:59 ipfire openvpnserver[3243]: 20.42.201.67:30213 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:06:59 ipfire openvpnserver[3243]: 20.42.201.67:30233 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:07:02 ipfire openvpnserver[3243]: 20.42.201.67:30213 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:08:01 ipfire openvpnserver[3243]: 20.42.201.67:30233 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:08:02 ipfire openvpnserver[3243]: 20.42.201.67:30213 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:09:03 ipfire openvpnserver[3243]: 20.42.201.67:30233 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:09:05 ipfire openvpnserver[3243]: 20.42.201.67:30213 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:09:32 ipfire openvpnserver[3243]: 20.42.201.67:30220 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:10:34 ipfire openvpnserver[3243]: 20.42.201.67:30220 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:11:36 ipfire openvpnserver[3243]: 20.42.201.67:30220 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:12:38 ipfire openvpnserver[3243]: 20.42.201.67:30220 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:13:40 ipfire openvpnserver[3243]: 20.42.201.67:30220 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:14:43 ipfire openvpnserver[3243]: 20.42.201.67:30220 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:15:44 ipfire openvpnserver[3243]: 20.42.201.67:30220 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:16:46 ipfire openvpnserver[3243]: 20.42.201.67:30220 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:17:48 ipfire openvpnserver[3243]: 20.42.201.67:30220 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:18:50 ipfire openvpnserver[3243]: 20.42.201.67:30220 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:19:51 ipfire openvpnserver[3243]: 20.42.201.67:30220 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:20:54 ipfire openvpnserver[3243]: 20.42.201.67:30220 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:21:55 ipfire openvpnserver[3243]: 20.42.201.67:30220 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:22:57 ipfire openvpnserver[3243]: 20.42.201.67:30220 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:23:59 ipfire openvpnserver[3243]: 20.42.201.67:30220 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:34:42 ipfire openvpnserver[3243]: 20.42.201.67:30115 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:35:44 ipfire openvpnserver[3243]: 20.42.201.67:30115 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:36:46 ipfire openvpnserver[3243]: 20.42.201.67:30115 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:37:48 ipfire openvpnserver[3243]: 20.42.201.67:30115 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:38:49 ipfire openvpnserver[3243]: 20.42.201.67:30115 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:39:51 ipfire openvpnserver[3243]: 20.42.201.67:30115 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:40:54 ipfire openvpnserver[3243]: 20.42.201.67:30115 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:41:56 ipfire openvpnserver[3243]: 20.42.201.67:30115 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:42:57 ipfire openvpnserver[3243]: 20.42.201.67:30115 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:43:59 ipfire openvpnserver[3243]: 20.42.201.67:30115 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:45:01 ipfire openvpnserver[3243]: 20.42.201.67:30115 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:46:03 ipfire openvpnserver[3243]: 20.42.201.67:30115 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:47:05 ipfire openvpnserver[3243]: 20.42.201.67:30115 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:48:07 ipfire openvpnserver[3243]: 20.42.201.67:30115 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 19:49:09 ipfire openvpnserver[3243]: 20.42.201.67:30115 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 20:20:23 ipfire openvpnserver[3243]: 193.42.11.190:22003 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 20:45:14 ipfire openvpnserver[3243]: 98.165.45.33:60352 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 20:45:30 ipfire openvpnserver[3243]: 98.165.45.33:11452 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 20:45:47 ipfire openvpnserver[3243]: 98.165.45.33:58592 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 20:46:04 ipfire openvpnserver[3243]: 98.165.45.33:19610 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 20:46:19 ipfire openvpnserver[3243]: 98.165.45.33:31406 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 20:46:36 ipfire openvpnserver[3243]: 98.165.45.33:62774 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 20:51:14 ipfire openvpnserver[3243]: 98.165.45.33:18499 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 20:51:31 ipfire openvpnserver[3243]: 98.165.45.33:37273 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 20:51:47 ipfire openvpnserver[3243]: 98.165.45.33:13885 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 20:52:03 ipfire openvpnserver[3243]: 98.165.45.33:11778 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 20:52:20 ipfire openvpnserver[3243]: 98.165.45.33:36554 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 20:52:36 ipfire openvpnserver[3243]: 98.165.45.33:47129 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 20:52:53 ipfire openvpnserver[3243]: 98.165.45.33:17459 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 20:53:09 ipfire openvpnserver[3243]: 98.165.45.33:24850 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 20:53:26 ipfire openvpnserver[3243]: 98.165.45.33:51007 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 20:53:43 ipfire openvpnserver[3243]: 98.165.45.33:62115 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 20:53:59 ipfire openvpnserver[3243]: 98.165.45.33:25498 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 20:54:15 ipfire openvpnserver[3243]: 98.165.45.33:10191 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 20:54:31 ipfire openvpnserver[3243]: 98.165.45.33:31685 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 20:54:48 ipfire openvpnserver[3243]: 98.165.45.33:16164 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 20:55:04 ipfire openvpnserver[3243]: 98.165.45.33:50565 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 20:55:21 ipfire openvpnserver[3243]: 98.165.45.33:2286 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 20:55:37 ipfire openvpnserver[3243]: 98.165.45.33:24664 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 20:55:54 ipfire openvpnserver[3243]: 98.165.45.33:12861 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 20:56:35 ipfire openvpnserver[3243]: 98.165.45.33:59822 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 20:56:52 ipfire openvpnserver[3243]: 98.165.45.33:198 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 20:57:08 ipfire openvpnserver[3243]: 98.165.45.33:30452 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 20:57:24 ipfire openvpnserver[3243]: 98.165.45.33:12279 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 20:57:42 ipfire openvpnserver[3243]: 98.165.45.33:47272 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 20:57:58 ipfire openvpnserver[3243]: 98.165.45.33:34154 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 20:58:15 ipfire openvpnserver[3243]: 98.165.45.33:45420 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 20:58:30 ipfire openvpnserver[3243]: 98.165.45.33:59599 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 20:58:47 ipfire openvpnserver[3243]: 98.165.45.33:17750 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 20:59:03 ipfire openvpnserver[3243]: 98.165.45.33:44496 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 20:59:20 ipfire openvpnserver[3243]: 98.165.45.33:45520 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 20:59:36 ipfire openvpnserver[3243]: 98.165.45.33:56239 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 20:59:52 ipfire openvpnserver[3243]: 98.165.45.33:62981 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:00:09 ipfire openvpnserver[3243]: 98.165.45.33:47382 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:00:25 ipfire openvpnserver[3243]: 98.165.45.33:11457 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:00:43 ipfire openvpnserver[3243]: 98.165.45.33:50502 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:00:58 ipfire openvpnserver[3243]: 98.165.45.33:3146 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:01:15 ipfire openvpnserver[3243]: 98.165.45.33:28445 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:04:57 ipfire openvpnserver[3243]: 98.165.45.33:60962 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:05:14 ipfire openvpnserver[3243]: 98.165.45.33:16295 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:05:30 ipfire openvpnserver[3243]: 98.165.45.33:17740 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:05:47 ipfire openvpnserver[3243]: 98.165.45.33:19062 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:06:03 ipfire openvpnserver[3243]: 98.165.45.33:20865 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:06:19 ipfire openvpnserver[3243]: 98.165.45.33:18150 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:06:36 ipfire openvpnserver[3243]: 98.165.45.33:33251 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:06:52 ipfire openvpnserver[3243]: 98.165.45.33:11056 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:07:09 ipfire openvpnserver[3243]: 98.165.45.33:37646 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:07:25 ipfire openvpnserver[3243]: 98.165.45.33:42296 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:07:42 ipfire openvpnserver[3243]: 98.165.45.33:57712 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:07:58 ipfire openvpnserver[3243]: 98.165.45.33:64464 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:08:15 ipfire openvpnserver[3243]: 98.165.45.33:17680 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:08:31 ipfire openvpnserver[3243]: 98.165.45.33:43938 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:08:47 ipfire openvpnserver[3243]: 98.165.45.33:53936 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:09:05 ipfire openvpnserver[3243]: 98.165.45.33:55465 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:09:20 ipfire openvpnserver[3243]: 98.165.45.33:4318 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:09:37 ipfire openvpnserver[3243]: 98.165.45.33:30816 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:09:54 ipfire openvpnserver[3243]: 98.165.45.33:41119 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:17:48 ipfire openvpnserver[3243]: 46.105.91.172:80 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:19:52 ipfire openvpnserver[3243]: 98.165.45.33:26043 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:20:09 ipfire openvpnserver[3243]: 98.165.45.33:14863 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:20:25 ipfire openvpnserver[3243]: 98.165.45.33:55887 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:20:41 ipfire openvpnserver[3243]: 98.165.45.33:37051 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:20:57 ipfire openvpnserver[3243]: 98.165.45.33:38252 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:21:14 ipfire openvpnserver[3243]: 98.165.45.33:17330 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:21:30 ipfire openvpnserver[3243]: 98.165.45.33:50384 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:21:46 ipfire openvpnserver[3243]: 98.165.45.33:37890 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:22:03 ipfire openvpnserver[3243]: 98.165.45.33:28761 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:22:20 ipfire openvpnserver[3243]: 98.165.45.33:30904 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:22:36 ipfire openvpnserver[3243]: 98.165.45.33:63368 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:22:52 ipfire openvpnserver[3243]: 98.165.45.33:3840 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:23:09 ipfire openvpnserver[3243]: 98.165.45.33:20145 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:23:25 ipfire openvpnserver[3243]: 98.165.45.33:37720 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:23:42 ipfire openvpnserver[3243]: 98.165.45.33:38368 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:23:59 ipfire openvpnserver[3243]: 98.165.45.33:5489 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:24:14 ipfire openvpnserver[3243]: 98.165.45.33:1342 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 21:24:31 ipfire openvpnserver[3243]: 98.165.45.33:28635 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 22:25:58 ipfire openvpnserver[3243]: 146.88.240.4:34169 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 22:36:03 ipfire openvpnserver[3243]: 187.157.9.79:80 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 23:08:12 ipfire openvpnserver[3243]: 20.42.201.67:30136 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 23:09:14 ipfire openvpnserver[3243]: 20.42.201.67:30136 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 23:10:16 ipfire openvpnserver[3243]: 20.42.201.67:30136 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 23:11:18 ipfire openvpnserver[3243]: 20.42.201.67:30136 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 23:12:20 ipfire openvpnserver[3243]: 20.42.201.67:30136 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 23:13:22 ipfire openvpnserver[3243]: 20.42.201.67:30136 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 18 23:14:24 ipfire openvpnserver[3243]: 20.42.201.67:30136 SIGUSR1[soft,tls-error] received, client-instance restarting

Sep 19 00:47:42 ipfire openvpnserver[3295]: 98.165.45.33:26377 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 19 00:47:59 ipfire openvpnserver[3295]: 98.165.45.33:31836 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 19 00:48:15 ipfire openvpnserver[3295]: 98.165.45.33:30567 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 19 00:48:32 ipfire openvpnserver[3295]: 98.165.45.33:47691 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 19 00:48:49 ipfire openvpnserver[3295]: 98.165.45.33:35090 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 19 00:49:05 ipfire openvpnserver[3295]: 98.165.45.33:59484 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 19 00:49:21 ipfire openvpnserver[3295]: 98.165.45.33:25550 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 19 08:19:34 ipfire openvpnserver[3308]: 2.48.75.8:43190 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 19 08:38:17 ipfire openvpnserver[3308]: 198.54.114.247:443 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 19 08:44:37 ipfire openvpnserver[3308]: 198.54.114.247:443 SIGUSR1[soft,tls-error] received, client-instance restarting
Sep 19 09:02:19 ipfire charon: 06[IKE] received NO_PROPOSAL_CHOSEN notify error
Sep 19 10:23:26 ipfire charon: 13[IKE] received NO_PROPOSAL_CHOSEN notify error
Sep 19 12:33:16 ipfire openvpnserver[3329]: ERROR: Linux route delete command failed: external program exited with error status: 2
Sep 19 12:33:16 ipfire openvpnserver[3329]: Linux ip addr del failed: external program exited with error status: 2
Sep 19 13:24:26 ipfire openvpnserver[6319]: ERROR: Linux route delete command failed: external program exited with error status: 2
Sep 19 13:24:26 ipfire openvpnserver[6319]: Linux ip addr del failed: external program exited with error status: 2
Sep 19 13:52:38 ipfire openvpnserver[6354]: 139.162.73.176:41737 SIGUSR1[soft,tls-error] received, client-instance restarting

There are all of these errors relating to openvpnserver, I don’t recognize these IP’s, could this be a DDOS attach on openvpn until the machine runs out of memory and crashes?

Chris

19

Hmmm,

Race condition perhaps.

BR
Joe.

20

How do you fix a race condition?

21

Hi all,
@cwensink i would check your OpenVPN logs not only by grepping for errors, may you can find there more informations. Nevertheless, there seems to be something really wrong with your OpenVPN configuration possibly you can also find this messages in the OpenVPN logs " TLS Error: TLS key negotiation failed to occur within 60 seconds", since this message appears very frequent, OpenVPN have decided to create an own page/checklist for this problem → https://openvpn.net/faq/tls-error-tls-key-negotiation-failed-to-occur-within-60-seconds-check-your-network-connectivity/ may you know it and or checked it ?

Another thing, it might be great if you can set the code tags if you post logs (easier to read).

Some ideas to one problem. May it might be also helpful if you sort the different problems (OOM, OpenVPN, Unbound, Smart status) a little more for your investigation but also for the community may some help comes faster ?

Best,

Erik

1 Like
22

Not sure if it helps.
I have had no problem with core 169
Core update did add 2factor authentication to OPENvpn.
A feature I’m not using.
Not a tor addon user either as a side note.

23

How long has this appliance been in service and were any updates or changes made, no matter how minor, before the first crash instance?
What version and core update are you running?

The most worrying thing I’ve seen in this entire post are the drive IO errors. If the drive is failing, it will cause all sorts of random errors. Same can be said for memory though. I get SMART says the drive is healthy, but it’s not entirely reliable. Could be a physical issue as well like a loose power or data cable, maybe a failing port.
Not sure how long you can take this appliance out of service, but if no software changes were made, I would install a new drive, restore the latest backup file and run memtest.

1 Like
24

Tuesday night I re-built a new IPFire machine, and the next two days have been quiet, errors for openvpn ended on 9/20 per the messages log. It’s entirely possible that all of this had nothing to do with IPFire and was related to hardware failure or possibly a security breach on the router, but the router is offline now.

Any suggestions on what utilities to run to test the hardware of the device? The machine is a Quad Core Atom processor 8, GB ram, 120 GB ssd Micro ITX Supermicro box 2.7.Ghz, and I am ok with any testing app in an offline environment.

What apps do you guys like to use?

25

DisturbedDragon,

This unit has been in service for about 5 years, and has had a number of changes in the configuration along the way. It first started having these major problems when running 169, and for troubleshooting I tried updating to 170, but that did not fix the issue. On this appliance there’s a power plug with an external power supply that’s similar to a laptop except the plug screws in on the end (like a water hose).

Chris

26

With the new machine in place and working I suppose the issue is resolved.
The issue appears to be an acute one. Working fine one day and then became crashy. I’m going with a hardware error again printing out the IO issues logged. You could still troubleshoot the old machine with the steps provided previously since it is now offline.
I will say that upgrading a machine experiencing issues is a bad idea. Updating the software could introduce a new bug compounding existing problems with new ones. Better to resolve any issues then upgrading.