System APU2c4
IPfire core 160
Connection type: red and green
ISP: comcast
DNS server status: working
Since updating to core 160, I randomly see IPS stop running and a restart will bring back the services. I am using Emerging threats community rules and set daily rule update. I am using ISP provided DNS I just wondering anyone run into something similar issue.
Hi @lowemissions!
How are you seeing this?
Is this in a log file or WebGUI screen?
Can you include a snippet of the log file when the issue occurs?
I checked the status running service WebGUI
where can I find the log to check for the log file?
While I try to get the IPS to run again, I also notice a hardware error but that did not affect the IPS service. It is running now. I think the hardware error could be related but not sure what the caused.
[root@ipfire /]# /etc/init.d/suricata restart
Stopping Intrusion Detection System…
/etc/rc.d/init.d/functions: line 366: k Not running.No such process [ WARN ]
Starting Intrusion Detection System… [ OK ]
[root@ipfire /]# /etc/init.d/suricata start
Starting Intrusion Detection System… [ FAIL ]
[root@ipfire /]# /etc/init.d/suricata stop
Stopping Intrusion Detection System… [ OK ]
[root@ipfire /]# /etc/init.d/suricata start
Starting Intrusion Detection System… [ OK ]
[root@ipfire /]#
Message from syslogd@ipfire at Wed Nov 17 12:40:24 2021 …
ipfire kernel: [Hardware Error]: Corrected error, no action required.
Message from syslogd@ipfire at Wed Nov 17 12:40:24 2021 …
ipfire kernel: [Hardware Error]: CPU:0 (16:30:1) MC1_STATUS[-|CE|-|AddrV|-|-|-]: 0x9400000000000151
Message from syslogd@ipfire at Wed Nov 17 12:40:24 2021 …
ipfire kernel: [Hardware Error]: Error Addr: 0x00000000005529e0
Message from syslogd@ipfire at Wed Nov 17 12:40:24 2021 …
ipfire kernel: [Hardware Error]: MC1 Error: Data/tag array parity error for a tag hit.
Message from syslogd@ipfire at Wed Nov 17 12:40:24 2021 …
ipfire kernel: [Hardware Error]: cache level: L1, tx: INSN, mem-tx: IRD
Under the menu Logs > System Logs. Set the Section to Intrusion Prevention and click Update.
The hardware could cause an issue if related to the RED Interface / ethernet port.
This is where is gets over my head so hopefully someone else will lend a hand!
Here are the log
The odd part the IPS were running fine till the end of day on the 16 and it stop without any error or any logs. The last known time the IPS was still running is around 22:46 till I try to restart the service the following day.
Log
Total hits for log section suricata November 16, 2021: 27
Older Newer
Time Section
01:26:11 suricata: rule reload starting
01:26:48 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘HTTP.UncompressedFlash’ is checked bu t not set. Checked in 2016396 and 2 other sigs
01:26:48 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘ET.JS.Obfus.Func’ is checked but not set. Checked in 2017247 and 0 other sigs
01:26:48 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘ET.pdf.in.http’ is checked but not se t. Checked in 2017150 and 4 other sigs
01:26:48 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘ET.http.binary’ is checked but not se t. Checked in 2019421 and 5 other sigs
01:26:48 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘ET.http.javaclient’ is checked but no t set. Checked in 2017181 and 11 other sigs
01:26:48 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘et.WinHttpRequest’ is checked but not set. Checked in 2019823 and 0 other sigs
01:26:48 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘exe.no.referer’ is checked but not se t. Checked in 2020500 and 0 other sigs
01:26:48 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘et.JavaArchiveOrClass’ is checked but not set. Checked in 2017772 and 1 other sigs
01:26:48 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘et.IE7.NoRef.NoCookie’ is checked but not set. Checked in 2023671 and 9 other sigs
01:26:48 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘ET.gocd.auth’ is checked but not set. Checked in 2034333 and 0 other sigs
01:26:48 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘is_proto_irc’ is checked but not set. Checked in 2002029 and 4 other sigs
01:26:48 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘ET.http.javaclient.vulnerable’ is che cked but not set. Checked in 2013036 and 0 other sigs
01:26:48 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘ET.ELFDownload’ is checked but not se t. Checked in 2019896 and 0 other sigs
01:26:48 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘et.DocVBAProject’ is checked but not set. Checked in 2020170 and 0 other sigs
01:26:48 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘ET.MSSQL’ is checked but not set. Che cked in 2020569 and 0 other sigs
01:26:48 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘ET.wininet.UA’ is checked but not set . Checked in 2021312 and 0 other sigs
01:26:48 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘et.MS.XMLHTTP.ip.request’ is checked but not set. Checked in 2022050 and 1 other sigs
01:26:48 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘et.MS.XMLHTTP.no.exe.request’ is chec ked but not set. Checked in 2022053 and 0 other sigs
01:26:48 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘et.MCOFF’ is checked but not set. Che cked in 2022303 and 0 other sigs
01:26:48 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘et.MS.WinHttpRequest.no.exe.request’ is checked but not set. Checked in 2022653 and 0 other sigs
01:26:48 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘ET.armwget’ is checked but not set. C hecked in 2024242 and 0 other sigs
01:26:48 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘ET.smb.binary’ is checked but not set . Checked in 2027402 and 4 other sigs
01:26:48 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘ET.Socks5.OnionReq’ is checked but no t set. Checked in 2027704 and 0 other sigs
01:26:48 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘ET.autoit.ua’ is checked but not set. Checked in 2019165 and 0 other sigs
01:26:48 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘min.gethttp’ is checked but not set. Checked in 2023711 and 0 other sigs
01:30:24 suricata: rule reload complete
Log
Total hits for log section suricata November 17, 2021: 66
Older Newer
Time Section
11:24:16 suricata: This is Suricata version 5.0.7 RELEASE running in SYSTEM mode
11:24:16 suricata: [ERRCODE: SC_ERR_INITIALIZATION(45)] - pid file ‘/var/run/suricata.pid’ exists b ut appears stale. Make sure Suricata is not running and then remove /var/run/sur icata.pid. Aborting!
11:24:25 suricata: This is Suricata version 5.0.7 RELEASE running in SYSTEM mode
11:24:25 suricata: [ERRCODE: SC_ERR_INITIALIZATION(45)] - pid file ‘/var/run/suricata.pid’ exists b ut appears stale. Make sure Suricata is not running and then remove /var/run/sur icata.pid. Aborting!
12:33:28 suricata: This is Suricata version 5.0.7 RELEASE running in SYSTEM mode
12:33:29 suricata: [ERRCODE: SC_WARN_NO_STATS_LOGGERS(261)] - stats are enabled but no loggers are active
12:33:29 suricata: all 4 packet processing threads, 2 management threads initialized, engine starte d.
12:33:29 suricata: rule reload starting
12:33:41 suricata: This is Suricata version 5.0.7 RELEASE running in SYSTEM mode
12:33:42 suricata: [ERRCODE: SC_ERR_INITIALIZATION(45)] - pid file ‘/var/run/suricata.pid’ exists a nd Suricata appears to be running. Aborting!
12:34:06 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘ET.http.binary’ is checked but not se t. Checked in 2019421 and 5 other sigs
12:34:06 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘ET.http.javaclient’ is checked but no t set. Checked in 2017181 and 11 other sigs
12:34:06 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘et.JavaArchiveOrClass’ is checked but not set. Checked in 2017772 and 1 other sigs
12:34:06 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘ET.pdf.in.http’ is checked but not se t. Checked in 2017150 and 4 other sigs
12:34:06 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘et.IE7.NoRef.NoCookie’ is checked but not set. Checked in 2023671 and 9 other sigs
12:34:06 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘ET.gocd.auth’ is checked but not set. Checked in 2034333 and 0 other sigs
12:34:06 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘is_proto_irc’ is checked but not set. Checked in 2002029 and 4 other sigs
12:34:06 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘ET.http.javaclient.vulnerable’ is che cked but not set. Checked in 2013036 and 0 other sigs
12:34:06 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘ET.ELFDownload’ is checked but not se t. Checked in 2019896 and 0 other sigs
12:34:06 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘et.DocVBAProject’ is checked but not set. Checked in 2020170 and 0 other sigs
12:34:06 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘ET.MSSQL’ is checked but not set. Che cked in 2020569 and 0 other sigs
12:34:06 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘ET.wininet.UA’ is checked but not set . Checked in 2021312 and 0 other sigs
12:34:06 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘et.MS.XMLHTTP.ip.request’ is checked but not set. Checked in 2022050 and 1 other sigs
12:34:06 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘et.MS.XMLHTTP.no.exe.request’ is chec ked but not set. Checked in 2022053 and 0 other sigs
12:34:06 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘et.MCOFF’ is checked but not set. Che cked in 2022303 and 0 other sigs
12:34:06 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘et.MS.WinHttpRequest.no.exe.request’ is checked but not set. Checked in 2022653 and 0 other sigs
12:34:06 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘ET.armwget’ is checked but not set. C hecked in 2024242 and 0 other sigs
12:34:06 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘ET.smb.binary’ is checked but not set . Checked in 2027402 and 4 other sigs
12:34:06 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘ET.Socks5.OnionReq’ is checked but no t set. Checked in 2027704 and 0 other sigs
12:34:06 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘ET.autoit.ua’ is checked but not set. Checked in 2019165 and 0 other sigs
12:34:06 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘min.gethttp’ is checked but not set. Checked in 2023711 and 0 other sigs
12:34:06 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘et.WinHttpRequest’ is checked but not set. Checked in 2019823 and 0 other sigs
12:34:06 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘HTTP.UncompressedFlash’ is checked bu t not set. Checked in 2016396 and 2 other sigs
12:34:06 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘exe.no.referer’ is checked but not se t. Checked in 2020500 and 0 other sigs
12:34:06 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘ET.JS.Obfus.Func’ is checked but not set. Checked in 2017247 and 0 other sigs
12:34:26 suricata: This is Suricata version 5.0.7 RELEASE running in SYSTEM mode
12:34:27 suricata: [ERRCODE: SC_WARN_NO_STATS_LOGGERS(261)] - stats are enabled but no loggers are active
12:34:27 suricata: all 4 packet processing threads, 2 management threads initialized, engine starte d.
12:34:27 suricata: rule reload starting
12:35:13 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘ET.http.binary’ is checked but not se t. Checked in 2019421 and 5 other sigs
12:35:13 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘ET.http.javaclient’ is checked but no t set. Checked in 2017181 and 11 other sigs
12:35:13 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘et.JavaArchiveOrClass’ is checked but not set. Checked in 2017772 and 1 other sigs
12:35:13 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘ET.pdf.in.http’ is checked but not se t. Checked in 2017150 and 4 other sigs
12:35:13 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘et.IE7.NoRef.NoCookie’ is checked but not set. Checked in 2023671 and 9 other sigs
12:35:13 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘ET.gocd.auth’ is checked but not set. Checked in 2034333 and 0 other sigs
12:35:13 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘is_proto_irc’ is checked but not set. Checked in 2002029 and 4 other sigs
12:35:13 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘ET.http.javaclient.vulnerable’ is che cked but not set. Checked in 2013036 and 0 other sigs
12:35:13 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘ET.ELFDownload’ is checked but not se t. Checked in 2019896 and 0 other sigs
12:35:13 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘et.DocVBAProject’ is checked but not set. Checked in 2020170 and 0 other sigs
12:35:13 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘ET.MSSQL’ is checked but not set. Che cked in 2020569 and 0 other sigs
12:35:13 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘ET.wininet.UA’ is checked but not set . Checked in 2021312 and 0 other sigs
12:35:13 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘et.MS.XMLHTTP.ip.request’ is checked but not set. Checked in 2022050 and 1 other sigs
12:35:13 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘et.MS.XMLHTTP.no.exe.request’ is chec ked but not set. Checked in 2022053 and 0 other sigs
12:35:13 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘et.MCOFF’ is checked but not set. Che cked in 2022303 and 0 other sigs
12:35:13 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘et.MS.WinHttpRequest.no.exe.request’ is checked but not set. Checked in 2022653 and 0 other sigs
12:35:13 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘ET.armwget’ is checked but not set. C hecked in 2024242 and 0 other sigs
12:35:13 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘ET.smb.binary’ is checked but not set . Checked in 2027402 and 4 other sigs
12:35:13 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘ET.Socks5.OnionReq’ is checked but no t set. Checked in 2027704 and 0 other sigs
12:35:13 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘ET.autoit.ua’ is checked but not set. Checked in 2019165 and 0 other sigs
12:35:13 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘min.gethttp’ is checked but not set. Checked in 2023711 and 0 other sigs
12:35:13 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘et.WinHttpRequest’ is checked but not set. Checked in 2019823 and 0 other sigs
12:35:13 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘HTTP.UncompressedFlash’ is checked bu t not set. Checked in 2016396 and 2 other sigs
12:35:13 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘exe.no.referer’ is checked but not se t. Checked in 2020500 and 0 other sigs
12:35:13 suricata: [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit ‘ET.JS.Obfus.Func’ is checked but not set. Checked in 2017247 and 0 other sigs
12:39:20 suricata: rule reload complete
I found this log in the kernel section before IPS stop running.
| 22:41:11 | kernel: | W-NFQ#2[17607]: segfault at f4 ip 00000000000000f4 sp 0000796cbea7c7f0 error 14 in suricata[400000+9000] |
|---|---|---|
| 22:41:11 | kernel: | Code: Unable to access opcode bytes at RIP 0xca. |
Set all of these aside for the moment. There are rules enabled in IPS that are missing “something”. I don’t know or understand IPS well enough to assist but they are WARNings and not errors.
If we just look at the second Log above then I see this below
(I removed the ERRCODE: SC_WARN_FLOWBIT(306) warnings)
Log
Total hits for log section suricata November 17, 2021: 66
Older Newer
Time Section
11:24:16 suricata: This is Suricata version 5.0.7 RELEASE running in SYSTEM mode
11:24:16 suricata: [ERRCODE: SC_ERR_INITIALIZATION(45)] - pid file ‘/var/run/suricata.pid’ exists b ut appears stale. Make sure Suricata is not running and then remove /var/run/sur icata.pid. Aborting!
11:24:25 suricata: This is Suricata version 5.0.7 RELEASE running in SYSTEM mode
11:24:25 suricata: [ERRCODE: SC_ERR_INITIALIZATION(45)] - pid file ‘/var/run/suricata.pid’ exists b ut appears stale. Make sure Suricata is not running and then remove /var/run/sur icata.pid. Aborting!
12:33:28 suricata: This is Suricata version 5.0.7 RELEASE running in SYSTEM mode
12:33:29 suricata: [ERRCODE: SC_WARN_NO_STATS_LOGGERS(261)] - stats are enabled but no loggers are active
12:33:29 suricata: all 4 packet processing threads, 2 management threads initialized, engine starte d.
12:33:29 suricata: rule reload starting
12:33:41 suricata: This is Suricata version 5.0.7 RELEASE running in SYSTEM mode
12:33:42 suricata: [ERRCODE: SC_ERR_INITIALIZATION(45)] - pid file ‘/var/run/suricata.pid’ exists a nd Suricata appears to be running. Aborting!
. . .
12:34:26 suricata: This is Suricata version 5.0.7 RELEASE running in SYSTEM mode
12:34:27 suricata: [ERRCODE: SC_WARN_NO_STATS_LOGGERS(261)] - stats are enabled but no loggers are active
12:34:27 suricata: all 4 packet processing threads, 2 management threads initialized, engine starte d.
12:34:27 suricata: rule reload starting
. . .
12:39:20 suricata: rule reload complete
Again this is over my head. But it looks like Suricata is being run more than once.
I am new at this as well and if someone with more knowledge can explain that would be great. All I know that the suricata service pid appear stale so that prevent me from restart the IPS from webgui. I have checked multiple sources and found some suggested like removing suricata.pid and restart the service from the webgui. Instead of removing suricata.pid, I stop suricata which also remove the suricata pid and start suricata again. What cause the suricata pid to go stale?
this is the command line I use to stop and start suricata just incase someone may need it.
/etc/init.d/suricata stop
/etc/init.d/suricata start
This Hardware error is a memory fault. Run memtest86 from the bios and let it run for at least a day to check. If it faulty replace the apu.