I have strange behavior on a firewall, J1900 with 4GB ram, guardian and wio configuration R, G, O, B with core 168
I have a reboot schedule at 1am every day.
Sometimes the firewall does not restart.
I understand this from the graphs and from the fact that I have to restart it by hand.
Is there any way from the logs to tell if something hangs in the reboot phase?
Giuseppe
You need to dig in the system logs. I would use the console. SSH to your IPFire and use:
cat /var/log/messages | grep "ipfire shutdown"
Keep in mind that the logs are rotated periodically and the older ones are gzip archived. In that case you could use zcat instead of cat:
zcat /var/log/messages.1.gz | grep "ipfire shutdown"
the scheme of the archiving is messages.n.gz, where n is an incremental number, and the most recent archived log is messages.1.gz while the oldest is messages.49.gz
Once you identify the exact moment of the reboot and in which archive it was logged you can zoom in and see what happened next, by using -An and -Bn grep switches. Where -An shows n lines after the positive term found by grep and -Bn shows n lines before.
For example, in my case the last reboot was logged in messages.3.gz. If I want to see one line before the reboot, and 5 lines after I type:
zcat /var/log/messages.1.gz | grep -A5 -B1 "ipfire shutdown"
and this is what I get:
Jun 14 14:20:52 ipfire ipfire: Rebooting IPFire
Jun 14 14:20:52 ipfire shutdown[19165]: shutting down for system reboot
Jun 14 14:20:52 ipfire init: Switching to runlevel: 6
Jun 14 14:20:58 ipfire squid[18484]: Squid Parent: squid-1 process 18486 exited with status 0
Jun 14 14:20:59 ipfire squid: squid shutdown time: 6 seconds
Jun 14 14:21:00 ipfire kernel: DROP_INPUT IN=red0 OUT= MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=64.44.139.166 DST=cut LEN=40 TOS=0x00 PREC=0x00 TTL=231 ID=18384 PROTO=TCP SPT=54126 DPT=53033 WINDOW=1024 RES=0x00 SYN URGP=0
Jun 14 14:21:02 ipfire ntpd[20328]: ntpd exiting on signal 15 (Terminated)
--
Jun 14 14:26:59 ipfire kernel: DROP_INPUT IN=red0 OUT= MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=192.241.221.102 DST=cut LEN=40 TOS=0x00 PREC=0x00 TTL=233 ID=54321 PROTO=TCP SPT=44799 DPT=18245 WINDOW=65535 RES=0x00 SYN URGP=0
Jun 14 14:27:00 ipfire shutdown[20967]: shutting down for system reboot
Jun 14 14:27:00 ipfire init: Switching to runlevel: 6
Jun 14 14:27:00 ipfire kernel: DROP_HOSTILE IN=red0 OUT= MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=2.57.122.35 DST=cut LEN=40 TOS=0x00 PREC=0x00 TTL=237 ID=54321 PROTO=TCP SPT=42017 DPT=83 WINDOW=65535 RES=0x00 SYN URGP=0
Jun 14 14:27:06 ipfire kernel: DNAT IN=red0 OUT= MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=130.223.140.50 DST=cut LEN=64 TOS=0x00 PREC=0x00 TTL=52 ID=0 DF PROTO=TCP SPT=50640 DPT=443 WINDOW=65535 RES=0x00 SYN URGP=0
Jun 14 14:27:06 ipfire kernel: FORWARDFW IN=red0 OUT=orange0 MAC=00:0d:b9:42:68:92:00:00:f7:f7:02:56:08:00 SRC=130.223.140.50 DST=10.1.2.100 LEN=64 TOS=0x00 PREC=0x00 TTL=51 ID=0 DF PROTO=TCP SPT=50640 DPT=443 WINDOW=65535 RES=0x00 SYN URGP=0
Jun 14 14:27:06 ipfire kernel: IPv4: martian source 130.223.140.50 from 10.1.2.100, on dev green0
as you can see, the kernel keeps logging everything and therefore you have also events uncorrelated to what you are looking for and it might be necessary to zoom in more deeply, like using “-A20”.
A quick and dirty way to search all the archived logs in one go, is to use in the prompt of the console a one line shell script like this:
for LOG in /var/log/messages.*; do zcat $LOG | grep "ipfire shutdown"; done
There is more the one way to search the log. This is just how I do it.
Edit: I forgot, there is a very handy version of grep that directly reads a gzip file, zgrep. This simplify the command line by eliminating completely zcat. This is useful if you want to know which file contains the log of interest.
for LOG in /var/log/messages.*; do zgrep -l "ipfire shutdown" $LOG; done
If you want to see the matched line AND the file where it was matched, instead of -l switch you can use the -H swtich.
for example, in my system:
for LOG in /var/log/messages.*; do zgrep -H "ipfire shutdown" $LOG; done
gives me this (cut after third line):
/var/log/messages.10.gz:Apr 30 10:26:30 ipfire shutdown[29053]: shutting down for system reboot
/var/log/messages.11.gz:Apr 23 19:50:46 ipfire shutdown[24444]: shutting down for system reboot
/var/log/messages.12.gz:Apr 15 17:24:44 ipfire shutdown[11662]: shutting down for system reboot
Hi @sky7176,
just for the sake of completeness: How did things turn out here?
@cfusco’s post has been marked as a solution, but it would be good to hear from you as the OP whether scheduled reboots now work fine for you, and what the precise issue has been. 
Thanks, and best regards,
Peter MĂĽller
I did not find any anomalies from the register.
I preferred to replace the machine, it belongs to a customer, in any case I want to investigate the matter since the old hardware is in my office.
The procedure indicated by cfusco helped me to learn how to move in the logs with the various commands grep, cap etc. etc.
you know coming from the windows world … HI.
P.S. cfusco I send you a private msg about “Traverse Net-to-Net VPN from Road Warrior” I need same info about a configuration, I prefere send you a private msg, but if you want I open a case.
Giuseppe
Some J1900 boards has problems that it hang in bios at reboot. Try to add reboot=b or reboot=k to the kernel commandline in /etc/default/grub and regenerate the grub config.
grub-mkconfig -o /boot/grub/grub.cfg
Thanks.
I try this.
Giuseppe
just a question, in case of a core upgrade, are these changes lost or remain?
Giuseppe
It remain because /etc/grub/default is in the exclude list of core updates.
Would you mind to send it again? I can’t find it.
ok I resend you.