Hi all,
Here is my log ,from my upgrade, when my WUI also froze after some time and then I received “ipfire refused to connect” as I mentioned earlier:
Stopping SSH Server…
e[1Ae[0Ge[24G e[1;33mNot running.
e[1Ae[0Ge[-8Ge[1;34m[e[1;33m WARN e[1;34m]e[0;39m
Stopping Apache daemon…
e[1Ae[0Ge[-8Ge[1;34m[e[1;32m OK e[1;34m]e[0;39m
Starting Apache daemon…
httpd (pid 2812) already running
e[1Ae[0Ge[-8Ge[1;34m[e[1;32m OK e[1;34m]e[0;39m
Stopping Intrusion Detection System…
e[1Ae[0Ge[-8Ge[1;34m[e[1;32m OK e[1;34m]e[0;39m
Starting Intrusion Detection System…
e[1Ae[0Ge[-8Ge[1;34m[e[1;32m OK e[1;34m]e[0;39m
Stopping Unbound DNS Proxy…
e[1Ae[0Ge[-8Ge[1;34m[e[1;32m OK e[1;34m]e[0;39m
Starting Unbound DNS Proxy…
e[1Ae[0Ge[-8Ge[1;34m[e[1;32m OK e[1;34m]e[0;39m
Setting time on boot…
e[1Ae[0Ge[-8Ge[1;34m[e[1;32m OK e[1;34m]e[0;39m
Starting ntpd…
e[1Ae[0Ge[-8Ge[1;34m[e[1;32m OK e[1;34m]e[0;39m
Creating Squid swap directories…
e[1Ae[0Ge[-8Ge[1;34m[e[1;32m OK e[1;34m]e[0;39m
Starting Squid Proxy Server…
e[1Ae[0Ge[-8Ge[1;34m[e[1;32m OK e[1;34m]e[0;39m
Stopping Collection daemon…
e[1Ae[0Ge[-8Ge[1;34m[e[1;32m OK e[1;34m]e[0;39m
e[1Ae[0Ge[-8Ge[1;34m[e[1;32m OK e[1;34m]e[0;39m
Starting Collection daemon…
e[1Ae[0Ge[-8Ge[1;34m[e[1;32m OK e[1;34m]e[0;39m
mv: cannot stat ‘/var/ipfire/ovpn/ovpnconfig.new’: No such file or directory
dracut: Executing: /usr/bin/dracut --kver=6.6.15-ipfire --force
dracut: *** Including module: modsign ***
dracut: *** Including module: i18n ***
dracut: *** Including module: dm ***
dracut: Skipping udev rule: 64-device-mapper.rules
dracut: Skipping udev rule: 60-persistent-storage-dm.rules
dracut: Skipping udev rule: 55-dm.rules
dracut: *** Including module: kernel-modules ***
dracut: *** Including module: kernel-modules-extra ***
dracut: *** Including module: lvm ***
dracut: Skipping udev rule: 64-device-mapper.rules
dracut: Skipping udev rule: 56-lvm.rules
dracut: Skipping udev rule: 60-persistent-storage-lvm.rules
dracut: *** Including module: mdraid ***
dracut: Skipping udev rule: 64-md-raid.rules
dracut: *** Including module: qemu ***
dracut: *** Including module: rootfs-block ***
dracut: *** Including module: terminfo ***
dracut: *** Including module: udev-rules ***
dracut: Skipping udev rule: 40-redhat.rules
dracut: Skipping udev rule: 50-firmware.rules
dracut: Skipping udev rule: 50-udev.rules
dracut: Skipping program /bin/loginctl using in udev rule 71-seat.rules as it cannot be found
dracut: Skipping udev rule: 91-permissions.rules
dracut: Skipping udev rule: 80-drivers-modprobe.rules
dracut: *** Including module: base ***
dracut: *** Including module: fs-lib ***
dracut: *** Including modules done ***
dracut: *** Installing kernel module dependencies ***
dracut: *** Installing kernel module dependencies done ***
dracut: *** Resolving executable dependencies ***
dracut: *** Resolving executable dependencies done ***
dracut: *** Hardlinking files ***
dracut: Mode: real
dracut: Method: sha256
dracut: Files: 1070
dracut: Linked: 5 files
dracut: Compared: 0 xattrs
dracut: Compared: 49 files
dracut: Saved: 1.11 MiB
dracut: Duration: 0.022479 seconds
dracut: *** Hardlinking files done ***
dracut: Could not find ‘strip’. Not stripping the initramfs.
dracut: *** Generating early-microcode cpio image ***
dracut: *** Constructing AuthenticAMD.bin ***
dracut: *** Constructing GenuineIntel.bin ***
dracut: *** Store current command line parameters ***
dracut: *** Creating image file ‘/boot/initramfs-6.6.15-ipfire.img’ ***
dracut: *** Creating initramfs image file ‘/boot/initramfs-6.6.15-ipfire.img’ done ***
Generating grub configuration file …
Found background: /boot/grub/splash.png
Found linux image: /boot/vmlinuz-6.6.15-ipfire
Found initrd image: /boot/initramfs-6.6.15-ipfire.img
Adding boot menu entry for UEFI Firmware Settings …
done
It is very interesting as it also shows a different message to what I had, which was just Restarting Apache daemon... .
Yours has a Stopping and a Starting line. The Stopping line says OK but the Starting line says httpd is already running.
This may be a timing issue. The stopping of apache, including removing its pid may not have stopped fully when the OK for stopping Apache has been provided, so the start says it doesn’t need top start as it is already starting. However, maybe after this the pid is finally removed resulting in apache not being started again after being stopped.
Still need to understand why the messages are different between different people. Maybe some changes to initscripts did not get updated for all systems. I will have to check this out.
Stopping SSH Server…
ESC[1AESC[0GESC[24G ESC[1;33mNot running.
ESC[1AESC[0GESC[-8GESC[1;34m[ESC[1;33m WARN ESC[1;34m]ESC[0;39m
Restarting Apache daemon…
ESC[1AESC[0GESC[-8GESC[1;34m[ESC[1;32m OK ESC[1;34m]ESC[0;39m
Stopping Intrusion Detection System…
ESC[1AESC[0GESC[-8GESC[1;34m[ESC[1;32m OK ESC[1;34m]ESC[0;39m
Starting Intrusion Detection System…
ESC[1AESC[0GESC[-8GESC[1;34m[ESC[1;32m OK ESC[1;34m]ESC[0;39m
Stopping Unbound DNS Proxy…
ESC[1AESC[0GESC[-8GESC[1;34m[ESC[1;32m OK ESC[1;34m]ESC[0;39m
Starting Unbound DNS Proxy…
ESC[1AESC[0GESC[-8GESC[1;34m[ESC[1;32m OK ESC[1;34m]ESC[0;39m
Setting time on boot…
ESC[1AESC[0GESC[-8GESC[1;34m[ESC[1;32m OK ESC[1;34m]ESC[0;39m
Starting ntpd…
ESC[1AESC[0GESC[-8GESC[1;34m[ESC[1;32m OK ESC[1;34m]ESC[0;39m
Creating Squid swap directories…
ESC[1AESC[0GESC[-8GESC[1;34m[ESC[1;32m OK ESC[1;34m]ESC[0;39m
Starting Squid Proxy Server…
ESC[1AESC[0GESC[-8GESC[1;34m[ESC[1;32m OK ESC[1;34m]ESC[0;39m
Stopping Collection daemon…
ESC[1AESC[0GESC[-8GESC[1;34m[ESC[1;32m OK ESC[1;34m]ESC[0;39m
ESC[1AESC[0GESC[-8GESC[1;34m[ESC[1;32m OK ESC[1;34m]ESC[0;39m
Starting Collection daemon…
ESC[1AESC[0GESC[-8GESC[1;34m[ESC[1;32m OK ESC[1;34m]ESC[0;39m
mv: cannot stat ‘/var/ipfire/ovpn/ovpnconfig.new’: No such file or directory
dracut: Executing: /usr/bin/dracut --kver=6.6.15-ipfire --force
dracut: *** Including module: modsign ***
dracut: *** Including module: i18n ***
dracut: *** Including module: dm ***
dracut: Skipping udev rule: 64-device-mapper.rules
dracut: Skipping udev rule: 60-persistent-storage-dm.rules
dracut: Skipping udev rule: 55-dm.rules
dracut: *** Including module: kernel-modules ***
dracut: *** Including module: kernel-modules-extra ***
dracut: *** Including module: lvm ***
dracut: Skipping udev rule: 64-device-mapper.rules
dracut: Skipping udev rule: 56-lvm.rules
dracut: Skipping udev rule: 60-persistent-storage-lvm.rules
dracut: *** Including module: mdraid ***
dracut: Skipping udev rule: 64-md-raid.rules
dracut: *** Including module: qemu ***
dracut: *** Including module: rootfs-block ***
dracut: *** Including module: terminfo ***
dracut: *** Including module: udev-rules ***
dracut: Skipping udev rule: 40-redhat.rules
dracut: Skipping udev rule: 50-firmware.rules
dracut: Skipping udev rule: 50-udev.rules
dracut: Skipping program /bin/loginctl using in udev rule 71-seat.rules as it cannot be found
dracut: Skipping udev rule: 91-permissions.rules
dracut: Skipping udev rule: 80-drivers-modprobe.rules
dracut: *** Including module: base ***
dracut: *** Including module: fs-lib ***
dracut: *** Including modules done ***
dracut: *** Installing kernel module dependencies ***
dracut: *** Installing kernel module dependencies done ***
dracut: *** Resolving executable dependencies ***
dracut: *** Resolving executable dependencies done ***
dracut: *** Hardlinking files ***
dracut: Mode: real
dracut: Method: sha256
dracut: Files: 1067
dracut: Linked: 5 files
dracut: Compared: 0 xattrs
dracut: Compared: 49 files
dracut: Saved: 1.1 MiB
dracut: Duration: 0.013425 seconds
dracut: *** Hardlinking files done ***
dracut: Could not find ‘strip’. Not stripping the initramfs.
dracut: *** Generating early-microcode cpio image ***
dracut: *** Constructing AuthenticAMD.bin ***
dracut: *** Constructing GenuineIntel.bin ***
dracut: *** Store current command line parameters ***
dracut: *** Creating image file ‘/boot/initramfs-6.6.15-ipfire.img’ ***
dracut: *** Creating initramfs image file ‘/boot/initramfs-6.6.15-ipfire.img’ done ***
Generating grub configuration file …
Found background: /boot/grub/splash.png
Found linux image: /boot/vmlinuz-6.6.15-ipfire
Found initrd image: /boot/initramfs-6.6.15-ipfire.img
Adding boot menu entry for UEFI Firmware Settings …
done
(END)
I had initiated the upgrade from WUI, which is why SSH server was not running.
Re the WUI - I was certainly able to change tabs to “System”, then execute “Shutdown > Reboot” from the WUI, thus avoiding a hardware reset. Many are reporting resorting to hardware reset, although they might not have tried navigating the (apparently hung) WUI.
@rodneyp Well, in my case, I definiely DID try navigating to other tabs on the WUI and ALL gave me “ipfire refused to connect”, hence my having to hard reboot with reset button.
@bbitsch I hear you, however, I do not have a KVM/serial connction to my box, so I had no choice but to hard reset. Still ,lesson learned, will use SSH to install updates in future.
Okay, I may be getting closer to understanding what is driving this.
In November last year with Core Update 181 a change was made to the apache initscript.
This change was because there had been a problem with apache stopping working when an update had been made to glibc. So the initscript was changed to have the restart command use the stop followed by the start command with some changes in the stop command also.
The changed apache initscript was in the list of files to be changed in the update.
For some reason it looks like the change did not occur for some people.
My production system and the three vm’s I use all still have the apache initscript from before Core Update 181. I ran each CU in turn as they occurred so it is not due to missing a CU.
Other users, the ones having the problem, look to have received the updated apache initscript.
The likelihood is that the changed initscript needs a delay between the stop and start command to ensure that the pid has been fully removed from the filesystem so that the start really can start rather than finding the pid still hanging around on its way out.
I will raise a bug on this, because we need to understand why the apache initscript only got applied to some users and not to others.
A question for all the users that had a frozen WUI screen with CU185, where none of the menu items worked at all.
Did any of you have your IPFire in existence at or earlier than Core Update 181.
You can confirm this by checking in /var/log/pakfire
to see if you have the log file
update-core-upgrade-181.log
I am trying to understand if the apache initscript update did not get installed to anyone and all users having this issue had a fresh install of IPFire from CU181 or later.
If any users have the CU181 log file present then it means that the apache initscript was only updated on some systems and that will be harder to figure out why but we need to know which way round the situation is.
I updated from 184 to 185 and then WUI was frozen. But OpenVPN connections, firewall stuff etc. all seemed to be running fine. I pushed the power-button and system shut down flawlessly. After a new start, everything ran fine. The firewall had been running for several years and I updated step by step. So it might well be that there was some old stuff from before 181 on the system.
Hope this helps.
It looks like it is related to the apache initscript from Core181 and apache not shutting down before restarting.
I had several Core185 upgrades (various Intel based systems) with the apache WUI issue. All systems had previously existed and upgraded from versions before Core181.
The IPFire Core185 upgrades where the WUI did not restart had an apache initscript dated a few days before the Core181 upgrade was done (Nov-Dec2023). Those systems also showed a running apache pid (so apache did not restart)
Successful upgrades did not show the running apache pid in the upgrade log.
All the other systems on similar hardware upgraded fine.
Those systems that successfully upgraded had either older apache initscripts (dated 2020-2021 so for some reason Core181 did not update those) or new scripts dated a few days before the 185 upgrade.
[root@ipfire ~]# ls -al /etc/init.d/apache
-rwxr-xr-- 1 root root 3714 Dec 9 02:48 /etc/init.d/apache
For all the problematic upgrades (all with initscripts from 181), the update-core-upgrade-185.log showed an apache pid still running after the stop / before the restart.
We now understand that the apache initscript update from CU181 was implemented due to issues that were only found after the full release.
The updated initscript went into CU181 but anyone who had already updated to CU181 before that initscript was changed would not have got it.
The plan was to ship the updated apache initscript also with CU182 to catch all those who had updated early to CU181. Unfortunately that was missed to be done with CU182.
The updated apache initscript, together with a delay between the stop and start commands for the restart, will be shipped with CU186.
