I will, next week end when I fix my friend’s machine.
@pmueller reading to fast after work😅, it’s a Apu family but not a Ipfire mini.
Sorry for the fast reading and reply.
Best regards.
Ok, related question, I have two work IPFire boxes, one is local, one is remote, and both are currently running 156 but I processed the 157 update on both of them, they have to restart for the upgrade to to take effect, is there a way that I can cancel the pending install of 157? I do NOT want to restart and try to boot 157 after hearing the challenges that multiple people are having.
@neopegasus, @angrytux, @thier28, @cfusco -
Folks - you’ve been asked to supply some information to help the IPFire Developers figure out this issue.
There are two easy commands:
cat /var/log/pakfire
df -h
Please help us help you!
EDIT:
Zip files are fine.
You will have to reinstall IPFire and then restore your backup.
Carrying out the upgrade will have installed all the new files. Some won’t be properly active till after a reboot.
Fir info, my production IPFire system upgraded from 156 to 157 without a murmur.
All my services that were running before were running after update and reboot.
I have also done several vm updates from 156 to 157 and again no hiccups.
Some people are having issues but I suspect the majority not.
Edit:
Currently 27.03% of users are on Core Update 156
Already there are 23.83% of users on Core Update 157
Hi,
to share my current guess (!) publicly: We have increased the size of the /boot
partition several times during the last ~ 5 years, as the Linux kernel grew bigger and bigger. Unfortunately, this requires reinstalling IPFire, since mounted partitions cannot be changed by the same operating system that is running out of them.
I could imagine the affected people to run some IPFire installations that have been active for a very long time (> 4 years), hence still having a relatively small /boot
partition. However, as far as I am aware, we do not install a kernel if we can foresee running out of disk space - an upgrade should simply stop before touching anything on disk.
So, there might be a bug in this routine, causing disk space miscalculations. However, that would not really explain why Grub is affected, but is the only idea I have at hand given the patchy facts.
This is why I am asking for df -h
information. Let’s see if the affected systems share a small /boot
partition in common.
Thanks, and best regards,
Peter Müller
Peter hi , this is my df -h :
[root@PegasusFirewall ~]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 3.9G 4.0K 3.9G 1% /dev
tmpfs 3.9G 12K 3.9G 1% /dev/shm
tmpfs 3.9G 616K 3.9G 1% /run
/dev/sda4 457G 3.7G 430G 1% /
/dev/sda1 110M 35M 67M 34% /boot
/dev/sda2 32M 254K 32M 1% /boot/efi
/var/lock 8.0M 12K 8.0M 1% /var/lock
none 3.9G 468K 3.9G 1% /var/log/vnstat
none 3.9G 53M 3.8G 2% /var/log/rrd
This is the reinstall ipfire 157 , I hope I understand you right.
and i hope it helps.
ps i just did a upgrade last weekend ( so after the problem with 157) the old system has 350g hdd the new have 500ssd
Here is ours - it is unknown if we are affected here or not yet.
[root@ipfire pakfire]# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 3.9G 4.0K 3.9G 1% /dev
tmpfs 3.9G 12K 3.9G 1% /dev/shm
tmpfs 3.9G 668K 3.9G 1% /run
/dev/sda4 116G 85G 26G 77% /
/dev/sda1 110M 35M 67M 34% /boot
/dev/sda2 32M 254K 32M 1% /boot/efi
/var/lock 8.0M 12K 8.0M 1% /var/lock
Hey Jon, my log/zip file is here BTW ;
https://community.ipfire.org/t/after-update-ipfire-to-157-no-boot/5641/10?u=neopegasus
Hi All,
It happened to me too. I have a relatively new firewall appliance that has a intel J3160 processor. Purchased the device back in January of this year so it has not been running long. At the initial install I would have put the latest version on the device, so it would have been a Jan 2021 version (not sure what release). To recover, I had to do a new install with backup recovery to get it up and running again. The profile is 052661e7f9c06c03afdb1fb849ac27cbbf7f860e, assuming it does not change from a complete rebuild. ( Which is a question, does the profile id change after a complete rebuild of the system?, just curious. ) On the other hand, my backup firewall device, which has a i3-4005U processor, updated just fine (it started at release 154), its profile id is 3dce541a03a2724dab08ca9ccfd03875fc3dc899.
I am not sure thattelling you the disk allocation after recovery will tell you anything as I have no way to determine if things changed from the pre-core 157 upgrade.
I hope this helps, P
@pmueller: my input for consideration as you are narrowing in on root cause. I purchased my IPFire Mini Appliance about one year back. Your assumption that it was running an old installation >4 years is still possible but would be strange.
I can’t provide evidence as I reinstalled right away before I have seen your request for logs.
I would recommend to everybody to create a full image with clonezilla before updating a new core.
Yes, it’s an additional downtime but it’s worth!
Greetz
I did update my machine, which is an apu2. Everything worked as intended. The partitioning is following the last IPFire scheme:
Filesystem Size Used Avail Use% Mounted on
devtmpfs 2.0G 4.0K 2.0G 1% /dev
tmpfs 2.0G 12K 2.0G 1% /dev/shm
tmpfs 2.0G 584K 2.0G 1% /run
/dev/sda4 54G 3.0G 49G 6% /
/dev/sda1 110M 35M 67M 34% /boot
/dev/sda2 32M 254K 32M 1% /boot/efi
/var/lock 8.0M 12K 8.0M 1% /var/lock
I will check during the week end my friend’s apu1 IPFire machine and see what happened. To be precise, I only know that after the update it did not boot. Maybe it is not this issue but something else. I will report here asap.
For now, I can only state that my update went well, without any problem.
Hi all,
thanks for your replies. Looks like we can rule out the /boot
partition size issue.
While upgrading to Core Update 157, we reconfigure Grub (see this for details), which should look like this in /var/log/pakfire/update-core-upgrade-157.log
:
Generating grub configuration file ...
Found background: /boot/grub/splash.png
Found linux image: /boot/vmlinuz-4.14.232-ipfire
Found initrd image: /boot/initramfs-4.14.232-ipfire.img
done
However, in the log file provided by @neopegasus (thanks), it looks like Grub missed the kernel:
Generating grub configuration file ...
Found background: /boot/grub/splash.png
done
Whyever that is - I have no idea at the moment. Apparently, storage size, partition layouts or age of the IPFire installation are not the root cause of this.
The only thing I can think of currently is some extremly slow storage, as we do not conduct a sync
after running extract_files()
- but I don’t know if this can possibly (in means of file system internals) to cause Grub to miss the kernel files completely.
@neopegasus, @angrytux, @thier28, @cfusco: You are not running IPFire on SD or Compaq flash cards, are you? (At least the IPFire Mini appliances should not.)
Clonezilla is not necessary for this. The web interface is capable of creating backups as well, and does so automatically before installing a Core Update.
@cwensink: Could you run grub-mkconfig -o /boot/grub/grub.cfg
manually on one of your system which has Core Update 157 installed but not rebooted since? If the output of that command looks like the first log snippet above, you can safely reboot. If not, please let us know.
Thanks, and best regards,
Peter Müller
Looks like I am safe, here is the output for both systems
[root@ipfire ~]# grub-mkconfig -o /boot/grub/grub.cfg
Generating grub configuration file ...
Found background: /boot/grub/splash.png
Found linux image: /boot/vmlinuz-4.14.232-ipfire
Found initrd image: /boot/initramfs-4.14.232-ipfire.img
done
[root@ipfire ~]#
Hi,
may I ask why? Hardware failures? Was the Grub issue reproducible?
Thanks, and best regards,
Peter Müller
Hi,
all right, I wish you all the best and keep my fingers crossed.
Thanks, and best regards,
Peter Müller
Peter,
For those that are affected by this issue, is there going to be a 157.1 update or a 158 update published to fix this issue once the core cause is identified?
Chris
Hi,
I cannot answer your question at the moment as we do not even have an educated guess about what the root cause is. Sorry.
Thanks, and best regards,
Peter Müller
For my part, no. msata SSD