After update ipfire to 157, no boot

I will, next week end when I fix my friend’s machine.

@pmueller reading to fast after work😅, it’s a Apu family but not a Ipfire mini.

Sorry for the fast reading and reply.

Best regards.

Ok, related question, I have two work IPFire boxes, one is local, one is remote, and both are currently running 156 but I processed the 157 update on both of them, they have to restart for the upgrade to to take effect, is there a way that I can cancel the pending install of 157? I do NOT want to restart and try to boot 157 after hearing the challenges that multiple people are having.

@neopegasus, @angrytux, @thier28, @cfusco -

Folks - you’ve been asked to supply some information to help the IPFire Developers figure out this issue.

There are two easy commands:

cat /var/log/pakfire

df -h

Please help us help you!

EDIT:
Zip files are fine.

3 Likes

You will have to reinstall IPFire and then restore your backup.

Carrying out the upgrade will have installed all the new files. Some won’t be properly active till after a reboot.

Fir info, my production IPFire system upgraded from 156 to 157 without a murmur.
All my services that were running before were running after update and reboot.

I have also done several vm updates from 156 to 157 and again no hiccups.

Some people are having issues but I suspect the majority not.

Edit:
Currently 27.03% of users are on Core Update 156
Already there are 23.83% of users on Core Update 157

2 Likes

Hi,

to share my current guess (!) publicly: We have increased the size of the /boot partition several times during the last ~ 5 years, as the Linux kernel grew bigger and bigger. Unfortunately, this requires reinstalling IPFire, since mounted partitions cannot be changed by the same operating system that is running out of them.

I could imagine the affected people to run some IPFire installations that have been active for a very long time (> 4 years), hence still having a relatively small /boot partition. However, as far as I am aware, we do not install a kernel if we can foresee running out of disk space - an upgrade should simply stop before touching anything on disk.

So, there might be a bug in this routine, causing disk space miscalculations. However, that would not really explain why Grub is affected, but is the only idea I have at hand given the patchy facts. :slight_smile:

This is why I am asking for df -h information. Let’s see if the affected systems share a small /boot partition in common.

Thanks, and best regards,
Peter Müller

3 Likes

Peter hi , this is my df -h :

[root@PegasusFirewall ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        3.9G  4.0K  3.9G   1% /dev
tmpfs           3.9G   12K  3.9G   1% /dev/shm
tmpfs           3.9G  616K  3.9G   1% /run
/dev/sda4       457G  3.7G  430G   1% /
/dev/sda1       110M   35M   67M  34% /boot
/dev/sda2        32M  254K   32M   1% /boot/efi
/var/lock       8.0M   12K  8.0M   1% /var/lock
none            3.9G  468K  3.9G   1% /var/log/vnstat
none            3.9G   53M  3.8G   2% /var/log/rrd

This is the reinstall ipfire 157 , I hope I understand you right.

and i hope it helps.

ps i just did a upgrade last weekend ( so after the problem with 157) the old system has 350g hdd the new have 500ssd

1 Like

Here is ours - it is unknown if we are affected here or not yet.

[root@ipfire pakfire]# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        3.9G  4.0K  3.9G   1% /dev
tmpfs           3.9G   12K  3.9G   1% /dev/shm
tmpfs           3.9G  668K  3.9G   1% /run
/dev/sda4       116G   85G   26G  77% /
/dev/sda1       110M   35M   67M  34% /boot
/dev/sda2        32M  254K   32M   1% /boot/efi
/var/lock       8.0M   12K  8.0M   1% /var/lock
1 Like

Hey Jon, my log/zip file is here BTW ;
https://community.ipfire.org/t/after-update-ipfire-to-157-no-boot/5641/10?u=neopegasus

1 Like

Hi All,
It happened to me too. I have a relatively new firewall appliance that has a intel J3160 processor. Purchased the device back in January of this year so it has not been running long. At the initial install I would have put the latest version on the device, so it would have been a Jan 2021 version (not sure what release). To recover, I had to do a new install with backup recovery to get it up and running again. The profile is 052661e7f9c06c03afdb1fb849ac27cbbf7f860e, assuming it does not change from a complete rebuild. ( Which is a question, does the profile id change after a complete rebuild of the system?, just curious. ) On the other hand, my backup firewall device, which has a i3-4005U processor, updated just fine (it started at release 154), its profile id is 3dce541a03a2724dab08ca9ccfd03875fc3dc899.

I am not sure thattelling you the disk allocation after recovery will tell you anything as I have no way to determine if things changed from the pre-core 157 upgrade.

I hope this helps, P

@pmueller: my input for consideration as you are narrowing in on root cause. I purchased my IPFire Mini Appliance about one year back. Your assumption that it was running an old installation >4 years is still possible but would be strange.

I can’t provide evidence as I reinstalled right away before I have seen your request for logs.

I would recommend to everybody to create a full image with clonezilla before updating a new core.

Yes, it’s an additional downtime but it’s worth!

Greetz

I did update my machine, which is an apu2. Everything worked as intended. The partitioning is following the last IPFire scheme:

Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        2.0G  4.0K  2.0G   1% /dev
tmpfs           2.0G   12K  2.0G   1% /dev/shm
tmpfs           2.0G  584K  2.0G   1% /run
/dev/sda4        54G  3.0G   49G   6% /
/dev/sda1       110M   35M   67M  34% /boot
/dev/sda2        32M  254K   32M   1% /boot/efi
/var/lock       8.0M   12K  8.0M   1% /var/lock

I will check during the week end my friend’s apu1 IPFire machine and see what happened. To be precise, I only know that after the update it did not boot. Maybe it is not this issue but something else. I will report here asap.

For now, I can only state that my update went well, without any problem.

1 Like

Hi all,

thanks for your replies. Looks like we can rule out the /boot partition size issue.

While upgrading to Core Update 157, we reconfigure Grub (see this for details), which should look like this in /var/log/pakfire/update-core-upgrade-157.log:

Generating grub configuration file ...
Found background: /boot/grub/splash.png
Found linux image: /boot/vmlinuz-4.14.232-ipfire
Found initrd image: /boot/initramfs-4.14.232-ipfire.img
done

However, in the log file provided by @neopegasus (thanks), it looks like Grub missed the kernel:

Generating grub configuration file ...
Found background: /boot/grub/splash.png
done

Whyever that is - I have no idea at the moment. Apparently, storage size, partition layouts or age of the IPFire installation are not the root cause of this.

The only thing I can think of currently is some extremly slow storage, as we do not conduct a sync after running extract_files() - but I don’t know if this can possibly (in means of file system internals) to cause Grub to miss the kernel files completely.

@neopegasus, @angrytux, @thier28, @cfusco: You are not running IPFire on SD or Compaq flash cards, are you? (At least the IPFire Mini appliances should not.) :slight_smile:

Clonezilla is not necessary for this. The web interface is capable of creating backups as well, and does so automatically before installing a Core Update.

@cwensink: Could you run grub-mkconfig -o /boot/grub/grub.cfg manually on one of your system which has Core Update 157 installed but not rebooted since? If the output of that command looks like the first log snippet above, you can safely reboot. If not, please let us know. :slight_smile:

Thanks, and best regards,
Peter Müller

2 Likes

Looks like I am safe, here is the output for both systems

[root@ipfire ~]# grub-mkconfig -o /boot/grub/grub.cfg
Generating grub configuration file ...
Found background: /boot/grub/splash.png
Found linux image: /boot/vmlinuz-4.14.232-ipfire
Found initrd image: /boot/initramfs-4.14.232-ipfire.img
done
[root@ipfire ~]#

Hi,

may I ask why? Hardware failures? Was the Grub issue reproducible?

Thanks, and best regards,
Peter Müller

Hi,

all right, I wish you all the best and keep my fingers crossed. :slight_smile:

Thanks, and best regards,
Peter Müller

Peter,

For those that are affected by this issue, is there going to be a 157.1 update or a 158 update published to fix this issue once the core cause is identified?

Chris

Hi,

I cannot answer your question at the moment as we do not even have an educated guess about what the root cause is. Sorry.

Thanks, and best regards,
Peter Müller

1 Like

For my part, no. msata SSD