XFS-installed IPFire wont boot up again

1

Once upon a time, XFS was available as an installation choice, and I used it for a couple of systems at the time.

It seems like the /boot partition is filling up, probably due to XFS journaling, and I rebooted a system with 0% available /boot space – it won’t come up again. Unfortunately, it’s a remote system.

I’m not really asking how to bring this system up again at this point. However it would be good to have checks for this scenario, or if XFS journaling is filling up /boot, run something to clear the journal before reboot/before running pakfire. Or even deprecate XFS, showing a warning in the UI if an XFS installation is detected.

2

Hi,

is is really hard to judge what went wrong here and there could be many reasons…

XFS is working very well, and there are checks for filesystems filling up and updates won’t install if there is not enough space available.

Why would we deprecate XFS?

3

Well, its a small drive, and I think /boot was 64 megs or so. Obviously, the check wasn’t working – and I have actually verified this on another system, also with a small /boot partition, also showing 0%. That machine also updated without error message and didn’t make the reboot but it is getting changed out tomorrow anyway, so…

But the problem is real.

4

I now have one broken machine accessible. Although “updated” to 137 without error messages, the GRUB menu shows core136. When selecting, this shows:

Loading Linux 4.14.138-ipfire...
error: file /vmlinuz-4.14.138-ipfire not found.
Loading initial ramdisk...
error: you need to load the kernel first.
5

The files on boot should need 36MB
24MB is the kernel and the rest is grub.

Can you run fsck on /boot

6

Nothing wrong with the file system.
Previous threads on this behavior:
https://forum.ipfire.org/viewtopic.php?t=20711
https://forum.ipfire.org/viewtopic.php?t=22731
Like in the second thread, my
(hd0,msdos3)/boot/
is completely empty.

7

It would be great to have the log from pakfire installing the update. That would tell us what went wrong.

On new installations the /boot partition is already larger so that no out of space condition should happen.

But since all systems have exactly the same files on this partition, they should all have the same issue.

Unfortunately nobody has ever opened a ticket on our bug tracker for this.

8

I can upload the whole enchilada somewhere if you want but the relevant lines (from update-core-upgrade-137) seems to be these:

Extracting files...
boot/
boot/vmlinuz-4.14.150-ipfire
boot/System.map-4.14.150-ipfire
boot/initramfs-4.14.150-ipfire.img
boot/config-4.14.150-ipfire
tar: boot/config-4.14.150-ipfire: Wrote only 1024 of 10240 bytes

…

var/lib/suricata/classification.config
tar: Exiting with failure status due to previous errors
...Finished.
Stopping strongSwan IPsec...

…

Generating grub configuration file ...
Found background: /boot/grub/splash.png
Found linux image: /boot/vmlinuz-4.14.150-ipfire
Found initrd image: /boot/initramfs-4.14.150-ipfire.img
/usr/sbin/grub-mkconfig: line 262: echo: write error: No space left on device
9

There must be a bug in XFS handling or a filesystem corruption. The updater erase the old kernel which is 24MB before unpack the new one so it need nearly no additional disk space.

Is xfs not freeing the space of erased files properly?

10

Preallocation?

11

This is what I wanted to see - or rather would have preferred not to.

I think there isn’t anything we can do about these systems now apart from trialing the mount options. However, they are up and running and we cannot change the partition size.

On new installations, the /boot partition should be large enough so that we won’t run into these problems any more.

12

How large is /boot nowadays, I see 110M on my reinstalled system?
At the very least, there should be a warning that one must reinstall the system (and pakfire shouldn’t run on XFS + small sized partition, I guess).
I guess you can still use gparted to enlarge the partition though?

13

@ms is there any table who report the boot partition size during the history of IPFire releases?

14

Yes of course.

Yes, it is in the source of the installer:

https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=src/installer/hw.c;h=71a1f1cce369fb027ce65a2e5b5ae919f4a6ae66;hb=HEAD#l434

Monitor that line for /boot.

15

So the size of /boot partition depends on the size of the disk. Such deadly as clever.

16

No, it does not.

But of course the whole layout does.